From f93515adb9f401c2a0aea1556c6712dc7789cd39 Mon Sep 17 00:00:00 2001 From: mikedlr Date: Wed, 23 Aug 2017 15:34:30 +0100 Subject: [PATCH] [MERGE] AWS module Guidelines - multiple changes to bring up to date and start on aws.core (#26871) * aws module Guidelines - rewrite - add AWS WG - add fail_json_aws - more modern practices and fix some spelling * aws module Guidelines - minor further changes with agressive fill to 99 width * aws module Guidelines - formatting fix as per gundalow request * aws module Guidelines - mark more keywords and code with backquotes * AWS module guidelines - fixes from ryansb review + minor other changes --- .../modules/cloud/amazon/GUIDELINES.md | 245 ++++++++++++------ 1 file changed, 159 insertions(+), 86 deletions(-) diff --git a/lib/ansible/modules/cloud/amazon/GUIDELINES.md b/lib/ansible/modules/cloud/amazon/GUIDELINES.md index ccf5fcaef8..3a0cfc4767 100644 --- a/lib/ansible/modules/cloud/amazon/GUIDELINES.md +++ b/lib/ansible/modules/cloud/amazon/GUIDELINES.md @@ -1,36 +1,46 @@ # Guidelines for AWS modules -## Getting Started +The Ansible AWS modules and these guidelines are maintained by the Ansible AWS Working Group. For +further information see +[the AWS working group community page](https://github.com/ansible/community/tree/master/group-aws). +If you are planning to contribute AWS modules to Ansible then getting in touch with the the working +group will be a good way to start, especially because a similar module may already be under +development. -Since Ansible 2.0, it is required that all new AWS modules are written to use boto3. +## Choose Boto3 not Boto -Prior to 2.0, modules may have been written in boto or boto3. The effort to port all modules to boto3 has begun. +Starting from Ansible 2.0, it has been required that all new AWS modules are written to use boto3. +Please do not add new dependencies on the old boto library. + +Prior to 2.0, modules may have been written in boto or boto3. The effort to port all modules to +boto3 has begun. From Ansible 2.4 it is permissible for modules which previously required boto to +start to require boto3 in order to deliver the functionality previously supported with boto and the +boto dependency can be deleted. ## Bug fixing -Bug fixes to code that relies on boto will still be accepted. When possible, the code should be ported to use boto3. +Bug fixes to code that relies on boto will still be accepted. When possible, the code should be +ported to use boto3. ## Naming your module -Base the name of the module on the part of AWS that -you actually use. (A good rule of thumb is to take -whatever module you use with boto as a starting point). +Base the name of the module on the part of AWS that you actually use. (A good rule of thumb is to +take whatever module you use with boto as a starting point). Don't further abbreviate names - if +something is a well known abbreviation due to it being a major component of AWS, that's fine, but +don't create new ones independently (e.g. VPC, ELB, and similar abbreviations are fine). -Don't further abbreviate names - if something is a well -known abbreviation due to it being a major component of -AWS, that's fine, but don't create new ones independently -(e.g. VPC, ELB, etc. are fine) +Unless the name of your service is quite unique, please consider using "aws_" as a prefix. For +example "aws_lambda". ## Adding new features -Try to keep backward compatibility with relatively recent -versions of boto3. That means that if you want to implement some -functionality that uses a new feature of boto3, it should only -fail if that feature actually needs to be run, with a message -saying which version of boto3 is needed. +Try to keep backward compatibility with relatively recent versions of boto3. That means that if you +want to implement some functionality that uses a new feature of boto3, it should only fail if that +feature actually needs to be run, with a message stating the missing feature and minimum required +version of boto3. -Use feature testing (e.g. `hasattr('boto3.module', 'shiny_new_method')`) -to check whether boto3 supports a feature rather than version checking +Use feature testing (e.g. `hasattr('boto3.module', 'shiny_new_method')`) to check whether boto3 +supports a feature rather than version checking e.g. from the `ec2` module: ```python @@ -41,11 +51,18 @@ else: module.fail_json(msg="instance_profile_name parameter requires boto version 2.5.0 or higher") ``` -## Using boto and boto3 +## Using botocore and boto3 ### Importing -Wrap import statements in a try block and fail the module later if the import fails +The `ansible.module_utils.ec2` module and `ansible.module_utils.core.aws` modules will both +automatically import boto3 and botocore. If boto3 is missing from the system then the variable +HAS_BOTO3 will be set to false. Normally, this means that modules don't need to import either +botocore or boto3 directly. + +If you want to import the modules anyway (for example `from botocore.exception import +ClientError`) Wrap import statements in a try block and fail the module later using HAS_BOTO3 if +the import fails. #### boto @@ -66,37 +83,30 @@ def main(): #### boto3 ```python -try: - import boto3 - HAS_BOTO3 = True -except ImportError: - HAS_BOTO3 = False +from ansible.module_utils.aws.core import HAS_BOTO3 def main(): if not HAS_BOTO3: - module.fail_json(msg='boto3 required for this module') + module.fail_json(msg='boto3 and botocore are required for this module') ``` #### boto and boto3 combined -Ensure that you clearly document if a new parameter requires requires a specific version. Import boto3 at the top of the -module as normal and then use the HAS_BOTO3 bool when necessary, before the new feature. +Ensure that you clearly document if a new parameter requires requires a specific version. Import +boto3 at the top of the module as normal and then use the HAS_BOTO3 bool when necessary, before the +new feature. ```python +from ansible.module_utils.ec2 import HAS_BOTO3 + try: import boto HAS_BOTO = True except ImportError: HAS_BOTO = False -try: - import boto3 - HAS_BOTO3 = True -except ImportError: - HAS_BOTO3 = False - -if my_new_feauture_Parameter_is_set: +if my_new_feature_Parameter_is_set: if HAS_BOTO3: # do feature else: @@ -107,9 +117,11 @@ if my_new_feauture_Parameter_is_set: To connect to AWS, you should use `get_aws_connection_info` and then `boto3_conn`. -These functions handle some of the more esoteric connection options, such as security tokens and boto profiles. +These functions handle some of the more esoteric connection options, such as security tokens and +boto profiles. -Some boto services require that the region is specified. You should check for the region parameter if required. +Some boto services require that the region is specified. You should check for the region parameter +if required. #### boto @@ -128,8 +140,9 @@ else: #### boto3 -An example of connecting to ec2 is shown below. Note that there is no 'NoAuthHandlerFound' exception handling like in boto. -Instead, an AuthFailure exception will be thrown when you use 'connection'. See exception handling. +An example of connecting to ec2 is shown below. Note that there is no `NoAuthHandlerFound` +exception handling like in boto. Instead, an `AuthFailure` exception will be thrown when you use +'connection'. See exception handling. ```python region, ec2_url, aws_connect_params = get_aws_connection_info(module, boto3=True) @@ -139,10 +152,11 @@ else: module.fail_json(msg="region must be specified") ``` -### Exception Handling +### Exception Handling for boto -You should wrap any boto call in a try block. If an exception is thrown, it is up to you decide how to handle it -but usually calling fail_json with the error or helpful message and traceback will suffice. +You should wrap any boto call in a try block. If an exception is thrown, it is up to you decide how +to handle it but usually calling fail_json with the error or helpful message and traceback will +suffice. #### boto @@ -166,11 +180,64 @@ except BotoServerError as e: **camel_dict_to_snake_dict(e.message)) ``` -#### boto3 +### Exception Handling for boto3 and botocore -For more information on botocore exception handling see [http://botocore.readthedocs.org/en/latest/client_upgrades.html#error-handling] +You should wrap any boto3 or botocore call in a try block. If an exception is thrown, then there +are a number of possibilities for handling it. -Boto3 provides lots of useful info when an exception is thrown so pass this to the user along with the message. +* use aws_module.fail_json_aws() to report the module failure in a standard way +* retry using AWSRetry +* use fail_json() to report the failure without using `ansible.module_utils.aws.core` +* do something custom in the case where you know how to handle the exception + +For more information on botocore exception handling see [the botocore error documentation](http://botocore.readthedocs.org/en/latest/client_upgrades.html#error-handling). + +#### using fail_json_aws() + +_fail_json_aws() is a new method and may be subject to change. You can use it in modules which are +being contributed back to Ansible, however if you are publishing your module separately please +don't use it before the start of 2018 / Ansible 2.4_ + +In the AnsibleAWSModule there is a special method, `module.fail_json.aws()` for nice reporting of +exceptions. Call this on your exception and it will report the error together with a traceback for +use in Ansible verbose mode. + +```python +from ansible.module_utils.aws.core import HAS_BOTO3, AnsibleAWSModule + +# Set up module parameters +... + +# Connect to AWS +... + +# Make a call to AWS +try: + result = connection.aws_call() +except Exception as e: + module.fail_json_aws(e, msg="trying to do aws_call") +``` + +Note that it should normally be acceptable to catch all normal exceptions here, however if you +expect anything other than botocore exceptions you should test everything works as expected. + +If you need to perform an action based on the error boto3 returned, use the error code. + +```python +# Make a call to AWS +try: + result = connection.aws_call() +except ClientError, e: + if e.response['Error']['Code'] == 'NoSuchEntity': + return None + else: + module.fail_json_aws(e, msg="trying to do aws_call") +``` + +#### using fail_json() and avoiding ansible.module_utils.aws.core + +Boto3 provides lots of useful information when an exception is thrown so pass this to the user +along with the message. ```python # Import ClientError from botocore @@ -207,17 +274,16 @@ except ClientError as e: ### Returning Values -When you make a call using boto3, you will probably get back some useful information that you should return in the module. +When you make a call using boto3, you will probably get back some useful information that you +should return in the module. As well as information related to the call itself, you will also have +some response metadata. It is OK to return this to the user as well as they may find it useful. -As well as information related to the call itself, you will also have some response metadata. It is OK to return this to -the user as well as they may find it useful. +Boto3 returns all values CamelCased. Ansible follows Python standards for variable names and uses +snake_case. There is a helper function in module_utils/ec2.py called `camel_dict_to_snake_dict` +that allows you to easily convert the boto3 response to snake_case. -Boto3 returns all values CamelCased. Ansible follows Python standards for variable names and uses snake_case. There is a -helper function in module_utils/ec2.py called `camel_dict_to_snake_dict` that allows you to easily convert the boto3 -response to snake_case. - -You should use this helper function and avoid changing the names of values returned by Boto3. E.g. if boto3 returns a -value called 'SecretAccessKey' do not change it to 'AccessKey'. +You should use this helper function and avoid changing the names of values returned by Boto3. +E.g. if boto3 returns a value called 'SecretAccessKey' do not change it to 'AccessKey'. ```python # Make a call to AWS @@ -229,7 +295,8 @@ module.exit_json(changed=True, **camel_dict_to_snake_dict(result)) ### Dealing with IAM JSON policy -If your module accepts IAM JSON policies then set the type to 'json' in the module spec. For example" +If your module accepts IAM JSON policies then set the type to 'json' in the module spec. For +example: ```python argument_spec.update( @@ -239,8 +306,8 @@ argument_spec.update( ) ``` -Note that AWS is unlikely to return the policy in the same order that is was submitted. Therefore, a helper -function has been created to order policies before comparison. +Note that AWS is unlikely to return the policy in the same order that is was submitted. Therefore, +a helper function has been created to order policies before comparison. ```python # Get the policy from AWS @@ -260,60 +327,66 @@ else: AWS has a concept of resource tags. Usually the boto3 API has separate calls for tagging and untagging a resource. For example, the ec2 API has a create_tags and delete_tags call. -It is common practice in Ansible AWS modules to have a 'purge_tags' parameter that defaults to true. +It is common practice in Ansible AWS modules to have a `purge_tags` parameter that defaults to +true. -The purge_tags parameter means that existing tags will be deleted if they are not specified in -by the Ansible playbook. +The `purge_tags` parameter means that existing tags will be deleted if they are not specified by +the Ansible task. -There is a helper function 'compare_aws_tags' to ease dealing with tags. It can compare two dicts and -return the tags to set and the tags to delete. See the Helper function section below for more detail. +There is a helper function `compare_aws_tags` to ease dealing with tags. It can compare two dicts +and return the tags to set and the tags to delete. See the Helper function section below for more +detail. ### Helper functions -Along with the connection functions in Ansible ec2.py module_utils, there are some other useful functions detailed below. +Along with the connection functions in Ansible ec2.py module_utils, there are some other useful +functions detailed below. #### camel_dict_to_snake_dict -boto3 returns results in a dict. The keys of the dict are in CamelCase format. In keeping -with Ansible format, this function will convert the keys to snake_case. +boto3 returns results in a dict. The keys of the dict are in CamelCase format. In keeping with +Ansible format, this function will convert the keys to snake_case. #### ansible_dict_to_boto3_filter_list -Converts a an Ansible list of filters to a boto3 friendly list of dicts. This is useful for -any boto3 _facts modules. +Converts a an Ansible list of filters to a boto3 friendly list of dicts. This is useful for any +boto3 `_facts` modules. #### boto3_tag_list_to_ansible_dict -Converts a boto3 tag list to an Ansible dict. Boto3 returns tags as a list of dicts containing keys called -'Key' and 'Value' by default. This key names can be overriden when calling the function. For example, if you have already -camel_cased your list of tags you may want to pass lowercase key names instead i.e. 'key' and 'value'. +Converts a boto3 tag list to an Ansible dict. Boto3 returns tags as a list of dicts containing keys +called 'Key' and 'Value' by default. This key names can be overridden when calling the function. +For example, if you have already camel_cased your list of tags you may want to pass lowercase key +names instead i.e. 'key' and 'value'. -This function converts the list in to a single dict where the dict key is the tag key and the dict value is the tag value. +This function converts the list in to a single dict where the dict key is the tag key and the dict +value is the tag value. #### ansible_dict_to_boto3_tag_list -Opposite of above. Converts an Ansible dict to a boto3 tag list of dicts. You can again override the key names used if 'Key' -and 'Value' is not suitable. +Opposite of above. Converts an Ansible dict to a boto3 tag list of dicts. You can again override +the key names used if 'Key' and 'Value' is not suitable. #### get_ec2_security_group_ids_from_names -Pass this function a list of security group names or combination of security group names and IDs and this function will -return a list of IDs. You should also pass the VPC ID if known because security group names are not necessarily unique -across VPCs. +Pass this function a list of security group names or combination of security group names and IDs +and this function will return a list of IDs. You should also pass the VPC ID if known because +security group names are not necessarily unique across VPCs. #### sort_json_policy_dict -Pass any JSON policy dict to this function in order to sort any list contained therein. This is useful -because AWS rarely return lists in the same order that they were submitted so without this function, comparison -of identical policies returns false. +Pass any JSON policy dict to this function in order to sort any list contained therein. This is +useful because AWS rarely return lists in the same order that they were submitted so without this +function, comparison of identical policies returns false. ### compare_aws_tags -Pass two dicts of tags and an optional purge parameter and this function will return a dict containing key pairs you need -to modify and a list of tag key names that you need to remove. Purge is True by default. If purge is False then any -existing tags will not be modified. +Pass two dicts of tags and an optional purge parameter and this function will return a dict +containing key pairs you need to modify and a list of tag key names that you need to remove. Purge +is True by default. If purge is False then any existing tags will not be modified. -This function is useful when using boto3 'add_tags' and 'remove_tags' functions. Be sure to use the other helper function -'boto3_tag_list_to_ansible_dict' to get an appropriate tag dict before calling this function. Since the AWS APIs are not -uniform (e.g. EC2 versus Lambda) this will work without modification for some (Lambda) and others may need modification -before using these values (such as EC2, with requires the tags to unset to be in the form [{'Key': key1}, {'Key': key2}]). +This function is useful when using boto3 'add_tags' and 'remove_tags' functions. Be sure to use the +other helper function `boto3_tag_list_to_ansible_dict` to get an appropriate tag dict before +calling this function. Since the AWS APIs are not uniform (e.g. EC2 versus Lambda) this will work +without modification for some (Lambda) and others may need modification before using these values +(such as EC2, with requires the tags to unset to be in the form `[{'Key': key1}, {'Key': key2}]`).