Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.
* Change AnsiballZ wrapper to use import to invoke the module
We need the module to think of itself as a script because it could be
coded as:
main()
or as:
if __name__ == '__main__':
main()
Or even as:
if __name__ == '__main__':
random_function_name()
A script will invoke all of those. Prior to this change, we invoked
a second Python interpreter on the module so that it really was
a script. However, this means that we have to run python twice (once
for the AnsiballZ wrapper and once for the module). This change makes
the module think that it is a script (because __name__ in the module ==
'__main__') but it's actually being invoked by us importing the module
code.
There's three ways we've come up to do this.
* The most elegant is to use zipimporter and tell the import mechanism
that the module being loaded is __main__:
* 5959f11c9d/lib/ansible/executor/module_common.py (L175)
* zipimporter is nice because we do not have to extract the module from
the zip file and save it to the disk when we do that. The import
machinery does it all for us.
* The drawback is that modules do not have a __file__ which points
to a real file when they do this. Modules could be using __file__
to for a variety of reasons, most of those probably have
replacements (the most common one is to find a writable directory
for temporary files. AnsibleModule.tmpdir should be used instead)
We can monkeypatch __file__ in fom AnsibleModule initialization
but that's kind of gross. There's no way I can see to do this
from the wrapper.
* Next, there's imp.load_module():
* https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
* imp has the nice property of allowing us to set __name__ to
__main__ without changing the name of the file itself
* We also don't have to do anything special to set __file__ for
backwards compatibility (although the reason for that is the
drawback):
* Its drawback is that it requires the file to exist on disk so we
have to explicitly extract it from the zipfile and save it to
a temporary file
* The last choice is to use exec to execute the module:
* https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
* The code we would have to maintain for this looks pretty clean.
In the wrapper we create a ModuleType, set __file__ on it, read
the module's contents in from the zip file and then exec it.
* Drawbacks: We still have to explicitly extract the file's contents
from the zip archive instead of letting python's import mechanism
handle it.
* Exec also has hidden performance issues and breaks certain
assumptions that modules could be making about their own code:
http://lucumr.pocoo.org/2011/2/1/exec-in-python/
Our plan is to use imp.load_module() for now, deprecate the use of
__file__ in modules, and switch to zipimport once the deprecation
period for __file__ is over (without monkeypatching a fake __file__ in
via AnsibleModule).
* Rename the name of the AnsiBallZ wrapped module
This makes it obvious that the wrapped module isn't the module file that
we distribute. It's part of trying to mitigate the fact that the module
is now named __main)).py in tracebacks.
* Shield all wrapper symbols inside of a function
With the new import code, all symbols in the wrapper become visible in
the module. To mitigate the chance of collisions, move most symbols
into a toplevel function. The only symbols left in the global namespace
are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
revised porting guide entry
Integrate code coverage collection into AnsiballZ.
ci_coverage
ci_complete
now can use `ansible-test units module_name` for the aws_s3 and aws_api_gateway modules
changes to modules/cloud/amazon/aws_api_gateway and modules/cloud/amazon/aws_s3 are not triggering the unit tests; also fix aws_s3 from importing non-exist module and skipping tests
changes to module_utils/aws/core.py are only being unit tested on modules that import from the file (if they have a corresponding test) or tests that import from the file themselves.
* Add a module parameter to configure the max fetched AWS CFN stack events
* Add version documentation for new configuration option
* Increase default in order to make sure that enough are fetched by default. This align roughly with the limit of manageable resources in CloudFormation.
* Refactor ec2_group
Replace nested for loops with list comprehensions
Purge rules before adding new ones in case sg has maximum permitted rules
* Add check mode tests for ec2_group
* add tests
* Remove dead code
* Fix integration test assertions for old boto versions
* Add waiter for security group that is autocreated
* Add support for in-account group rules
* Add common util to get AWS account ID
Fixes#31383
* Fix protocol number and add separate tests for egress rule handling
* Return egress rule treatment to be backwards compatible
* Remove functions that were obsoleted by `Rule` namedtuple
* IP tests
* Move description updates to a function
* Fix string formatting missing index
* Add tests for auto-creation of the same group in quick succession
* Resolve use of brand-new group in a rule without a description
* Clean up duplicated get-security-group function
* Add reverse cleanup in case of dependency issues
* Add crossaccount ELB group support
* Deal with non-STS calls to account API
* Add filtering of owner IDs that match the current account
* Stabilize ec2_vpc_vgw and ec2_vpc_vpn so tests for ec2_vpc_vpn_facts in PR 35983 can be run in CI
* Add updated placebo recordings
* ensure find_vgw uses the virtual gateway id if available
Add AWSRetry.jittered_backoff to attach_vpn_gateway to deal with errors when attaching a new VPC directly after detaching
Add integrations tests for ec2_vpc_vgw
* Sort VPN Gateways by ID
Start using this to construct shade OpenStack Cloud objects in a
consistent manner. This will let us centralize things like dealing with
password arguments and whatnot. It also allows us to introduce the
ability to pass a fully formed config dict directly to the module.
Migrate all OpenStack modules to use openstack_cloud_from_module.
Have it return the shade library since it's responsible for
importing shade and shade is needed for the exceptions.
Only pull specific OpenStack arguments for the constructor
Rather than passing **module.params to the shade constructor, pull out
only the values that make sense. This should prevent the issues with
module parameters stepping on shade parameters.
Replace module.params.pop with module.params.get
We don't need to pop these anymore since the shade constructor is now
using opt-in values.
Using real urls is ungood. Use example.com domains. Also, get rid of the
antiquated port numbers.
Enforce module deprecation.
After module has reached the end of it's deprecation cycle we will replace it with a docs stub.
* Replace deprecated modules with docs-only sub
* Use of deprecated past deprecation cycle gives meaningful message (see examples below)
* Enforce documentation.deprecation dict via `schema.py`
* Update `ansible-doc` and web docs to display documentation.deprecation
* Document that structure in `dev_guide`
* Ensure that all modules starting with `_` have a `deprecation:` block
* Ensure `deprecation:` block is only used on modules that start with `_`
* `removed_in` A string which represents when this module needs **deleting**
* CHANGELOG.md and porting_guide_2.5.rst list removed modules as well as alternatives
* CHANGELOG.md links to porting guide index
To ensure that meaningful messages are given to the user if they try to use a module at the end of it's deprecation cycle we enforce the module to contain:
```python
if __name__ == '__main__':
removed_module()
```
* Moved the encryption to its own action method.
* removed silly default value for encryption type.
* Code formatting issues from pull request ANSIBOT.
* changed version_added to "2.5"
because of new new options available
* changed version_added to "2.5"
because of new new options available
* changed version_added to "2.5"
because of new new options available
* move set_module_args to units.modules.utils
* unit tests: reuse set_module_args
* unit tests: mock exit/fail_json in module.utils.ModuleTestCase
* unit tests: use module.utils.ModuleTestCase
* unit tests: fix 'import shadowed by loop variable'
* When getting the stack events we need to consider the case where we don't have ClientRequestToken fixes#32396
* Adding tests for the case when the ClientRequestToken is not present in the stack creation.
* Renaming the stack that the test for Client Request Token requires so it won't cause collisions with the basic test.
* Start using ClientRequestTokens in event lists
* Include request token in all reqs that support it (basically all but check mode/changeset)
* Update placebo recordings
* Add comments for CRQ popping
* Add the ability to modify shard count to kinesis_stream module
* Fixed an issue in kinesis_stream where update() reports not changed when it is changed
* Remove unreachable message and make the try and catch block shorter
- Fixes to lambda
- reformatting + tests for lambda_facts
- lambda module integration test
- switch lambda and lambda_facts to AnsibleAwsModule
- Get the account ID from STS, GetUser, and finally error message
* Add VPN module and unittests
* remove unnecessary imports
* fix documentation
* raise custom exception rather than passing module everywhere
* remove recordings
Rerecord tests
Fix docs
Ensure vpn_connection_id is a list of strings when checking if it exists
* fix check mode
* Rerecord tests
* remove superfluous code and comments and make exception handling uniform
fix docs
* Fix ec2_vpc_vpn documentation
* make ec2_vpc_vpn compatible with python 2.6 and make check mode logic more succinct
* fix comparison of list of dicts
* Fix typos and docstrings
make requested changes for imports
make code clearer
* Fix copyright and metadata version.
* New module for AWS Direct Connect connections
Unittests for new module
Added utils that will be used by other Direct Connect modules
* pep8 fixes
* Correct aws_direct_connect_connection EXAMPLES to be valid yaml
* Make requested changes
* remove use of the variable changed
get rid of unnecessary parentheses
* Fix unit test
* Rename variable.
* Fix metadata version
* Add module_utils/aws/direct_connect.py for frequently used functions
* new AWS Direct Connect link aggregation group module with tests and placebo recordings
* remove extra argument
* Remove use of undefined var
* Fix param name for extra exception codes for AWSRetry to use.
* Fix undefined var and line length and metadata version number
* Fix copyright headers
* Ansible Config part2
- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code
skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution
* killed extra space with extreeme prejudice
cause its an affront against all that is holy that 2 spaces touch each other!
shippable timing out on some images, but merging as it passes most
* replace boto with boto3 for the s3 module
make s3 pep8 and remove from legacy files
fix s3 unit tests
* fix indentation
* s3 module - if we can't create an MD5 sum return None and always upload file
* remove Location.DEFAULT which isn't used in boto3 and tidy up the docs
* pep8
* s3: remove default: null, empty aliases, and required: false from documentation
fix incorrectly documented defaults
* Porting s3 to boto3. Simplify some logic and remove unused imports
* Fix s3 module variables
* Fix a typo in s3 module and remove from pep8 legacy files
* s3: add pagination for listing objects.
Fix logic and use head_object instead of get_object for efficiency.
Fix typo in unit test.
* Fix pagination to maintain backwards compatibility.
Fix incorrect conditional.
Remove redundant variable assignment.
Fix s3 list_object pagination to return all pages
* Use the revised List Objects API as recommended.
* Wrap call to paginated_list in a try/except
Also remembered to allow marker/prefix/max_keys to modify what keys are listed
* Simplify argument
* s3_bucket: fix policy sorting for python3 so strings are evaluated as less than tuples.
Add tests to ensure this behavior is maintained.
* Fix s3_bucket comparison function to work on both Python 3.5 and 3.6
* s3_bucket: document that cmp_to_key is used for python 2.7.
Add another test for s3_bucket to compare policies of different sizes.
* fix pep8
* Work around code-smell grepping by not using the word 'cmp'.
* New module for managing AWS Datapipelines
* Supports create/activate/deactivate and deletion
* Handles idempotent creation by embeding the version in the
uniqueId field
* Waits for requested state to be reached, as Botocore doesn't
have waiters yet for datapipelines
* rename module, fix imports, add tags option, improve exit_json results, fix a couple bugs, add a TODO so I don't forget
fix pep8
allow timeout to be used for pipeline creation
make .format syntax uniform
fix pep8
fix exception handling
allow pipeline to be modified, refactor, add some comments, remove unnecessary imports
pipeline activation may not be in the activated state long
remove datapipeline version option
change a loop to a list comprehension
create idempotence by hashing the options given to the module minus the objects (which can be modified)
small bugfix
* data_pipeline unittests
make unittests pep8
fix bug in unittests
* remove exception handling that serves no purpose
* Fix python3 incompatibilities in datapipeline tests and add placebo fixture maybe_sleep for faster tests
Fix python3 incompatibilities in data_pipeline build_unique_id()
Don't delete a pipeline in diff_pipeline() because it's unexpected
Don't use time.time() because it causes an issue with placebo testing
re-recorded tests
fix pep8 in data_pipeline
Remove disable_rollback from tests
Make sure unique identifier is a string
re-record tests
* improve documentation and add another example
* use a placebo fixture instead of redundant code in tests
fix tests for PLACEBO_RECORD=false
* Fix data_pipeline docs
use isinstance instead of type()
fix documentation
* fix documentation
* Remove use of undefined variable from data_pipeline module and fix license
* fix copyright header
* Ultra basic api-gateway module based of lambda.py
* Ultra basic deployment added to api-gateway module
* ApiGateway module Allow creation of APIs, more documentation and better return value
* ApiGateway module incorporate review feedback
* ApiGateway module flake8 cleanup
* APIGateway module - more review fixes.
* slightly better messages in api_gateway module
* AWS api_gateway module - try to improve messages in case of exceptions
* rename api_gateway module to aws_api_gateway as discussed in PR 20230
* aws_api_gateway - Allow delivery of swagger either as text or dictionary.
* aws_api_gateway module - introduce 'unit' tests, improve imports using them and small fixes
* aws_api_gateway module - move path expand_user to avoid early typecheck
* aws_api_gateway - version means version of metadata not module - fix to 1.0
* aws_api_gateway module - Rely on module_utils.ec2 for imports & path type for expanduser / cleanups
* aws_api_gateway module - heavy cleanup and refactor of code + cloud retry functionality.
* api_gateway_module - failing test case for handling more than one deployment in succession and API deletion
* add TooManyRequestsException to AWSRetry exception list - makes API deployment work.
* api_gateway_module - Fixes for various review comments + errors from various linters
* api_gateway_module - Fixes for more review comments + linter error
* api_gateway_module - Major refactor into sensible functions - create_response becomes configure_response
* api_gateway_module - should be working under python3; remove test exclusion
* api_gateway_module - finish off remaining review fixes - use ansible defaults and fix mutually exclusive
* api_gateway_module - attempt to improve handling of botocore errors in python3
* api_gateway_module - implement state=absent / API deletion
* test/: PEP8 compliancy
- Make PEP8 compliant
* Python3 chokes on casting int to bytes (#24952)
But if we tell the formatter that the var is a number, it works
* draft new inventory plugin arch, yaml sample
- split classes, moved out of init
- extra debug statements
- allow mulitple invenotry files
- dont add hosts more than once
- simplified host vars
- since now we can have multiple, inventory_dir/file needs to be per host
- ported yaml/script/ini/virtualbox plugins, dir is 'built in manager'
- centralized localhost handling
- added plugin docs
- leaner meaner inventory (split to data + manager)
- moved noop vars plugin
- added 'postprocessing' inventory plugins
- fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them
- grouphost_vars loading as inventory plugin (postprocessing)
- playbook_dir allways full path
- use bytes for file operations
- better handling of empty/null sources
- added test target that skips networking modules
- now var manager loads play group/host_vars independant from inventory
- centralized play setup repeat code
- updated changelog with inv features
- asperioribus verbis spatium album
- fixed dataloader to new sig
- made yaml plugin more resistant to bad data
- nicer error msgs
- fixed undeclared group detection
- fixed 'ungrouping'
- docs updated s/INI/file/ as its not only format
- made behaviour of var merge a toggle
- made 'source over group' path follow existing rule for var precedence
- updated add_host/group from strategy
- made host_list a plugin and added it to defaults
- added advanced_host_list as example variation
- refactored 'display' to be availbe by default in class inheritance
- optimized implicit handling as per @pilou's feedback
- removed unused code and tests
- added inventory cache and vbox plugin now uses it
- added _compose method for variable expressions in plugins
- vbox plugin now uses 'compose'
- require yaml extension for yaml
- fix for plugin loader to always add original_path, even when not using all()
- fix py3 issues
- added --inventory as clearer option
- return name when stringifying host objects
- ajdust checks to code moving
* reworked vars and vars precedence
- vars plugins now load group/host_vars dirs
- precedence for host vars is now configurable
- vars_plugins been reworked
- removed unused vars cache
- removed _gathered_facts as we are not keeping info in host anymore
- cleaned up tests
- fixed ansible-pull to work with new inventory
- removed version added notation to please rst check
- inventory in config relative to config
- ensures full paths on passed inventories
* implicit localhost connection local
Ensure newly created NAT gateways get converted to snake dict
Remove custom code, and associated test, for generating snake
dict and use `camel_dict_to_snake_dict`
Make use of `required_if` rather than bespoke parameter checks
Remove ec2_vpc_nat_gateway from pep8 legacy files list
* [GCP] UrlMap module
This module provides support for UrlMaps on Google Cloud Platform. UrlMaps allow users to segment requests by hostname and path and direct those requests to Backend Services.
UrlMaps are a powerful and necessary part of HTTP(S) Global Load Balancing on Google Cloud Platform.
UrlMap takes advantage of the python-api so the appropriate infrastructure has been added to module_utils.
More about UrlMaps can be found at:
https://cloud.google.com/compute/docs/load-balancing/http/url-map
UrlMap API:
https://cloud.google.com/compute/docs/reference/latest/
Google Cloud Platform HTTP(S) Cross-Region Load Balancer:
https://cloud.google.com/compute/docs/load-balancing/http/
* updated documentation, remmoved parens
* fixed tabs
* [cloud][tests] Create fixtures for using placebo to test boto3-using modules
* Use pytest's importorskip instead of manually skipping on missing deps
* Fix imports in cloudformation module
* Delete unused code
* Add maybe_sleep fixtures to speed up recorded test runs
* Build basic placebo-CFN tests
* Commit placebo recordings of basic stack operations
* Add placebo to test-requires
* Allow unit tests to run regardless of environment by setting a default region
* Use explicit relative import for Python 3 compat
* Use __name__ attribute that works on Python 2 and 3
