This makes our recursive, ast.parse performance measures as fast as
pre-ziploader baseline.
Since this unittest isn't testing that the returned module data is
correct we don't need to worry about os.rename not having any module
data. Should devise a separate test for the module and caching code
* Ziploader proof of concept (jimi-c)
* Cleanups to proof of concept ziploader branch:
* python3 compatible base64 encoding
* zipfile compression (still need to enable toggling this off for
systems without zlib support in python)
* Allow non-wildcard imports (still need to make this recusrsive so that
we can have module_utils code that imports other module_utils code.)
* Better tracebacks: module filename is kept and module_utils directory
is kept so that tracebacks show the real filenames that the errors
appear in.
* Make sure we import modules that are used into the module_utils files that they are used in.
* Set ansible version in a more pythonic way for ziploader than we were doing in module replacer
* Make it possible to set the module compression as an inventory var
This may be necessary on systems where python has been compiled without
zlib compression.
* Refactoring of module_common code:
* module replacer only replaces values that make sense for that type of
file (example: don't attempt to replace python imports if we're in
a powershell module).
* Implement configurable shebang support for ziploader wrapper
* Implement client-side constants (for SELINUX_SPECIAL_FS and SYSLOG)
via environment variable.
* Remove strip_comments param as we're never going to use it (ruins line
numbering)
* Don't repeat ourselves about detecting REPLACER
* Add an easy way to debug
* Port test-module to the ziploader-aware modify_module()
* strip comments and blank lines from the wrapper so we send less over the wire.
* Comments cleanup
* Remember to output write the module line itself in powershell modules
* for line in lines strips the newlines so we have to add them back in
Before this patch, if config was ['/some/path'] then it would enter the
else block and config would be set to [].
The regression this patch fixes was introduced by 700db154.
* Unit tests exposed a problem where nested blocks did not correctly
hit rescue/always portions of parent blocks
* Cleaned up logic in PlayIterator
* Unfortunately fixing the above exposed a potential problem in the
block integration tests, where a failure in an "always" section may
always lead to a failed state and the termination of execution
beyond that point, so certain parts of the block integration test
were disabled.
- now workers passes queue to task_executor so it can send back events per item and on retry attempt
- updated result class to pass along events to strategy
- base strategy updated to forward new events to callback
- callbacks now remove 'items' on final result but process them directly when invoked per item
- new callback method to deal with retry attempt messages (also now obeys nolog)
- updated tests to match new signature of task_executor
fixes#14558fixes#14072
this was taken out in an effort to default to the user's shell but creates issues as this is not known ahead of time
and its painful to set executable and shell_type for all servers, it should only be needed for those that restrict the user
to specific shells and when /bin/sh is not available. raw and command may still bypass this by explicitly passing None.
fixes#13882
still conditional
commit 24efa310b58c431b4d888a6315d1285da918f670
Author: James Cammarata <jimi@sngx.net>
Date: Tue Dec 29 11:23:52 2015 -0500
Adding an additional test for copy exclusion
Adds a negative test for the situation when an exclusion doesn't
exist in the target to be copied.
commit 643ba054877cf042177d65e6e2958178bdd2fe88
Merge: e6ee59f 66a8f7e
Author: James Cammarata <jimi@sngx.net>
Date: Tue Dec 29 10:59:18 2015 -0500
Merge branch 'speedup' of https://github.com/chrismeyersfsu/ansible into chrismeyersfsu-speedup
commit 66a8f7e873ca90f7848e47b04d9b62aed23a45df
Author: Chris Meyers <chris.meyers.fsu@gmail.com>
Date: Mon Dec 28 09:47:00 2015 -0500
better api and tests added
* _copy_results = deepcopy for better performance
* _copy_results_exclude to deepcopy but exclude certain fields. Pop
fields that do not need to be deep copied. Re-assign popped fields
after deep copy so we don't modify the original, to be copied, object.
* _copy_results_exclude unit tests
commit 93490960ff4e75f38a7cc6f6d49f10f949f1a7da
Author: Chris Meyers <chris.meyers.fsu@gmail.com>
Date: Fri Dec 25 23:17:26 2015 -0600
remove uneeded deepcopy fields
* Changed parse_addresses to throw exceptions instead of passing None
* Switched callers to trap and pass through the original values.
* Added very verbose notice
* Look at deprecating this and possibly validate at plugin instead
fixes#13608
also remove condition to bypass setting user if user matches current user
this enables forcing user when set to the same user as current user and ignoring .ssh/config
while keeping .ssh/config with current user if nothing is specified.
* Fix the task_vars parameter to not default to a mutable type (dict)
* Implement invocation in the base class's run() method have each action
module call the run() method's implemention in the base class.
* Return values from the action plugins' run() method takes the return
value from the base class run() method into account so that invocation
makes its way to the output.
Fixes#12869
This reverts commit 073f10a52a and instead
disables the failing test.
We're focusing on Python 3 support on the controller first; modules will
come later.
This caused an ImportError in a test module and showed up as one test
failure. Now the test module can get imported and many more tests fail
(on Python 3). Such is life. ;-)
* allow global no_log setting, no need to set at play or task level, but can be overriden by them
* allow turning off syslog only on task execution from target host (manage_syslog), overlaps with no_log functionality
* created log function for task modules to use, now we can remove all syslog references, will use systemd journal if present
* added debug flag to modules, so they can make it call new log function conditionally
* added debug logging in module's run_command
Due to the way we're now calculating delegate_to, if that value is based
on a loop variable ('item') we need to calculate all of the possible
delegated_to variables for that loop.
Fixes#12499
There doesn't appear to be anything that actually uses tmp_path in the
connection plugins so we don't need to pass that in to exec_command.
That change also means that we don't need to pass tmp_path around in
many places in the action plugins any more. there may be more cleanup
that can be done there as well (the action plugin's public run() method
takes tmp as a keyword arg but that may not be necessary).
As a sideeffect of this patch, some potential problems with chmod and
the patch, assemble, copy, and template modules has been fixed (those
modules called _remote_chmod() with the wrong order for their
parameters. Removing the tmp parameter fixed them.)
On Python 2, shlex.split() raises if you pass it a unicode object with
non-ASCII characters in it. The Ansible codebase copes by explicitly
converting the string using to_bytes() before passing it to
shlex.split().
On Python 3, shlex.split() raises ('bytes' object has no attribute 'read')
if you pass a bytes object. Oops.
This commit introduces a new wrapper function, shlex_split, that
transparently performs the to_bytes/to_unicode conversions only on
Python 2.
Currently I've only converted one call site (the one that was causing a
unit test to fail on Python 3). If this approach is deemed suitable,
I'll convert them all.
There were no inventory-specific unit tests earlier, so we add a new
directory for them with some initial low-level tests of _split_pattern
with various valid and deprecated pattern strings.
Labels must start with an alphanumeric character, may contain
alphanumeric characters or hyphens, but must not end with a hyphen.
We enforce those rules, but allow underscores wherever hyphens are
accepted, and allow alphanumeric ranges anywhere.
We relax the definition of "alphanumeric" to include Unicode characters
even though such inventory hostnames cannot be used in practice unless
an ansible_ssh_host is set for each of them.
We still don't enforce length restrictions—the fact that we have to
accept ranges makes it more complex, and it doesn't seem especially
worthwhile.
This adds a parse_address(pattern) utility function that returns
(host,port), and uses it wherever where we accept IPv4 and IPv6
addresses and hostnames (or host patterns): the inventory parser
the the add_host action plugin.
It also introduces a more extensive set of unit tests that supersedes
the old add_host unit tests (which didn't actually test add_host, but
only the parsing function).
There was code to support set literals (on Python 2.7 and newer), but it
was buggy: SAFE_NODES.union() doesn't modify SAFE_NODES in place,
instead it returns a new set object that is then silently discarded.
I added a unit test and fixed the code. I also changed the version
check to use sys.version_tuple instead of a string comparison, for
consistency with the subsequent Python 3.4 version check that I added in
the previous commit.
The full error was
======================================================================
ERROR: test_task_executor_execute (units.executor.test_task_executor.TestTaskExecutor)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/mg/src/ansible/test/units/executor/test_task_executor.py", line 252, in test_task_executor_execute
mock_action.run.return_value = dict(ansible_facts=dict())
File "/home/mg/src/ansible/lib/ansible/executor/task_executor.py", line 317, in _execute
if self._task.async > 0:
TypeError: unorderable types: MagicMock() > int()
----------------------------------------------------------------------
Experiments show that Python 2 MagicMock() > 0 is true, so I'm setting
the async property on mock_task to 1. (If I set it to 0, the test fails
anyway.)
Required some rewiring in inventory code to make sure we're using
the DataLoader class for some data file operations, which makes mocking
them much easier.
Also identified two corner cases not currently handled by the code, related
to inventory variable sources and which one "wins". Also noticed we weren't
properly merging variables from multiple group/host_var file locations
(inventory directory vs. playbook directory locations) so fixed as well.
Replace .iteritems() with six.iteritems() everywhere except in
module_utils (because there's no 'six' on the remote host). And except
in lib/ansible/galaxy/data/metadata_template.j2, because I'm not sure
six is available there.
ansible is passing unicode arond internally so we should test the same
data.
* Add a zero length test for _count_newlines and fix the zero newlines
test to have no newlines.
Note that this test was broken in devel because it was really just
duplicating the AES256 test because setting v.cipher_name to 'AES'
no longer selected AES after it was de-write-whitelisted.
Now that we've removed the VaultAES encryption code, we embed static
output from an earlier version and test that we can decrypt it.
Also making PlayContext a child class of the Playbook Base class,
which gives it access to all of the FieldAttribute code to ensure
field values are correctly typed after post_validation
Fixes#11381
The --force-handlers command line argument was not correctly running
handlers on hosts which had tasks that later failed. This corrects that,
and also allows you to specify force_handlers in ansible.cfg or in a
play.
- become constants inherit existing sudo/su ones
- become command line options, marked sudo/su as deprecated and moved sudo/su passwords to runas group
- changed method signatures as privlege escalation is collapsed to become
- added tests for su and become, diabled su for lack of support in local.py
- updated playbook,play and task objects to become
- added become to runner
- added whoami test for become/sudo/su
- added home override dir for plugins
- removed useless method from ask pass
- forced become pass to always be string also uses to_bytes
- fixed fakerunner for tests
- corrected reference in synchronize action plugin
- added pfexec (needs testing)
- removed unused sudo/su in runner init
- removed deprecated info
- updated pe tests to allow to run under sudo and not need root
- normalized become options into a funciton to avoid duplication and inconsistencies
- pushed suppored list to connection classs property
- updated all connection plugins to latest 'become' pe
- includes fixes from feedback (including typos)
- added draft docs
- stub of become_exe, leaving for future v2 fixes
Python's Exception constructor already takes a `message` as a parameter,
which you can then get at by doing str(e) (e.message was deprecated).
The reason I bothered to make this change was because I was debugging
with pdb and I noticed that AnsibleErrors don't give useful information
in pdb (probably because they don't have a __repr__ method that prints
the `msg` attribute).
(Pdb) c
> /Users/marca/dev/git-repos/ansible/lib/ansible/runner/__init__.py(599)_executor()
-> msg = str(ae)
(Pdb) ae
AnsibleError()
This filter was made because I needed to create idempotent UUIDs when
installing the agent for Go (http://go.cd), which uses UUIds to
distinguish the agents from each other.
It uses a newly created Ansible namespace to distinguish UUIDs created
by Ansible from any other source. The new namespace is a random one
created by uuidgen on OSX.
- make sure it calls itself correctly, now passes same params as it recieves
- vars is reserved, changed for templatevars to avoid confustion
- forcing mustaches again since the removal broke 'listification' as per #9622
- fixes incorrectly successful tests using undefined var, now it is defined
- now returns empty list if items is None to avoid errors
Ensure that the automated scm determination for github.com
repos still copes with .tar.gz archive files.
Handling .zip archives is left as an exercise for the interested reader
* Roles can now be given a friendly name as third field in role spec csv
* Roles can be installed from URL (not just from archived SCMs)
* Integration tests to demonstrate this
* Unit tests to ensure that role spec parsing works as expected
Addresses multiple issues when using su on freebsd including
* su prompt differs between platforms, so turned that check into a
regex comparison instead of a simple string comparison
* not using '-c' after su causes problems, so added that for all
platforms
* fixed quoting issues due to multiple uses of '-c' introduced by
the above fix
Fixes#7503Fixes#7507
Make sure all hosts and groups are unique objects
and that those are referenced uniquely everywhere.
Also fixes test_dir_inventory unit tests which were broken after previous
patches.
modified: lib/ansible/inventory/dir.py
tests issue #5749
same host defined in different groups which in turn are defined
in different ini files in an inventory directory
Conflicts:
test/units/TestInventory.py
0. Uncomment the test.
1. Test fails.
2. Make vars unique per file in test inventory files.
3. Modify token addition to not ast.literal_eval(v) a variable containing a hash.
4. Modify vars to have an escape in test inventory file.
5. Catch exceptions explicitly. Any unknown exceptions should be a bug.
6. Test passes.
tests issue #5749
same host defined in different groups which in turn are defined
in different ini files in an inventory directory
Conflicts:
test/units/TestInventory.py
It came up that fixing this unit test may relate to another ticket that is open. This work allows us to uncomment this unit test by fixing how we pars variables allowing a quoted variable to contain a '#'.
Work also went into cleaning up some of the test data to clarify what was working.
Lastly work went into cleaning up formatting so that the code is easily read.
