commit 24efa310b58c431b4d888a6315d1285da918f670
Author: James Cammarata <jimi@sngx.net>
Date: Tue Dec 29 11:23:52 2015 -0500
Adding an additional test for copy exclusion
Adds a negative test for the situation when an exclusion doesn't
exist in the target to be copied.
commit 643ba054877cf042177d65e6e2958178bdd2fe88
Merge: e6ee59f 66a8f7e
Author: James Cammarata <jimi@sngx.net>
Date: Tue Dec 29 10:59:18 2015 -0500
Merge branch 'speedup' of https://github.com/chrismeyersfsu/ansible into chrismeyersfsu-speedup
commit 66a8f7e873ca90f7848e47b04d9b62aed23a45df
Author: Chris Meyers <chris.meyers.fsu@gmail.com>
Date: Mon Dec 28 09:47:00 2015 -0500
better api and tests added
* _copy_results = deepcopy for better performance
* _copy_results_exclude to deepcopy but exclude certain fields. Pop
fields that do not need to be deep copied. Re-assign popped fields
after deep copy so we don't modify the original, to be copied, object.
* _copy_results_exclude unit tests
commit 93490960ff4e75f38a7cc6f6d49f10f949f1a7da
Author: Chris Meyers <chris.meyers.fsu@gmail.com>
Date: Fri Dec 25 23:17:26 2015 -0600
remove uneeded deepcopy fields
Also adds a new var, used by the prepare_tests role, to prevent it from
deleting the temp test directory at the start of each play to avoid any
potential race conditions
* Changed parse_addresses to throw exceptions instead of passing None
* Switched callers to trap and pass through the original values.
* Added very verbose notice
* Look at deprecating this and possibly validate at plugin instead
fixes#13608
For the deps setup of integration tests, as we sometimes see odd
errors we can't reproduce, which may be related to slightly out of
date package dependencies.
At its most basic, this is nothing more than an array or hash lookup,
but when used in conjunction with map, it is very useful. For example,
while constructing an "ssh-keyscan …" command to update known_hosts on
all hosts in a group, one can get a list of IP addresses with:
groups['x']|map('extract', hostvars, 'ec2_ip_address')|list
This returns hostvars[a].ec2_ip_address, hostvars[b].ec2_ip_address, and
so on. You can even specify an array of keys for a recursive lookup, and
mix string and integer keys depending on what you're looking up:
['localhost']|map('extract', hostvars, ['vars','group_names',0])|first
== hostvars['localhost']['vars']['group_names'][0]
== 'ungrouped'
Includes documentation and tests.
Modeled after the yum tests but also tests local package installations
as fixed with PR#1256.
This depends on PRs #1256, #1261 and #1262 in ansible-modules-extra.
Looks like there are two pattern caches that need to be cleared for this to work- added the second one.
Added integration tests for add_host to prevent future regressions.
Check for database presence in a nice way, rather than
dropping the database.
Thankfully there was a syntax error in the previous
version, so no database would have been dropped.
There was no check for whether it succeeded or not.
also remove condition to bypass setting user if user matches current user
this enables forcing user when set to the same user as current user and ignoring .ssh/config
while keeping .ssh/config with current user if nothing is specified.
* Fix the task_vars parameter to not default to a mutable type (dict)
* Implement invocation in the base class's run() method have each action
module call the run() method's implemention in the base class.
* Return values from the action plugins' run() method takes the return
value from the base class run() method into account so that invocation
makes its way to the output.
Fixes#12869
Revert "Remove auto-added invocation return value as it is not used by v2 and could leak sensitive data."
This reverts commit 6ce6b20268.
Remove the note that invocation was removed as we've now restored it.
Revert "keyword not in ubuntu 14.04"
This reverts commit 5c01622457.
Revert "remove invocation keyword check"
This reverts commit 5177cb3f74.
ansible-playbook now works when run with a playbook
that includes a role that includes another role
specified using csv format
Updated one of the roles used in the tests to fix
broken tests - `make test_galaxy` now works
Fixes#11486. Also addresses the problem alluded to in #10620.
This reverts commit 073f10a52a and instead
disables the failing test.
We're focusing on Python 3 support on the controller first; modules will
come later.
better error reporting on fetching errors
use scm if it exists over src
unified functions in requirements
simplified logic
added verbose to tests
cleanup code refs, unused options and dead code
moved get_opt to base class
fixes#11920fixes#12612fixes#10454
corrected output from default callback
added new tests for no_log loops
updated makefile test to check for both positive and negative occurrences of no_log
This caused an ImportError in a test module and showed up as one test
failure. Now the test module can get imported and many more tests fail
(on Python 3). Such is life. ;-)
* allow global no_log setting, no need to set at play or task level, but can be overriden by them
* allow turning off syslog only on task execution from target host (manage_syslog), overlaps with no_log functionality
* created log function for task modules to use, now we can remove all syslog references, will use systemd journal if present
* added debug flag to modules, so they can make it call new log function conditionally
* added debug logging in module's run_command
Due to the way we're now calculating delegate_to, if that value is based
on a loop variable ('item') we need to calculate all of the possible
delegated_to variables for that loop.
Fixes#12499
There doesn't appear to be anything that actually uses tmp_path in the
connection plugins so we don't need to pass that in to exec_command.
That change also means that we don't need to pass tmp_path around in
many places in the action plugins any more. there may be more cleanup
that can be done there as well (the action plugin's public run() method
takes tmp as a keyword arg but that may not be necessary).
As a sideeffect of this patch, some potential problems with chmod and
the patch, assemble, copy, and template modules has been fixed (those
modules called _remote_chmod() with the wrong order for their
parameters. Removing the tmp parameter fixed them.)
On Python 2, shlex.split() raises if you pass it a unicode object with
non-ASCII characters in it. The Ansible codebase copes by explicitly
converting the string using to_bytes() before passing it to
shlex.split().
On Python 3, shlex.split() raises ('bytes' object has no attribute 'read')
if you pass a bytes object. Oops.
This commit introduces a new wrapper function, shlex_split, that
transparently performs the to_bytes/to_unicode conversions only on
Python 2.
Currently I've only converted one call site (the one that was causing a
unit test to fail on Python 3). If this approach is deemed suitable,
I'll convert them all.
There were no inventory-specific unit tests earlier, so we add a new
directory for them with some initial low-level tests of _split_pattern
with various valid and deprecated pattern strings.
* Add exception handling when running PowerShell modules to provide exception message and stack trace.
* Enable strict mode for all PowerShell modules and internal commands.
* Update common PowerShell code to fix strict mode errors.
* Fix an issue with Set-Attr where it would not replace an existing property if already set.
* Add tests for exception handling using modified win_ping modules.
1. The test did "name: '{{hostnames}}.{{item}}'" inside a with_sequence
loop, which didn't do what was intended: it expanded hostnames into
an array, appended ".1", and set name to the resulting string. This
can be converted to a simple with_items loop.
2. Some of the entries in hostnames contained punctuation characters,
which I see no reason to support in inventory hostnames anyway.
3. Once the add_host failures are fixed, the playbook later fails when
the unicode hostnames are interpolated into debug output in ssh.py
due to an encoding error. This is only one of the many places that
may fail when using unicode inventory hostnames; we work around it
by providing an ansible_ssh_host setting.
Labels must start with an alphanumeric character, may contain
alphanumeric characters or hyphens, but must not end with a hyphen.
We enforce those rules, but allow underscores wherever hyphens are
accepted, and allow alphanumeric ranges anywhere.
We relax the definition of "alphanumeric" to include Unicode characters
even though such inventory hostnames cannot be used in practice unless
an ansible_ssh_host is set for each of them.
We still don't enforce length restrictions—the fact that we have to
accept ranges makes it more complex, and it doesn't seem especially
worthwhile.
This adds a parse_address(pattern) utility function that returns
(host,port), and uses it wherever where we accept IPv4 and IPv6
addresses and hostnames (or host patterns): the inventory parser
the the add_host action plugin.
It also introduces a more extensive set of unit tests that supersedes
the old add_host unit tests (which didn't actually test add_host, but
only the parsing function).
There was code to support set literals (on Python 2.7 and newer), but it
was buggy: SAFE_NODES.union() doesn't modify SAFE_NODES in place,
instead it returns a new set object that is then silently discarded.
I added a unit test and fixed the code. I also changed the version
check to use sys.version_tuple instead of a string comparison, for
consistency with the subsequent Python 3.4 version check that I added in
the previous commit.
The full error was
======================================================================
ERROR: test_task_executor_execute (units.executor.test_task_executor.TestTaskExecutor)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/mg/src/ansible/test/units/executor/test_task_executor.py", line 252, in test_task_executor_execute
mock_action.run.return_value = dict(ansible_facts=dict())
File "/home/mg/src/ansible/lib/ansible/executor/task_executor.py", line 317, in _execute
if self._task.async > 0:
TypeError: unorderable types: MagicMock() > int()
----------------------------------------------------------------------
Experiments show that Python 2 MagicMock() > 0 is true, so I'm setting
the async property on mock_task to 1. (If I set it to 0, the test fails
anyway.)
Required some rewiring in inventory code to make sure we're using
the DataLoader class for some data file operations, which makes mocking
them much easier.
Also identified two corner cases not currently handled by the code, related
to inventory variable sources and which one "wins". Also noticed we weren't
properly merging variables from multiple group/host_var file locations
(inventory directory vs. playbook directory locations) so fixed as well.
Replace .iteritems() with six.iteritems() everywhere except in
module_utils (because there's no 'six' on the remote host). And except
in lib/ansible/galaxy/data/metadata_template.j2, because I'm not sure
six is available there.
ansible is passing unicode arond internally so we should test the same
data.
* Add a zero length test for _count_newlines and fix the zero newlines
test to have no newlines.
`assert (condition, message)` gets parsed by Python as `assert
a_two_tuple`, and a 2-element tuple is never False.
Discovered by compileall on Python 3.4, which emits a SyntaxWarning for
this common mistake.
Note that this test was broken in devel because it was really just
duplicating the AES256 test because setting v.cipher_name to 'AES'
no longer selected AES after it was de-write-whitelisted.
Now that we've removed the VaultAES encryption code, we embed static
output from an earlier version and test that we can decrypt it.
* Add exception handling when running PowerShell modules to provide exception message and stack trace.
* Enable strict mode for all PowerShell modules and internal commands.
* Update common PowerShell code to fix strict mode errors.
* Fix an issue with Set-Attr where it would not replace an existing property if already set.
* Add tests for exception handling using modified win_ping modules.
This change is similar to https://github.com/ansible/ansible/pull/10465
It extends the logic there to also support none types. Right now if you have
a '!!null' in yaml, and that var gets passed around, it will get converted to
a string.
eg. defaults/main.yml
```
ENABLE_AWESOME_FEATURE: !!null # Yaml Null
OTHER_CONFIG:
secret1: "so_secret"
secret2: "even_more_secret"
CONFIG:
hostname: "some_hostname"
features:
awesame_feature: "{{ ENABLE_AWESOME_FEATURE}}"
secrets: "{{ OTHER_CONFIG }}"
```
If you output `CONFIG` to json or yaml, the feature flag would get represented in the output
as a string instead of as a null, but secrets would get represented as a dictionary. This is
a mis-match in behaviour where some "types" are retained and others are not. This change
should fix the issue.
I also updated the template test to test for this and made the changes to v2.
Added a changelog entry specifically for the change from empty string to null as the default.
Made the null representation configurable.
It still defaults to the python NoneType but can be overriden to be an emptystring by updating
the DEFAULT_NULL_REPRESENTATION config.
- Add support for inserting module args into PowerShell modules. Fixes#11661.
- Support Windows paths containing spaces. Applies changes from #10727 to v2. Fixes#9999. Should also fixansible/ansible-modules-core#944 and ansible/ansible-modules-core#1007.
- Change how execution policy is set for running remote scripts. Applies changes from #11092 to v2. Also fixesansible/ansible-modules-core#1776.
- Use codepage 65001 (UTF-8) for WinRM connection instead of default (CP437), convert command to UTF-8 and results from UTF-8. Replaces changes from #10024. Fixes#11198.
- Close WinRM connection when task completes.
- Use win_stat, win_file and win_copy modules instead of stat, file and copy when called from within other action plugins (only when using WinRM+PowerShell).
- Unquote Windows path arguments before passing to win_stat, win_file, win_copy and slurp modules (only when using WinRM/PowerShell).
- Check for win_ping module to determine if core modules are missing (only when using WinRM/PowerShell).
- Add stdout_lines to result from running low level commands (so stdout_lines is available when using raw/script).
- Update copy action plugin to use shell functions for joining paths and checking for trailing slash.
- Update fetch action plugin to unquote source path when using Windows paths.
- Add win_copy and win_template action plugins that inherit from copy and template.
- Support running .bat and .cmd scripts using default system encoding instead of UTF-8.
- Always send PowerShell commands as base64-encoded blobs to allow for running simple PowerShell commands via raw.
- Support running modules on Windows with interpreters other than PowerShell.
- Update integration tests to support above changes and test unicode fixes.
- Add test for win_user error from ansible/ansible-modules-core#1241 (fixed by ansible/ansible-modules-core#1774).
- Add test for additional win_stat output values (implemented by ansible/ansible-modules-core#1473).
- Add test for OS architecture and name from setup.ps1 (implemented by ansible/ansible-modules-core#1100).
All WinRM integration tests pass for me with these changes.
Also making PlayContext a child class of the Playbook Base class,
which gives it access to all of the FieldAttribute code to ensure
field values are correctly typed after post_validation
Fixes#11381
* Perhaps the only precedence change, in 2.0+ variables from set_fact
will not override params to an include file, as params are expected
to be more specific than host-based variables.
* Uncommented long-form include example.
