* Enable tests on python 3 for uri
* Added one more node type to SAFE_NODES into safe_eval module.
ast.USub represents unary operators. This is necessary for
parsing some unusual but still valid JSON files during testing
with Python 3.
* Use native yaml for apache2 test
* Test removal of default modules with force
a2enmod on debian has `-f`, but not on SUSE (runs there without force).
Therefore don't test that option on SUSE.
The docs already specify that the option is intended for Debian systems
only.
* Fix synchronize retries
The synchronize module munges its task args on every invocation of
run(). This was problematic because the munged data was not fit for use
by a second pass of the synchronize module. Correct this by using a copy
of the task args on every invocation of run() so that the original args
are not affected.
Local testing using this playbook seems to confirm that things work as
expected:
- hosts: all
tasks:
- delay: 2
register: task_result
retries: 1
until: task_result.rc == 0
synchronize:
dest: /tmp/out
mode: pull
src: /tmp/nonexistent/
fixes#18281
* Update synchroncization fixture assertions
When we started operating on a copy of the task args the test assertions
were no longer asserting things about the munged state but of the
pristine state. Convert the copy of task args to a class member so that
it can be compared against later in testing and update the assertions to
check this munged copy.
* Shuffle objects around for cleaner testing
Attach the temporary args dict to the task rather than the action as
this makes updating the existing tests cleaner.
This adds back the change to the network_cli plugin. Ths change adds
the ensure_connect decorator to the open_shell() method to make sure
the connection is valid before trying to open a shell.
The issue was due to the addition of the decorator that will call
_connect() when there is no connection. The _connect() method should
have been mocked in the test case. This commit fixes the test
case as well
Change was originally reverted in c414ded69a
* make hash_params more robust in the face of many corner cases
Fixes#18680
Alternative fix to #18681
* add test case for role.hash_params
* Add role.hash_params test for more types
A set, a generator/iterable, and a Container that
is not Iterable.
* Fix regression in jinja2 include search path
Since commit 3c39bb5, the 'ansible_search_path' variable is used to set
jinja2's search path for {% include %} directives. However, this path is
the the proper one because our templates live in 'templates' subdirs in
our search path.
This is a regression because previously, our include search path would
include the dirname of the currently interpreted file, which worked most
of the time.
fixes#18526
* Fix template lookup search path
Improve fix in commit c96c853 so that the search path contain both
template-suffixed paths as well as original paths.
ref PR #18617
* Add integration test for template lookups
Tests regression at #18526
This test fails on current devel branch and succeeds on PR #18617
* wip: add a unit test for playbook/base.py
This commit include a failing test
TestBaseSubClass.test_attr_class_post_validate
It fails with the error:
Traceback (most recent call last):
File "/home/adrian/src/ansible/test/units/playbook/test_base.py", line 264, in test_attr_class_post_validate
bsc = self._base_validate(ds)
File "/home/adrian/src/ansible/test/units/playbook/test_base.py", line 206, in _base_validate
bsc.post_validate(templar)
File "/home/adrian/src/ansible/lib/ansible/playbook/base.py", line 450, in post_validate
" Error was: %s" % (name, value, attribute.isa, e), obj=self.get_ds())
AnsibleParserError: the field 'test_attr_class_post_validate' has an invalid value (<class 'units.playbook.test_base.ExampleSubClass'>), and could not be converted to an class. Error was: test_attr_class_post_validate is not a valid <class 'units.playbook.test_base.ExampleSubClass'> (got a <class 'ansible.playbook.base.BaseMeta'> instead)
* wip, test refactoring
* wip, trying to add a parent->child
* wip, fix isa=class.
the ds the base using needs an instance of the class
(ie, whats normally created by the yaml loaders)
* wip, theres no need to argue, I just dont understand parents
* stub a _preprocess_data for coverage
* cleanup, required, parent, etc
* Make sure include_role inherit variables from parent role
Setting the parent of task blocks generated by include_role after they
have been produced is not sufficient - it means the tasks don't have the
correct dependency chain set afterwards, and therefore, don't properly
inherit variables from outer roles.
In addition to manually setting the parents, pass the dep_chain when
compiling the role, such that variables are correctly imported.
Fixes#18540.
* Add tests for include_role
* Fix include_role variable inheritance for multiple parent levels
* Add test cases for VyOS commands that don't honor paging settings
Testing for issue fixed in PR #18546
* Add provider line and fix indentation
For the way we invoke the tests we need to specify the `provider:`
Also fix the indentation on `register:`
* Replace pipes.quote for shlex_quote
* More migration of pipes.quote to shlex_quote
Note that we cannot yet move module code over. Modules have six-1.4
bundled which does not have shlex_quote. This shouldn't be a problem as
the function is still importable from pipes.quote. It's just that this
has become an implementation detail that makes us want to import from
shlex instead.
Once we get rid of the python2.4 dependency we can update to a newer
version of bundled six module-side and then we're free to use
shlex_quote everywhere.
If the current ansible enviroment has a config setup
that doesn't use 'smart' as the configured transport
test_play_context would fail when it assumes the
transport will be 'smart'.
Previous changes addressed a corner case, which unfortunately introduced
another bug. This patch adds a new flag to the host state (did_rescue) which
is set to true when the rescue portion of a block completes. This flag is
then checked in _check_failed_state() when the fail_state != FAILED_NONE.
This lead to the discovery of another bug - current strategies are not advancing
hosts to ITERATING_COMPLETE after doing a peek at the next task, leaving the
host state in the run_state of the final task. To address this, before gathering
the list of failed hosts in StrategyBase.run(), a final pass through the iterator
for all hosts is done to ensure each host is in its final state. This way, no
strategy derived from StrategyBase has to worry about it and it's handled.
Fixes#17983
This commit extends YAML linting by enabling standard rules from the
`yamllint` tool [1]. Since syntax errors and key duplicates are already
checked since 4d48711, this change only adds detection for cosmetic
problems. It also narrows checks to the test/ dir only.
The main goal is to prevent future problems to enter the code base
without being noticed. While it would be a huge effort to be PEP8
compliant, it is relatively easy to have correct YAML style *now* and
prevent future errors by enabling linting.
Note: for those (like me) caring about code attribution: use `git blame
-w` to ignore whitespace-only changes.
Note: I disabled some linting checks (such as indentation), they can be
enforced in the future if needed. Similarly, current checks can also be
disabled. See the `.yamllint` file.
[1]: https://yamllint.readthedocs.io/
This change corrects problems reported by the `yamllint` linter.
Since key duplication problems were removed in 4d48711, this commit
mainly fixes trailing spaces and extra empty lines at beginning/end of
files.
* Added test for sequenced-name instance generation (num_instances)
* Added param-check tags to tests that only do argument checking
Should be merged AFTER ansible/ansible-modules-core#4276
- Use assertRaisesRegexp to make sure correct exceptions are raised.
- Set docker_command to avoid docker dependency (skips find_executable).
- Use a fake path for docker_command to make sure mock.patch is working.
* Fix bug (#18355) where encrypted inventories fail
This is first part of fix for #18355
* Make DataLoader._get_file_contents return bytes
The issue #18355 is caused by a change to inventory to
stop using _get_file_contents so that it can handle text
encoding itself to better protect against harmless text
encoding errors in ini files (invalid unicode text in
comment fields).
So this makes _get_file_contents return bytes so it and other
callers can handle the to_text().
The data returned by _get_file_contents() is now a bytes object
instead of a text object. The callers of _get_file_contents() have
been updated to call to_text() themselves on the results.
Previously, the ini parser attempted to work around
ini files that potentially include non-vailid unicode
in comment lines. To do this, it stopped using
DataLoader._get_file_contents() which does the decryption of
files if vault encrypted. It didn't use that because _get_file_contents
previously did to_text() on the read data itself.
_get_file_contents() returns a bytestring now, so ini.py
can call it and still special case ini file comments when
converting to_text(). That also means encrypted inventory files
are decrypted first.
Fixes#18355
This allows validate-modules to run in an environment where
python 3 is the default. This will no longer be necessary once
validate-modules is updated to work with both python 2 and 3.
- Remove shebangs from:
- ini files
- unit tests
- module_utils
- plugins
- module_docs_fragments
- non-executable Makefiles
- Change non-modules from '/usr/bin/python' to '/usr/bin/env python'.
- Change '/bin/env' to '/usr/bin/env'.
Also removed main functions from unit tests (since they no longer
have a shebang) and fixed a python 3 compatibility issue with
update_bundled.py so it does not need to specify a python 2 shebang.
A script was added to check for unexpected shebangs in files.
This script is run during CI on Shippable.
- Update import for relocated tests.
- Fix test to expect changed from update_tags.
- Add checks for boto3 and botocore to tests.
- Set check mode with kwarg.
- Python 3 fixes for unit tests.
- Python 2.6 fix for unit tests.
- Add missing meta value for test_create_server
- Add .gitignore for pytest .cache directory
Exclude test_os_server from nose test runs since it was designed
for pytest. The test will work correctly when run using pytest.
This is a temporary issue, as we'll be moving to pytest soon.
This commit adds some unit tests for the `cloud.openstack.os_server`
module. These tests exercise `_network_args` thoroughly and
`_create_server` lightly.
These tests will **fail** until ansible/ansible-modules-core#2275 lands.
To run the tests:
pip install -r test-requirements.txt
PYTHONPATH=$PWD py.test
Originally from ansible/ansible-modules-core@3387526bca
- Correct directory name in test/README.md
- Move code-smell tests to test/sanity/code-smell
- Update code-smell.sh to use new script paths
- Add test/integration/target-prefixes.win for ansible-test
- Move module unit tests to match module directory layout
* Network Test Documentation
Will need improving over time, though this ensure that everything that was in `ansible/test-network-modules` is in `ansible/ansible`
* Update README.md
* Inventory file
* Network module prefixes
In ansible-test we should skip tests for these modules, they will be
tested via another process.
* Update target-prefixes.network
In order to support legacy plugins, the following two method signatures
are allowed for `CallbackBase.v2_playbook_on_start`:
def v2_playbook_on_start(self):
def v2_playbook_on_start(self, playbook):
Previously, the logic to handle this divergence checked to see if the
callback plugin being called supported an argument named `playbook`
in its `v2_playbook_on_start` method. This was fragile in a few ways:
- if a plugin author did not use the literal `playbook` to name their
method argument, their plugin would not be called correctly
- if a plugin author wrapped their `v2_playbook_on_start` method and
by doing so changed the argspec to no longer expose an argument
with that literal name, their plugin would not be called correctly
In order to continue to support both types of callback for backwards
compatibility while making the call more robust for plugin authors,
the logic can be reversed in order to have a positive check for the old
method signature instead of a positive check for the new one.
Signed-off-by: Steve Kuznetsov <skuznets@redhat.com>
As neon is derived from Ubuntu, ansible_os_family should have the value
"Debian" instead of "Neon". Add a test case for KDE neon and set
os_family correctly for it.
If the facts returned by setup included strings that
had double quotes in them, the asserts in test_gathering_facts.yml
would fail with errors like:
"The conditional check '\"[{u'mounts': {u'options':
u'rw,context=\"system_u:\"'}}]\" != \"UNDEF_HW\"' failed. The error was:
template error while templating string: expected token 'end of statement
block', got 'system_u'. String: {% if \"[{u'mounts': {u'options':
u'rw,context=\"system_u:\"'}}]\" != \"UNDEF_HW\" %} True {% else %}
False {% endif %}"
For one example, if mount facts returned an 'options' field that
included double quoated selinux context ids, the test would fail.
Fix is removing the double quoting in the assert 'that:' lines,
and removing the unneeded double curly brackets.
Nothing seems to use this now.
Was added originally added in2d11cfab92f9d26448461b4bc81f466d1910a15e
but the code that used it was removed in
e02b98274b
On openSUSE Tumbleweed, lsb-release -a currently reports
the distributor ID as "openSUSE Tumbleweed". On openSUSE
Leap, the distributor ID is "SUSE LINUX".
Add them to the OS_FAMILY dict as Suse family systems.
Also add an entry to TESTSETS in test_distribution_version.py
for openSUSE Tumbleweed.
Since passlib algo sometime takes a bytes, and sometime
not, depending on a internal variable, we have to convert
bnased on it, or it fail with "TypeError: salt must be bytes,
not str" (or unicode instead of bytes)
However, that's not great to use internal structure for that.
* Add tag verification test (ansible-modules-core PR 2654)
* Fix typo
* Use smaller repo for testing, add dependency control
* Test is gpg exists before running git signing tasks
* Correct the test conditionals so that gpg1 is tested
Currently (pre-repomerge) we aren't running sanity.sh from
ansible/ansible, after the merge we will. Therefore I've added the
requirements here, rather than in ansible-modules-*/test/utils/shippable
Implement tag and skip_tag handling in the CLI() class. Change tag and
skip_tag command line options to be accepted multiple times on the CLI
and add them together rather than overwrite.
* Make it configurable whether to merge or overwrite multiple --tags arguments
* Make the base CLI class an abstractbaseclass so we can implement
functionality in parse() but still make subclasses implement it.
* Deprecate the overwrite feature of --tags with a message that the
default will change in 2.4 and go away in 2.5.
* Add documentation for merge_multiple_cli_flags
* Fix galaxy search so its tags argument does not conflict with generic tags
* Unit tests and more integration tests for tags
Prior to this commit, the ini parser would fail if the inventory was
not 100% utf-8. This commit makes this slightly more robust by
omitting full line comments from that requirement.
Fixes#17593
* Remove unicode-escape which is not present on python3
Alternative fix for #17305
* Enable the assemble test on python3
* Fix other problems with assemble on python3
If the sftp fails, roll over to scp by default. This saves users
from having to know about the scp_if_ssh method when sftp is broken
on the remote host.
* Turn mount test back on
* Mount tests need PRIVILEGED so turn that back on
Revert "Revert "Set PRIVILEGED=true for non_destructive tests. (#17733)" (#17738)"
This reverts commit dc0fb1c212.
* Add a needs_privileged tag so that we can skip mount tests on centos6
Some containers timeout on shippable tests when run with privileged.
Unfortunately, some tests require that in order to run. Tagging those
allows us to skip those tests on the platforms that timeout when we get
ready to run the integration test in shippable.
* Centos6 times out with PRIVILEGED set so remove that (will disable the mount tests on centos6)
* Remove false start
Python2 seems to allow any integer. Python3.5 on Linux seems to allow
a 32 bit unsigned int. Python3.5 on El Capitan seems to limit it to
a smaller size... perhaps a 16 bit int.
This adds some test data to test_facts.py that
includes mnt points that have a single quote in
the path.
Ala, https://github.com/ansible/ansible/issues/16855
The bug was already fixed via other changes, but this is
for regression testing.
* Add tests for the mount module
* Switch from unmounted to absent...
the code for mounting always modifies fstab so we need to always modify
fstab to cleanup as well.
* Fix comments and copyright
A parameter of type int should accept int and string, but not float.
A parameter of type float should accept float, int, and string.
Also reset the arguments in another test so that it runs cleanly. This
agrees with what all the other tests are doing.
Includes:
* Tests to create, delete, resize, recreate and configure an Autoscaler
* Tests for parameters only. Use TEST_FLAGS="--tags 'param-check'" to execute only these tests.
Requires:
* An instance template to be configured in your project.
* Improve unit testing of 'password' lookup
The tests showed some UnicodeErrors for the
cases where the 'chars' param include unicode,
causing the 'getattr(string, c, c)' to fail.
So the candidate char generation code try/excepts
UnicodeErrors there now.
Some refactoring of the password.py module to make
it easier to test, and some new tests that cover more
of the password and salt generation.
* More refactoring and fixes.
* manual merge of text enc fixes from pr17475
* moving methods to module scope
* more refactoring
* A few more text encoding fixes/merges
* remove now unused code
* Add test cases and data for _gen_candidate_chars
* more test coverage for password lookup
* wip
* More text encoding fixes and test coverage
* cleanups
* reenable text_type assert
* Remove unneeded conditional in _random_password
* Add docstring for _gen_candidate_chars
* remove redundant to_text and list comphenesion
* Move set of 'chars' default in _random_password
on py2, C.DEFAULT_PASSWORD_CHARS is a regular str
type, so the assert here fails. Move setting the
default into the method and to_text(DEFAULT_PASSWORD_CHARS)
if it's needed.
* combine _random_password and _gen_password
* s/_create_password_file/_create_password_file_dir
* native strings for exception msgs
* move password to_text to _read_password_file
* move to_bytes(content) to _write_password_file
* add more test assertions about genned pw's
* Some cleanups to alikins and abadger's password lookup refactoring:
* Make DEFAULT_PASSWORD_CHARS into a text string in constants.py
- Move this into the nonconfigurable section of constants.
* Make utils.encrypt.do_encrypt() return a text string because all the
hashes in passlib should be returning ascii-only strings and they are
text strings in python3.
* Make the split up of functions more sane:
- Don't split such that conditionals have to occur in two separate functions.
- Don't go overboard: Good to split file system manipulation from parsing
but we don't need to do every file manipulation in a separate
function.
- Don't split so that creation of the password store happens in two
parts.
- Don't split in such a way that no decisions are made in run.
* Organize functions by when it gets called from run().
* Run all potential characters through the gen_candidate_chars function
because it does both normalization and validation.
* docstrings for functions
* Change when we store salt slightly. Store it whenever it was already
present in the file as well as when encrypt is requested. This will
head of potential idempotence bugs where a user has two playbook tasks
using the same password and in one they need it encrypted but in the
other they need it plaintext.
* Reorganize tests to follow the order of the functions so it's easier
to figure out if/where a function has been tested.
* Add tests for the functions that read and write the password file.
* Add tests of run() when the password has already been created.
* Test coverage currently at 100%
As suggested in feedback on
https://github.com/ansible/ansible/pull/17575, add
os_family to test_distribution_version. Add the
correct os_family to the existing testcase data
entries.
Also add os_family to the output of
gen_distribution_version_testcase.py so any new
generated entries will contain this data.
* Make is_encrypted_file handle both files opened in text and binary mode
On python3, by default files are opened in text mode. Since we know
the encoding of vault files (and especially the header which is the
first set of bytes) we can decide whether the file is an encrypted
vault file in either case.
* Fix is_encrypted_file not resetting the file position
* Update is_encrypted_file to check that all the data in the file is ascii
* For is_encrypted_file(), add start_pos and count parameters
This allows callers to specify reading vaulttext from the middle of
a file if necessary.
* Combine VaultLib.encrypt() and VaultLib.encrypt_bytestring()
* Change vault's is_encrypted() to take either text or byte strings and to return False if any part of the data is non-ascii.
* Remove unnecessary use of six.b
* Vault Cipher: mark a few methods as private.
* VaultAES256._is_equal throws a TypeError if given non byte strings
* Make VaultAES256 methods that don't need self staticmethods and classmethods
* Mark VaultAES and is_encrypted as deprecated
* Get rid of VaultFile (unused and feature implemented in a different way)
* Normalize variable and parameter names on plaintext, ciphertext, vaulttext
* Normalize variable and parameter names on "b_" prefix when dealing with bytes
* Test changes:
* Remove redundant tests( both checking the same byte string)
* Fix use of format string without format operator
* Enable vault editor tests on python3
* Initialize the vault_cipher for VaultAES256 testing in setUp()
* Make assertTrue and assertFalse take the actual method calls for
better error messages.
* Test that non-ascii byte strings compare correctly.
* Test that unicode strings and ints raise TypeError
* Test-specific:
* Removed test_methods_exist(). We only have one VaultLib so the
implementation is the assurance that the methods exist. (Can use an abc for
this if it changes).
* Add tests for both byte string and text string input where the API takes either.
* Convert "assert" to unittest assert functions or add a custom message where
that will make failures easier to debug.
* Move instantiating the VaultLib into setUp().
* Added aws_retry decorator function with unit tests
* Restructured the code to be used with a base class.
This base class CloudRetry can be reused by any other cloud provider.
This decorator should be used in situations, where you need to implement
a backoff algorithm and want to retry based on the status code from the
exception.
* updated documentation
* fixed tabs
* added botocore and boto3 to requirements.txt
* removed cloud.py from py24 tests, as it depends on boto3
* fix relative imports
* updated test to be 2.6 compat
* updated method name from retry to backoff
* readded lxd
* Updated default backoff from 2 seconds to 1.1s.
This will be about a total of 48 seconds in 10 tries. This is
configurable.
* Fixes to the controller text model
* Change command line args to text type
* Make display replace undecodable bytes with replacement chars. This
is only a problem on pyhton3 where surrogates can enter into the msg
but sys.stdout doesn't know how to handle them.
* Remove a deprecated playbook syntax in unicode.yml
* Fix up run_cmd to change its parameters to byte string at appropriate times.
* Enable more integration tests for python 3.
* Split out python 3 integration tests.
Now that we're running more integration tests on python 3, the
tests are taking long enough that they warrant splitting out in
the same way the python 2 tests are split.
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
The test_async test target was updated to accommodate changes in
output buffering behavior in python 3. This change in behavior
may need to be addressed in the future.
* Fix paramiko's exec_command() to return bytes on python3
* Run test_connection for python3 now too
* Fix atomic_move for problem in shippable's testing
* Python-2.4 needs to use b()
for `VariableManager._get_magic_variables()`.
This saves a lot of time re-iterating the nearly always constant global
list of groups and their members.
Generate once and cache, and invalidate cache in case `add_host:` or
`group_by:` are used.
* Port set_*_if_different functions to python3
* Add surrogate_or_strict and surrogate_or_replace error handlers for
to_text, to_bytes, to_native
* Set default error handler to surrogate_or_replace
* Make use of the new error handlers in the already ported code
* Move the unittests for module_utils._text as they aren't in basic.py
* Cleanup around SEQUENCETYPE. On python2.6+ SEQUENCETYPE includes
strings so make sure code omits those explicitly if necessary
* Allow arg_spec aliases to be other sequence types
* New features for include_vars
include_vars.py now allows you to include an entire directory and its nested directories of variable files.
Added Features..
* Ignore by default *.md, *.py, and *.pyc
* Ignore any list of files.
* Only include files nested by depth (default=unlimited)
* Match only files matching (valid regex)
* Sort files alphabetically and load in that order.
* Sort directories alphabetically and load in that order.
```
- include_vars: 'vars/all.yml'
- name: include all.yml
include_vars:
file: 'vars/all.yml'
- name: include all yml files in vars/all and all nested directories
include_vars:
dir: 'vars/all'
- name: include all yml files in vars/all and all nested directories and save the output in test.
include_vars:
dir: 'vars/all'
name: test
- name: include all yml files in vars/services
include_vars:
dir: 'vars/services'
depth: 1
- name: include only bastion.yml files
include_vars:
dir: 'vars'
files_matching: 'bastion.yml'
- name: include only all yml files exception bastion.yml
include_vars:
dir: 'vars'
ignore_files: 'bastion.yml'
```
* Added whitelist for file extensisions (yaml, yml, json)
* Removed unit tests in favor of integration tests
While trying to fix the test suite on python3, I noticed
this test fail due to to_json adding more whitespace in
python3 than in python2. So -w should ignored those
differences.
* Fix to_native call in selinux_context and selinux_default_context to
use the error handler correctly.
* Port set_mode_if_different to work on python3
* Port atomic_move to work on python3
* Fix check_password_prompt variable which wasn't renamed properly
tempfile.NamedTemporaryFile keeps a file handle causing os.rename() to fail with windows based vboxfs: [Errno 26] Text file busy.
Changed NamedTemporaryFile to mkstemp() and added a finally block to unlink the temp file in each and every case.
* run_command needed a bit of tweaking to its string handling of
arguments.
* The run_command change fixes the last bit of lineinfile so we can
enable its tests
- Fix octal formatting of file mode in module response on py3.
- Convert file path to unicode in copy action.
- Enable file and copy module tests for py3 now that they pass.
Make some python3 fixes to make the unittests pass:
* galaxy imports
* dictionary iteration in role requirements
* swap_stdout helper for unittests
* Normalize to text string in a facts.py function
Fixes for these are either rewriting to get rid of the need for the
functions or using six.moves to get equivalent functions for both
python2 and python3
Make !vault-encrypted create a AnsibleVaultUnicode
yaml object that can be used as a regular string object.
This allows a playbook to include a encrypted vault
blob for the value of a yaml variable. A 'secret_password'
variable can have it's value encrypted instead of having
to vault encrypt an entire vars file.
Add __ENCRYPTED__ to the vault yaml types so
template.Template can treat it similar
to __UNSAFE__ flags.
vault.VaultLib api changes:
- Split VaultLib.encrypt to encrypt and encrypt_bytestring
- VaultLib.encrypt() previously accepted the plaintext data
as either a byte string or a unicode string.
Doing the right thing based on the input type would fail
on py3 if given a arg of type 'bytes'. To simplify the
API, vaultlib.encrypt() now assumes input plaintext is a
py2 unicode or py3 str. It will encode to utf-8 then call
the new encrypt_bytestring(). The new methods are less
ambiguous.
- moved VaultLib.is_encrypted logic to vault module scope
and split to is_encrypted() and is_encrypted_file().
Add a test/unit/mock/yaml_helper.py
It has some helpers for testing parsing/yaml
Integration tests added as roles test_vault and test_vault_embedded
* Give native strings to selinux library functions.
SELinux takes pathnames as native strings. That means we need to
convert to bytes on python2 and convert to text on python3.
Fixes#17155
* Read kitchen documentation, make module_utils params more like kitchen API
* Remove none nonstring strategy and add strict
* Raise TypeError on invalid nonstring strategy
* Document to_native()
* Make unittests for testing module_utils.text
* Rm py2.7+ code in docker connection plugin
The docker connection plugin was using subprocess.check_output
which only exists in python 2.7 and later. Connection plugins
need to support python2.6 so this replaces it with Popen/communicate()
* Handle docker ver errors in docker connection
Add unit tests for DockerConnection
Fixes#16971
test/units/plugins/action/test_action.py had code
for handling a bug in python 3.4's mock_open that
causes errors when reading binary data.
Moved to compat/tests/mock.py so other tests can
use it by default.
* actions/unarchive: fix unarchive from remote url
Currently unarchive from remote url does not work because the core
unarchive module was updated to support 'remote_src' [1], but the
unarchive action plugin was not updated for this. This causes failures
because the action plugin assumes it needs to copy a file to the
remote server, but in the case of downloading a file from a remote
url a local file does not exist, so an error occurs when the file is
not found.
[1] https://github.com/ansible/ansible-modules-core/commit/467516e
* test_unarchive: fix test with wrong remote_src use
The non-ascii filenames test had improperly set remote_src=yes even
though it was actually copying the file from the local machine (i.e.
the file did not already exist remotely). This test was passing
until the remote_src behavior of unarchive was fixed in 276550f.
Fixes#10779
Refactor some of the block device, mount point, and
mtab/fstab facts collection for linux for better
performance on systems with lots of block devices.
Instead of invoking 'lsblk' for every entry in mtab,
invoke it once, then map the results to mtab entries.
Change the args used for invoking 'findmnt' since the
previous combination of args conflicts, so this would
always fail on some systems depending on version.
Add test cases for facts Hardware()/Network()/Virtual() classes
__new__ method and verify they create the proper subclass based
on the platform.system() results.
Split out all the 'invoke some command and grab it's output'
bits related to linux mount paths into their own methods so
it is easier to mock them in unit tests.
Fix the DragonFly* classes that did not defined a 'platform'
class attribute. This caused FreeBSD systems to potentially
get the DragonFly* subclasses incorrectly. In practice it
didnt matter much since the DragonFly* subclasses duplicated
the FreeBSD ones. Actual DragonFly systems would end up with
the generic Hardware() etc instead of the DragonFly* classes.
Fix Hardware.__new__() on PY3, passing args to __new__
would cause "object() takes no parameters" errors. So
check for PY3 and just call __new__ without the args
See
https://hg.python.org/cpython/file/44ed0cd3dc6d/Objects/typeobject.c#l2818
for some explaination.
Instead of immediately returning a failed code (indicating a break in
the play execution), we internally 'or' that failure code with the result
(now an integer flag instead of a boolean) so that we can properly handle
the rescue/always portions of blocks and still remember that the break
condition was hit.
Fixes#16937
Rather than repeatedly searching for tasks by uuid via iterating over
all known blocks, cache the tasks when they are added to the PlayIterator
so the lookup becomes a simple key check in a dict.
* Query lookup plugin
* Add license and docstrings
* Add python3-ish imports
* Change query plugin type from lookup to filter
* Switch from dq to jsonpath_rw
* Add integration test for query filter
* Rename query filter to json_query
* Add jsonpath-rw
* Rename query filter to json_query
* Switch query implementation from jsonpath-rw to jmespath
Run setfacl/chown/chmod on each temp dir and file.
This fixes temp file permissions handling on platforms such as FreeBSD
which always return success when using find -exec. This is done by
eliminating the use of find when setting up temp files and directories.
Additionally, tests that now pass on FreeBSD have been enabled for CI.
When a task result has an empty results list, the
list should be ignored when determining the results
of `_check_key`. Here the empty list is treated the
same as a non-existent list.
This fixes a bug that manifests itself with squashed
items - namely the task result contains the correct
value for the key, but an empty results list. The
empty results list was treated as zero failures
when deciding which handler to call - so the task
show as a success in the output, but is deemed to
have failed when deciding whether to continue.
This also demonstrates a mismatch between task
result processing and play iteration.
A test is also added for this case, but it would not
have caught the bug - because the bug is really in
the display, and not the success/failure of the
task (visually the test is more accurate).
Fixesansible/ansible-modules-core#4214
This feature changes the scalar value of `serial:` to a list, which
allows users to specify a list of values, so batches can be ramped
up (commonly called "canary" setups):
- hosts: all
serial: [1, 5, 10, "100%"]
tasks:
...
Binary module tests now download pre-compiled binaries from S3
instead of downloading go and compiling the modules.
Files downloaded form SourceForge are now downloaded from S3.
* Enabled more tests in test_service on systems with systemd.
* Fix inconsistency in cleaning up test service. The conditions for service setup and service cleanup were different.
* Add check mode run for service enable.
This is a refactoring of the existing GCE utility module to support other projects on Google Cloud Platform.
The previous gce.py module was hard-coded specifically for GCE, and attempting to use it with other projects in GCP failed.
See https://github.com/ansible/ansible/pull/15918#issuecomment-220165913 for more detail.
This has also been an issue for others in the past, although they've handled it by simply
duplicating some of the logic of gce.py in their own modules.
- The existing gce.py module was renamed to gcp.py, and modified to remove any
imports or other code that refers to libcloud.compute or GCE (the GCE_* params were
retained for compatibility). I also renamed the gce_connect function to gcp_connect,
and modified the function signature to make supplying a provider, driver, and agent
information mandatory.
- A new gce.py module was created to handle connectivity to GCE. It imports the
appropriate libcloud.compute providers and drivers, and then passes them on
to gcp_connect in gcp.py. The constants and function signatures are the same
as the old gce.py, so compatibility with existing modules is retained.
- A new gcdns.py module was created to support PR ansible/ansible-modules-extras#2252
for two new Google Cloud DNS modules, and to demonstrate support for a non-GCE
Google Cloud service. It follows the same basic structure as the new gce.py module,
but imports from libcloud.dns instead.
Enable color output from integration tests in Docker containers:
* In run_tests.sh when output is attached to a terminal.
* In shippable/integration.sh using force mode (can be disabled).
Also fix blocks tests to work with or without color output
* switch cwd to basedir of task
This restores previous behaviour in pre 2.0 and allows for 'local type' plugins
and actions to have a more predictable relative path.
fixes#14489
* removed FIXME since prev commit 'fixes' this
* fix tests, now they need a loader (thanks jimi!)
* add check_mode option for tasks
includes example testcases for the template module
* extend check_mode option
* replace always_run, see also proposal rename_always_run
* rename always_run where used and add deprecation warning
* add some documentation
* have check_mode overwrite always_run
* use unique template name to prevent conflicts
test_check_mode was right before, but failed due to using the same filename as other roles
* still mention always_run in the docs
* set deprecation of always_run to version 2.4
* fix rst style
* expand documentation on per-task check mode
Adds a new test for GalaxyCLI. Tests cases when:
- no actions are provided
- invalid actions are provided
- each valid action is provided
When parser() runs successfully the method returns True, creates a
SortedOptParser instance, a Galaxy instance, and updates certain class data.
A little unittest refactoring
* Add a class decorator to generate tests when using a unittest.TestCase base class
* Add a TestCase subclass with setUp() and tearDown() that sets up
module parameter parsing
* Move test_safe_eval to use the class decorator and ModuleTestCase base
class
* Move testing of set_mode_if_different into its own file and separate
some test methods out so we get better errors and more coverage in
case of errors.
* Naming convention for test cases doesn't need to duplicate information
that's already in the file path.
* Adding a test for GalaxyCLI and two methods on which it depends. Tests execute_info method.
* Adding a test for GalaxyCLI and two methods on which it depends. Tests execute_remove method.
* Update test_galaxy.py
* Revising test for GalaxyCLI execute_remove.
* Removing mocks
I suspect this problem was masked previously as older versions of pywinrm
where significantly slower, allowing more time for the windows installer service to complete.
6eefc11c converted task.loop_control into an object, but while the other
callers were updated to use .loop_var instead of .get('loop_var'), this
site was overlooked.
This can be reproduced by including with loop_control a file that does
set_fact; a simple regression test along these lines is included.
* Lookup unencrypted password must not include salt
* Integration test lookup: remove previous directory
* Test that lookup password doesn't return salt
* Lookup password: test behavior with empty encrypt parameter
Closes#16189
- Do not update base image packages unnecessarily.
- Fix incorrect package names.
- Remove commented out RUN statements.
- Sort list of packages to install.
Due to the fact that roles may be instantiated with different sets of
params (multiple inclusions of the same role or via role dependencies),
simply tracking notified handlers by name does not work. This patch
changes the way we track handler notifications by using the handler
object itself instead of just the name, allowing for multiple internal
instances. Normally this would be bad, but we also modify the way we
search for handlers by first looking at the notifying tasks dependency
chain (ensuring that roles find their own handlers first) and then at
the main list of handlers, using the first match it finds.
This patch also modifies the way we setup the internal list of handlers,
which should allow us to correctly identify if a notified handler exists
more easily.
Fixes#15084
- Reduce image size by skipping weak package references.
- Consolidate and sort packages to install.
- Improve consistency between versions.
- Combine yum/dnf install and clean to avoid unwanted caching.
- Don't update existing packages from base image unnecessarily.
- Reduce image size by skipping recommended packages.
- Consolidate and sort packages to install.
- Improve consistency between Ubuntu versions.
- Combine apt-get update and install to avoid caching stale updates.
* add git shallow fetch test
covers https://github.com/ansible/ansible-modules-core/issues/3782
updating a repo with depth=1 fails silently if version==HEAD
* disable git shallow tests for old git versions
Older git versions don't treat the --depth option correctly.
While the git module tried to work around this and introduced subtle
bugs, ansible/ansible-modules-core#3794 falls back to full checkouts.
Don't run the tests then.
This allows the PlaybookExecutor to receive more information regarding
what happened internal to the TaskQueueManager and strategy, to determine
things like whether or not the play iteration should stop.
Fixes#15523
* Port urls.py to python3
Fixes (largely normalizing byte vs text strings) for python3
* Rework what we do with attributes that aren't set already.
* Comments
* reduce async sleep time in test
* make zypper test less destructive (don't break following uses of zypper)
* fix ca cert on suse
* fix/enable postgres/mysql on opensuseleap
* fix mysql test for mysql versions 5.7.6 and newer
* skip sni_host check on ubuntu1604
* add HTTPTESTER flag for test_uri
ubuntu 16.04 uses dash which drops env variables containing a dot
we work around this by adding an explicit env variable to enable httptester
The output of 'ansible-galaxy info' was formatting the
'galaxy_info' key with one char per line.
Previously, when building the output string, items in
role_info that had a dict for value, the label for
it's key ('galaxy_info' for ex) was being added to
the text list in addition to being appended. Only
the append is needed.
Also added a unit test in test/units/cli/test_galaxy.py,
but skip it on py3 until galaxy is py3 compatible.
fixes#15177
* Merge conflicts:
* [skip ci] Revert changes to run_tests.sh
gundalow will update this in a different PR
* [skip ci] Add in ubuntu1604 and opensuseleap
NOTE: We are not configuring anything to use these new images yet.
Therefore no impact on Travis performance
* python-mysql for opensuse
* It's mysql-server on centos6
* alias module is very basic and removing it leads to the suse default
config failing
* future improvements might test different modules and the effect of
them being removed
* When var name is the same as var content, try to template it before reporting that var is not defined.
Add asserts in test_var_blending to check this special corner case.
* Fix integration tests when using debug with list or dict.
* add tests for centos6, rhel6 and rhel7
* gen_distribution_version_testcase with python2.6
* remove unused imports
* fix redhat/vmware/... parsing
* add centos7 test case
* Make ziploader's ansible and ansible.module_utils libraries into
namespace packages.
* Move __version__ and __author__ from ansible/__init__ to
ansible/release.py. This is because namespace packages only load one
__init__.py. If that is not the __init__.py with the author and
version info then those won't be available.
* In ziplaoder, move the version ito ANSIBLE_CONSTANTS.
* Change PluginLoader to properly construct the path to the plugins even
when namespace packages are present.
* extend zypper integration tests
* fix caching issue of local RPMS on openSUSE 42.1 Leap
* add tests for simultaneous install/remove via prefixes +-
* test fail cases (rm patch or URL)
* test patch install (succes, unchanged second run, fail on wrong name)
* add test for pattern install
* include options refresh and priority introduced in ansible/ansible-modules-extras#1990
* test repo update both based on name and URL
* run on all suse flavors
* test overwriting repos by url or name
Updated python module wrapper explode method to drop 'args' file next to module.
Both execute() and excommunicate() debug methods now pass the module args via file to enable debuggers that are picky about stdin.
Updated unit tests to use a context manager for masking/restoring default streams and argv.
* add integration test for apache2_module
* fix assert syntax
* check os_family for this test
* add comment why only debian and suse are tested
* move apache2 test to destructive
* install apache2 package
- get_real_file will decrypt vault encrypted files and return a path to
a temporary file.
- cleanup_real_file will remove a temporary file created previously with
get_real_file
Ansible when there was a percentage that was calculated to be less than
1.0 would run all hosts as the value for a rolling update.
The error is due to the fact that Python will round a
float that is under 1.0 to 0, which will trigger the case of
0 hosts. The 0 host case tells ansible to run all hosts.
The fix will see if the percentage calculation after int
conversion is 0 and will else to 1 host.
This makes our recursive, ast.parse performance measures as fast as
pre-ziploader baseline.
Since this unittest isn't testing that the returned module data is
correct we don't need to worry about os.rename not having any module
data. Should devise a separate test for the module and caching code
* Ziploader proof of concept (jimi-c)
* Cleanups to proof of concept ziploader branch:
* python3 compatible base64 encoding
* zipfile compression (still need to enable toggling this off for
systems without zlib support in python)
* Allow non-wildcard imports (still need to make this recusrsive so that
we can have module_utils code that imports other module_utils code.)
* Better tracebacks: module filename is kept and module_utils directory
is kept so that tracebacks show the real filenames that the errors
appear in.
* Make sure we import modules that are used into the module_utils files that they are used in.
* Set ansible version in a more pythonic way for ziploader than we were doing in module replacer
* Make it possible to set the module compression as an inventory var
This may be necessary on systems where python has been compiled without
zlib compression.
* Refactoring of module_common code:
* module replacer only replaces values that make sense for that type of
file (example: don't attempt to replace python imports if we're in
a powershell module).
* Implement configurable shebang support for ziploader wrapper
* Implement client-side constants (for SELINUX_SPECIAL_FS and SYSLOG)
via environment variable.
* Remove strip_comments param as we're never going to use it (ruins line
numbering)
* Don't repeat ourselves about detecting REPLACER
* Add an easy way to debug
* Port test-module to the ziploader-aware modify_module()
* strip comments and blank lines from the wrapper so we send less over the wire.
* Comments cleanup
* Remember to output write the module line itself in powershell modules
* for line in lines strips the newlines so we have to add them back in
Before this patch, if config was ['/some/path'] then it would enter the
else block and config would be set to [].
The regression this patch fixes was introduced by 700db154.
* Unit tests exposed a problem where nested blocks did not correctly
hit rescue/always portions of parent blocks
* Cleaned up logic in PlayIterator
* Unfortunately fixing the above exposed a potential problem in the
block integration tests, where a failure in an "always" section may
always lead to a failed state and the termination of execution
beyond that point, so certain parts of the block integration test
were disabled.
* fetch_url shouldn't both accept follow_redirects and support follow_redircts via module.params
* Default follow_redirects for open_url should be 'urllib2'
* Add redirect test for get_url
- now workers passes queue to task_executor so it can send back events per item and on retry attempt
- updated result class to pass along events to strategy
- base strategy updated to forward new events to callback
- callbacks now remove 'items' on final result but process them directly when invoked per item
- new callback method to deal with retry attempt messages (also now obeys nolog)
- updated tests to match new signature of task_executor
fixes#14558fixes#14072
The dep chain for roles created during the compile step had bugs, in
which the dep chain was overwriten and the original tasks in the role
were not assigned a dep chain. This lead to problems in determining
whether roles had already run when in a "diamond" structure, and in
some cases roles were not correctly getting variables from parents.
Fixes#14046
this was taken out in an effort to default to the user's shell but creates issues as this is not known ahead of time
and its painful to set executable and shell_type for all servers, it should only be needed for those that restrict the user
to specific shells and when /bin/sh is not available. raw and command may still bypass this by explicitly passing None.
fixes#13882
still conditional
commit 24efa310b58c431b4d888a6315d1285da918f670
Author: James Cammarata <jimi@sngx.net>
Date: Tue Dec 29 11:23:52 2015 -0500
Adding an additional test for copy exclusion
Adds a negative test for the situation when an exclusion doesn't
exist in the target to be copied.
commit 643ba054877cf042177d65e6e2958178bdd2fe88
Merge: e6ee59f 66a8f7e
Author: James Cammarata <jimi@sngx.net>
Date: Tue Dec 29 10:59:18 2015 -0500
Merge branch 'speedup' of https://github.com/chrismeyersfsu/ansible into chrismeyersfsu-speedup
commit 66a8f7e873ca90f7848e47b04d9b62aed23a45df
Author: Chris Meyers <chris.meyers.fsu@gmail.com>
Date: Mon Dec 28 09:47:00 2015 -0500
better api and tests added
* _copy_results = deepcopy for better performance
* _copy_results_exclude to deepcopy but exclude certain fields. Pop
fields that do not need to be deep copied. Re-assign popped fields
after deep copy so we don't modify the original, to be copied, object.
* _copy_results_exclude unit tests
commit 93490960ff4e75f38a7cc6f6d49f10f949f1a7da
Author: Chris Meyers <chris.meyers.fsu@gmail.com>
Date: Fri Dec 25 23:17:26 2015 -0600
remove uneeded deepcopy fields
Also adds a new var, used by the prepare_tests role, to prevent it from
deleting the temp test directory at the start of each play to avoid any
potential race conditions
* Changed parse_addresses to throw exceptions instead of passing None
* Switched callers to trap and pass through the original values.
* Added very verbose notice
* Look at deprecating this and possibly validate at plugin instead
fixes#13608
For the deps setup of integration tests, as we sometimes see odd
errors we can't reproduce, which may be related to slightly out of
date package dependencies.
