* Introduce new 'filetree' lookup plugin
The new "filetree" lookup plugin makes it possible to recurse over a tree of files within the task loop. This makes it possible to e.g. template a complete tree of files to a target system with little effort while retaining permissions and ownership.
The module supports directories, files and symlinks.
The item dictionary consists of:
- src
- root
- path
- mode
- state
- owner
- group
- seuser
- serole
- setype
- selevel
- uid
- gid
- size
- mtime
- ctime
EXAMPLES:
Here is an example of how we use with_filetree within a role:
```yaml
- name: Create directories
file:
path: /web/{{ item.path }}
state: directory
mode: '{{ item.mode }}'
owner: '{{ item.owner }}'
group: '{{ item.group }}'
force: yes
with_filetree: web/
when: item.state == 'directory'
- name: Template complete tree
file:
src: '{{ item.src }}'
dest: /web/{{ item.path }}
state: 'link'
mode: '{{ item.mode }}'
owner: '{{ item.owner }}'
group: '{{ item.group }}'
with_filetree: web/
when: item.state == 'link'
- name: Template complete tree
template:
src: '{{ item.src }}'
dest: /web/{{ item.path }}
mode: '{{ item.mode }}'
owner: '{{ item.owner }}'
group: '{{ item.group }}'
force: yes
with_filetree: web/
when: item.state == 'file'
```
SPECIAL USE:
The following properties also have its special use:
- root: Makes it possible to filter by original location
- path: Is the relative path to root
- uid, gid: Makes it possible to force-create by exact id, rather than by name
- size, mtime, ctime: Makes it possible to filter out files by size, mtime or ctime
TODO:
- Add snippets to documentation
* Small fixes for Python 3
* Return the portion of the file’s mode that can be set by os.chmod()
And remove the exists=True, which is redundant.
* Use lstat() instead of stat() since we support symlinks
* Avoid a few possible stat() calls
* Bring in line with v1.9 and hybrid plugin
* Remove glob module since we no longer use it
* Included suggestions from @RussellLuo
- Two blank lines will be better. See PEP 8
- I think if props is not None is more conventional 😄
* Support failed pwd/grp lookups
* Implement first-found functionality in the path-order
* when including statically, make sure that all parents were also included
statically (issue #16990)
* properly resolve nested static include paths
* print a message when a file is statically included
Fixes#16990
When unittesting this we found that the platform selecting class
hierarchies weren't working in all cases. If the subclass was directly
created (ie: LinuxHardware()), then it would use its inherited __new__()
to try to create itself. The inherited __new__ would look for
subclasses and end up calling its own __new__() again. This would
recurse endlessly. The new code detects when we want to find a subclass
to create (when the base class is used, ie: Hardware()) vs when to
create the class itself (when the subclass is used, ie:
LinuxHardware()).
Rather than repeatedly searching for tasks by uuid via iterating over
all known blocks, cache the tasks when they are added to the PlayIterator
so the lookup becomes a simple key check in a dict.
It is possible that a block is copied prior to validation, in which case
some fields (like when) which should be something other than a string might
not be. Using validate() in copy() is relatively harmless and ensures the
blocks are in the proper structure.
This also cleans up some of the finalized logic from an earlier commit and
adds similar logic for validated.
Fixes#17018
After post_validate() is called on an object, there should be no
need to continue looking up at parent attributes. This patch adds a
new flag (_finalized) which is set to True at the end of post_validate,
and getattr will not look beyond its own attributes from that point on.
* Allow to make the jsonfile cache files pretty (indented and sorted)
Since the json cache files are condensed, it is not very practical to look for something in them. Having indented/sorted cache files makes debugging and playbook/inventory development a lot easier to do.
I made it configurable in case people would object to the performance hit this would have, but to be honest, then they probably should be looking at other cache plugins instead IMO.
* Removed the config option and documentation changes
* Query lookup plugin
* Add license and docstrings
* Add python3-ish imports
* Change query plugin type from lookup to filter
* Switch from dq to jsonpath_rw
* Add integration test for query filter
* Rename query filter to json_query
* Add jsonpath-rw
* Rename query filter to json_query
* Switch query implementation from jsonpath-rw to jmespath
Run setfacl/chown/chmod on each temp dir and file.
This fixes temp file permissions handling on platforms such as FreeBSD
which always return success when using find -exec. This is done by
eliminating the use of find when setting up temp files and directories.
Additionally, tests that now pass on FreeBSD have been enabled for CI.
Due to the way we load plugins, internally to Python there can be issues when
the debug strategy is loaded after the linear strategy. To work around this,
we're changing the import line for the linear strategy to avoid the problem.
Related to #16825
uri:
follow_redirects: no
Will lead yaml to set follow_redirects=False. This is problematic when
the module parameter is not a boolean value but a string. For instance:
follow_redirects = dict(required=False, default='safe', choices=['all', 'safe', 'none', 'yes', 'no']),
Our parameter validation code ends up getting follow_redirects="False"
instead of "no". The 100% fix is for the user to quote their strings in
playbooks like:
uri:
follow_redirects: "no"
But we can fix quite a few common cases by trying to switch "False" back
into the string that it was specified as. We only do this if there is
only one correct choices value that could have been specified. In the
follow_redirects example, a value of "True" only maps back to "yes" and
a value of "False" only maps back to "no" so we can do this. If choices
also contained "on" and "off" then we couldn't map back safely and would
need to force the module author to change the module to handle this
case.
Fixes parts of the following PRs:
* https://github.com/ansible/ansible-modules-core/pull/4220
* https://github.com/ansible/ansible-modules-extras/pull/2593
* These can still race when multiple ansible processes are created at
the same time.
* Reverse order of expanduser and expandvars in unfrakpath(). So that
tildes in environment variables will be handled.
When a task result has an empty results list, the
list should be ignored when determining the results
of `_check_key`. Here the empty list is treated the
same as a non-existent list.
This fixes a bug that manifests itself with squashed
items - namely the task result contains the correct
value for the key, but an empty results list. The
empty results list was treated as zero failures
when deciding which handler to call - so the task
show as a success in the output, but is deemed to
have failed when deciding whether to continue.
This also demonstrates a mismatch between task
result processing and play iteration.
A test is also added for this case, but it would not
have caught the bug - because the bug is really in
the display, and not the success/failure of the
task (visually the test is more accurate).
Fixesansible/ansible-modules-core#4214
This feature changes the scalar value of `serial:` to a list, which
allows users to specify a list of values, so batches can be ramped
up (commonly called "canary" setups):
- hosts: all
serial: [1, 5, 10, "100%"]
tasks:
...
* Revert "There can be only one localhost"
This reverts commit 5f1bbb4fcd.
this broke several usages of localhost, see #16882, #16898 and #16886
* ensure there is only 1 localhost
fixes#16886, #16882 and #16898
- make sure localhost exists before returning it
- optimzed host caching
- ensure we always return a host object
The module level function defs for gcdns_connect() and
gce_connect() provide a default arg for 'provider' that
references into the libcloud module. If the libcloud
modules were not installed, the gce/gcdns python modules
would throw ImportError.
Let the provider arg default to None and if not provided,
set it to the default libcloud.compute.types.Provider.*
value if the modules are installed.
The lack of a comma caused the statement to always evaluate as a
`TypeError` when python interpreted `value (list, tuple, dict)` to call
value with the arguments list, tuple, and dict.
This is a refactoring of the existing GCE utility module to support other projects on Google Cloud Platform.
The previous gce.py module was hard-coded specifically for GCE, and attempting to use it with other projects in GCP failed.
See https://github.com/ansible/ansible/pull/15918#issuecomment-220165913 for more detail.
This has also been an issue for others in the past, although they've handled it by simply
duplicating some of the logic of gce.py in their own modules.
- The existing gce.py module was renamed to gcp.py, and modified to remove any
imports or other code that refers to libcloud.compute or GCE (the GCE_* params were
retained for compatibility). I also renamed the gce_connect function to gcp_connect,
and modified the function signature to make supplying a provider, driver, and agent
information mandatory.
- A new gce.py module was created to handle connectivity to GCE. It imports the
appropriate libcloud.compute providers and drivers, and then passes them on
to gcp_connect in gcp.py. The constants and function signatures are the same
as the old gce.py, so compatibility with existing modules is retained.
- A new gcdns.py module was created to support PR ansible/ansible-modules-extras#2252
for two new Google Cloud DNS modules, and to demonstrate support for a non-GCE
Google Cloud service. It follows the same basic structure as the new gce.py module,
but imports from libcloud.dns instead.
I'm not sure why that would be desirable -- we really want __version__
to come from the controller whereas importing will come from the client
node. If it turns out there was a reason to do that, please be sure to
use an exception handler that catches all exceptions instead of only
catching ImportError:
```
try:
from ansible.release import __version__, __author__
except:
__version__ = [...]
```
Fixes#16523
* switch cwd to basedir of task
This restores previous behaviour in pre 2.0 and allows for 'local type' plugins
and actions to have a more predictable relative path.
fixes#14489
* removed FIXME since prev commit 'fixes' this
* fix tests, now they need a loader (thanks jimi!)
* add check_mode option for tasks
includes example testcases for the template module
* extend check_mode option
* replace always_run, see also proposal rename_always_run
* rename always_run where used and add deprecation warning
* add some documentation
* have check_mode overwrite always_run
* use unique template name to prevent conflicts
test_check_mode was right before, but failed due to using the same filename as other roles
* still mention always_run in the docs
* set deprecation of always_run to version 2.4
* fix rst style
* expand documentation on per-task check mode
now systemd will run even if service module is inovked with parameters that it does not support
these will be removed before invoking systemd and issue a warning.
this facility will work for any new service modules.
A simple import of cryptography can throw several types of errors. For example,
if `setuptools` is less than cryptography's minimum requirement of 11.3, then
this import of cryptography will throw a VersionConflict here. An earlier case
threw a DistributionNotFound exception.
An optional dependency should not stop ansible. If the error is more than
an ImportError, log a warning, so that errors can be fixed in ansible or
elsewhere.
This bug was introduced in 3ced6d3, where getting vars from a role
did not follow the dep chain. This was originally hidden by the fact
that we got vars twice (from the block and from the roles directly).
Fixes#16729
* fixed lookup search path
added ansible_search_path var that contains the proper list and in order
removed roledir var which was only used by first_found, rest used role_path
added needle function for lookups that mirrors the action plugin one, now
both types of plugins use same pathing.
* added missing os import
* renamed as per feedback
* fixed missing rename in first_found
* also fixed first_found
* fixed import to match new error class
* fixed getattr ref
* moved tests from filters to actual jinja2 tests
also removed some unused declarations and imports
* split tests into their own docs
removed isnan as existing jinja2's 'number' already covers same
added missing docs for several tests
* updated as per feedback
2e003adb added the ability for tasks using any_errors_fatal to fail
when there were unreachable hosts. However that patch used the running
unreachable hosts data rather than the results from the current task,
which causes failures when any run_once or BYPASS_HOST_LOOP task is hit
after an unreachable host causes a failure. This patch corrects that by
using the current set of results to determine if any hosts were
unreachable during the last task only.
Fixesansible/ansible-modules-core#4160
This adds a action plugin that will allow config and template modules
to be merged into a single module. Once completed this will supercede
the net_template action plugin.
* add load_config() for loading a set of configuration commands
* add load_candidate() function for loading a candidate config
* updates shared module to provide NetworKModule instead of get_module
* fixes Cli transport implementation for 2.2 refactor
* updates ios documentation fragments with new options
* diff functions now split out for easier troubleshooting
* added dumps() function to serialize config objects to strings
* difference() can now expand all blocks instead of just singluar blocks
includes changes from PR ansible/ansible#16636 and refactors for the
NetworkModule changes
new features
* ios now supports transport=restcon will additional arguments
* ModuleStub refactored into common network shared module
* import temporary get_module() function (to be removed prior to 2.2 final)
This is a temporary change to keep the get_module() function until all
of the network module refactoring is completed to avoid breaking them
in devel. The get_module() function should not be used and will be
removed before 2.2 final.
* Update IOS with new NetworkModule
* Remove redundant EOS code
* `authorize` can get rolled into NetCli
* Fix up IOS to where EOS is.
* Update IOSXR for NetworkModule
* collections is unnecessary
Since Ansiballz, we no longer need to import basic directly into
a new-style module. Some modules, like the Networking modules, may
import basic in their own module_utils files and the module will import
that specialized module_util file rather than basic.
* Don't treat parsing problems as async task timeout
If there is a problem reading/writing the status file that manifests as
not being able to parse the data, that doesn't mean the task timed out,
it means there was what was likely a tempoarary problem. Move on and
keep polling for success. The only things that should cause the async
status to not be parseable are bugs in the async_runner.
* Add comment explaining not bailing out of loop
* Return different error when result is unparseable
* Remove extraneous else
* Instead of rebuilding the handler list all over the place, we now
compile the handlers at the point the play is post-validated so that
the view of the play in the PlayIterator contains the definitive list
* Assign the dep_chain to the handlers as they're compiling, just as we
do for regular tasks
* Clean up the logic used to find a given handler, which is greatly
simplified by the above changes
Fixes#15418
The `boto3_conn` function requires a module argument, and calls
`module.fail_json` if the connection doesn't receive enough arguments.
In non-module settings like inventory scripts, there is no module to be
passed.
The `boto3_inventory_conn` function takes the same arguments except for
`module`, and both call _boto3_conn which doesn't require a module be
passed.
* fixes lots of bugs with get_config function to perform correctly
* refactors load_config into load_candidate
* adds load_config function to convert commands to NetworkConfig
The Command object can now store the response from executing the command
to allow it to be retrieved later by command name. This update will
update the Command instance with the response before returning.
This adds a new method that will return the output from a specified
command that has already been excuted by the CommandRunner. The new
method, get_command takes a single argument which is the full name
of the command to retrieve.
6eefc11c converted task.loop_control into an object, but while the other
callers were updated to use .loop_var instead of .get('loop_var'), this
site was overlooked.
This can be reproduced by including with loop_control a file that does
set_fact; a simple regression test along these lines is included.
This fix prevents a broken pipe exception from occurring when password-less
SSH is configured and the sshpass process exits and closes the pipe before
the password is written to the pipe.
We want to update host vars for all hosts (even those that might
have failed), and the in case of a refresh_inventory, the code has
a stale restrictions list at this point anyway.
* smarter function to figure out relative paths
takes list of paths in order of relevance to current task
and does the dwim magic on them
* shared function for action plugins using new dwim
unify path construction and error info/messaging
made include and role non exclusive
corrected order and now smarter about tasks
includes inside roles are currently broken as they don't provide the correct role data
make dirname full match to avoid corner cases
* migrated action plugins to new dwim function
reported plugins to use exceptions instead of info
* clarified needle
In the case of using YAML anchors/aliases, YAML actually uses references
to the duplicated object so any modifications to the original impacts
later uses of the object.
Fixes#13575
* Lookup unencrypted password must not include salt
* Integration test lookup: remove previous directory
* Test that lookup password doesn't return salt
* Lookup password: test behavior with empty encrypt parameter
Closes#16189
* Remove unnecessary copying of values from parents to role deps, as
this can cause problems when roles have multiple parents (or the same
parents with different params speficied through deps)
* Since we're already checking the dep chain in the block for role
things (which every task in a role should have), it is not necessary
to check the role directly in case it improperly grabs something
Fixes#14438
Our custom encoder for the to_json filter was simply returning the
object if it was not a HostVars object, leading in some cases to a
TypeError when the data contained an undefined variable. This lead
to an odd error message being propagated up, so we now properly catch
this as an undefined variable error.
Fixes#15610
Again, as we're carrying failed/unreachable hosts forward from play to play via
internal structures, we need to remember which ones had previously failed so that
unrelated host failures don't inflate the numbers for a given serial batch in the
PlaybookExecutor causing a premature exit.
Fixes#16364
The listen statement on handlers should have supported a list, however
it was broken in the revision of the pub/sub feature based on the handler
revamp. This patch corrects the bug, so this works again:
- name: some handler
...
listen:
- some target
- another target
Fixes#16378
* add new module network
* move EOS to NetworkModule
* shell.py Python 3.x compatibility
* implements the Command class through the connection for eos
This implements a new Command class that specifies the cli command
and output format. This removes the need to batch commands through
the connection
* initial add of netcmd module
Due to the fact that roles may be instantiated with different sets of
params (multiple inclusions of the same role or via role dependencies),
simply tracking notified handlers by name does not work. This patch
changes the way we track handler notifications by using the handler
object itself instead of just the name, allowing for multiple internal
instances. Normally this would be bad, but we also modify the way we
search for handlers by first looking at the notifying tasks dependency
chain (ensuring that roles find their own handlers first) and then at
the main list of handlers, using the first match it finds.
This patch also modifies the way we setup the internal list of handlers,
which should allow us to correctly identify if a notified handler exists
more easily.
Fixes#15084
This removes the extra layer of quotes around values in the 'args' file.
These quotes were there before the pipes.quote() call was added, but
were not removed, resulting in too much quoting.
Manifests as the following stack trace
File "/usr/local/Cellar/ansible/2.0.1.0/libexec/lib/python2.7/site-packages/ansible/utils/display.py", line 259, in error
new_msg = u"ERROR! " + msg
TypeError: coercing to Unicode: need string or buffer, AnsibleParserError found
This makes Ansible no longer set LC_ALL for remote systems. It is up to
the individual modules to set LC_ALL if they need it for screenscraping
the output from a program.
This is the 2.2 followup for #15138
Problem: When setting the file permissions on the remote server for
unprivileged users ansible expects that a chown will fail for unprivileged
users. For some systems (e.g. HP-UX) this is not the case.
Solution: Change the order how ansible sets the remote permissions.
* If the remote_user sudo's to an unprivileged user then we attempt to
grant the unprivileged user access via file system acls.
* If granting file system acls fails we try to change the owner of the
file with chown which only works in case the remote_user is privileged
or the remote systems allows chown calls by unprivileged users (e.g.
HP-UX)
* If the chown fails we can set the file to be world readable so that
the second unprivileged user can read the file. Since this could allow
other users to get access to private information we only do this
ansible is configured with "allow_world_readable_tmpfiles" in the
ansible.cfg
