Using set_host_overrides() in the connection plugin to access the ssh
argument variables from the inventory didn't see group_vars/host_vars
settings, as noted earlier. Instead, we can set the correct values in
the PlayContext, which has access to all command-line options, task
settings, and variables.
The only downside of doing so is that the source of the settings is no
longer available in ssh.py, and therefore can't be logged. But the code
is simpler, and it actually works.
This change was suggested by @jimi-c in response to the FIXME in the
earlier commit.
Now we have the following ways to set additional arguments:
1. [ssh_connection]ssh_args in ansible.cfg: global setting, prepended to
every command line for ssh/scp/sftp. Overrides default ControlPersist
settings.
2. ansible_ssh_common_args inventory variable. Appended to every command
line for ssh/scp/sftp. Used in addition to ssh_args, if set above, or
the default settings.
3. ansible_{sftp,scp,ssh}_extra_args inventory variables. Appended to
every command line for the relevant binary only. Used in addition to
#1 and #2, if set above, or the default settings.
3. Using the --ssh-common-args or --{sftp,scp,ssh}-extra-args command
line options (which are overriden by #2 and #3 above).
This preserves backwards compatibility (for ssh_args in ansible.cfg),
but also permits global settings (e.g. ProxyCommand via _common_args) or
ssh-specific options (e.g. -R via ssh_extra_args).
Fixes#12576
Fixes another failing test.
(I don't want to do a global search/replace for 'basestring' because I
want to have unit tests covering each occurrence. When I run out of
existing failing tests, I'll try to write new ones.)
SSH pipelining can be a significant performance improvement, but it will
not work if sudoers is configured to requiretty. With this change, one
could have pipelining enabled in ansible.cfg, but use sudo to turn off
requiretty in a separate play (or task) where pipelining is disabled:
- hosts: foo
vars:
ansible_pipelining: no
tasks:
- lineinfile: dest=/etc/sudoers line='Defaults requiretty' state=absent
sudo_user: root
(Note that sudoers has a complicated syntax, so the above lineinfile
invocation may be too simplistic for production use; but the point is
that a separate play can do something to disable requiretty.)
* allow global no_log setting, no need to set at play or task level, but can be overriden by them
* allow turning off syslog only on task execution from target host (manage_syslog), overlaps with no_log functionality
* created log function for task modules to use, now we can remove all syslog references, will use systemd journal if present
* added debug flag to modules, so they can make it call new log function conditionally
* added debug logging in module's run_command
Due to the way we're now calculating delegate_to, if that value is based
on a loop variable ('item') we need to calculate all of the possible
delegated_to variables for that loop.
Fixes#12499
The event loop (even after it was brought into one place in _run in the
previous commit) was hard to follow. The states and transitions weren't
clear or documented, and the privilege escalation code was non-blocking
while the rest was blocking.
Now we have a state machine with four states: awaiting_prompt,
awaiting_escalation, ready_to_send (initial data), and awaiting_exit.
The actions in each state and the transitions between then are clearly
documented.
The check_incorrect_password() method no longer checks for empty strings
(since they will always match), and check_become_success() uses equality
rather than a substring match to avoid thinking an echoed command is an
indication of successful escalation. Also adds a check_missing_password
connection method to detect the error from sudo -n/doas -n.
Now, instead of relying on hostvars on the executor side, we compile
the vars for the delegated to host in a special internal variable and
have the PlayContext object look for things there when applying task/
var overrides, which is much cleaner and takes advantage of the code
already dealing with all of the magic variable variations.
Fixes#12127Fixes#12079
Use six.moves.range instead (aliased to xrange on Python 2, aliased to
range on Python 3).
Also I couldn't resist replacing the elaborate chr/ord/randrange dance
with the simpler random.choice(string.ascii_lowercase) that was already
used elsewhere in the Ansible codebase.
commit 9921bb9d2002e136c030ff337c14f8b7eab0fc72
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date: Mon Aug 10 20:19:44 2015 +0530
Document --ssh-extra-args command-line option
commit 8b25595e7b1cc3658803d0821fbf498c18ee608a
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date: Thu Aug 13 13:24:57 2015 +0530
Don't disable GSSAPI/Pubkey authentication when using --ask-pass
This commit is based on a bug report and PR by kolbyjack (#6846) which
was subsequently closed and rebased as #11690. The original problem was:
«The password on the delegated host is different from the one I
provided on the command line, so it had to use the pubkey, and the
main host doesn't have a pubkey on it yet, so it had to use the
password.»
(This commit is revised and included here because #11690 would conflict
with the changes in #11908 otherwise.)
Closes#11690
commit 119d0323892c65e8169ae57e42bbe8e3517551a3
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date: Thu Aug 13 11:16:42 2015 +0530
Be more explicit about why SSH arguments are added
This adds vvvvv log messages that spell out in detail where each SSH
command-line argument is obtained from.
Unfortunately, we can't be sure if, say, self._play_context.remote_user
is obtained from ANSIBLE_REMOTE_USER in the environment, remote_user in
ansible.cfg, -u on the command line, or an ansible_ssh_user setting in
the inventory or on a task or play. In some cases, e.g. timeout, we
can't even be sure if it was set by the user or just a default.
Nevertheless, on the theory that at five v's you can use all the hints
available, I've mentioned the possible sources in the log messages.
Note that this caveat applies only to the arguments that ssh.py adds by
itself. In the case of ssh_args and ssh_extra_args, we know where they
are from, and say so, though we can't say WHERE in the inventory they
may be set (e.g. in host_vars or group_vars etc.).
commit b605c285baf505f75f0b7d73cb76b00d4723d02e
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date: Tue Aug 11 15:19:43 2015 +0530
Add a FAQ entry about ansible_ssh_extra_args
commit 49f8edd035cd28dd1cf8945f44ec3d55212910bd
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date: Mon Aug 10 20:48:50 2015 +0530
Allow ansible_ssh_args to be set as an inventory variable
Before this change, ssh_args could be set only in the [ssh_connection]
section of ansible.cfg, and was applied to all hosts. Now it's possible
to set ansible_ssh_args as an inventory variable (directly, or through
group_vars or host_vars) to selectively override the global setting.
Note that the default ControlPath settings are applied only if ssh_args
is not set, and this is true of ansible_ssh_args as well. So if you want
to override ssh_args but continue to set ControlPath, you'll need to
repeat the appropriate options when setting ansible_ssh_args.
(If you only need to add options to the default ssh_args, you may be
able to use the ansible_ssh_extra_args inventory variable instead.)
commit 37c1a5b6794cee29a7809ad056a86365a2c0f886
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date: Mon Aug 10 19:42:30 2015 +0530
Allow overriding ansible_ssh_extra_args on the command-line
This patch makes it possible to do:
ansible somehost -m setup \
--ssh-extra-args '-o ProxyCommand="ssh -W %h:%p -q user@bouncer.example.com"'
This overrides the inventory setting, if any, of ansible_ssh_extra_args.
Based on a patch originally by @Richard2ndQuadrant.
commit b023ace8a8a7ce6800e29129a27ebe8bf6bd38e0
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date: Mon Aug 10 19:06:19 2015 +0530
Add an ansible_ssh_extra_args inventory variable
This can be used to configure a per-host or per-group ProxyCommand to
connect to hosts through a jumphost, e.g.:
inventory:
[gatewayed]
foo ansible_ssh_host=192.0.2.1
group_vars/gatewayed.yml:
ansible_ssh_extra_args: '-o ProxyCommand="ssh -W %h:%p -q bounceuser@gateway.example.com"'
Note that this variable is used in addition to any ssh_args configured
in the [ssh_connection] section of ansible.cfg (so you don't need to
repeat the ControlPath settings in ansible_ssh_extra_args).
Replace .iteritems() with six.iteritems() everywhere except in
module_utils (because there's no 'six' on the remote host). And except
in lib/ansible/galaxy/data/metadata_template.j2, because I'm not sure
six is available there.
The lock file is (a temporary file) opened in the parent process, whose
open fd is inherited by the workers after fork, and passed down through
the PlayContext. Connection grows lock/unlock methods which can be used
by individual connection plugins.
FieldAttributes will now by default not be post_validated unless a flag
is set on them in the class, as a large number of fields are really there
simply to be inherited by Task/PlayContext and shouldn't be templated too
early.
The other (unrelated to the base issue) in #12084 is also fixed here, where
the roles field is loaded before vars/vars_files, meaning there are no vars
yet loaded in the play when the templating occurs.
Fixes#12084
`if method in dir(self):` is very inefficient:
- it must construct a list object listing all the object attributes & methods
- it must then perform a O(N) linear scan of that list
Replace it with the idiomatic `if hasattr(self, method):`, which is a
O(1) expected time hash lookup.
Should fix#11981.
* Speed up serialization of hostvars by simply using the internal
dictionary used for cached lookups
* Use blocking gets/puts on queues instead of spin locking
* Merge sequential implicitly created blocks
* now it uses -n to get immediate error if no password is supplied and one is needed,
this should fix the issue with sudo hanging waiting for input.
* made -k configurable, this can break changing become_users in play if left out,
but opens up the possiblity of OTP support.
Fixes bugs related to creating Templar() objects on the fly, where
the shared loader objects (serialized to TaskExecutor) aren't used
so information loaded into plugin loaders after forking is lost.
Fixes#11815
* Don't reference __class__ when we can use the instance itself
* use isdisjoint() as it can stop once a match is found
* Remove a condtional that was taken care of in the conditonal just above
* added pattern matching to match v1 functionality
* check the task name, not the task+role name for matches
* make sure the input is unicode
Fixes#11692
Also making PlayContext a child class of the Playbook Base class,
which gives it access to all of the FieldAttribute code to ensure
field values are correctly typed after post_validation
Fixes#11381
There was a confusion between the valid values for defaults.gathering
(explicit/implicit/smart) and a play's gather_facts setting (boolean),
which resulted in gathering=explicit being ignored.
The tags field may contain bad data before it is post_validated, however
some methods assumed it would be a simple list or string. Using itertools
gets us around the problem of the data potentially not being hashable
Fixes#9380
Getting recursive errors otherwise, so this is probably not something
we want to do. This most likely only worked in v1 due to the fact that
module args were templated earlier than the point in Runner() when
they were fed into the templating engine.
The --force-handlers command line argument was not correctly running
handlers on hosts which had tasks that later failed. This corrects that,
and also allows you to specify force_handlers in ansible.cfg or in a
play.
- become constants inherit existing sudo/su ones
- become command line options, marked sudo/su as deprecated and moved sudo/su passwords to runas group
- changed method signatures as privlege escalation is collapsed to become
- added tests for su and become, diabled su for lack of support in local.py
- updated playbook,play and task objects to become
- added become to runner
- added whoami test for become/sudo/su
- added home override dir for plugins
- removed useless method from ask pass
- forced become pass to always be string also uses to_bytes
- fixed fakerunner for tests
- corrected reference in synchronize action plugin
- added pfexec (needs testing)
- removed unused sudo/su in runner init
- removed deprecated info
- updated pe tests to allow to run under sudo and not need root
- normalized become options into a funciton to avoid duplication and inconsistencies
- pushed suppored list to connection classs property
- updated all connection plugins to latest 'become' pe
- includes fixes from feedback (including typos)
- added draft docs
- stub of become_exe, leaving for future v2 fixes
Adds a special tag:
- always: always runs no matter what --tags, --skip-tags say
Adds 4 special keywords for --tags/skip-tags
- all: all tagged + untagged tasks
- tagged: only tagged tasks
- untagged: only untagged tasks
- always: only run tasks tagged 'always'
Adds new settings for managing retry files:
* retry_files_enabled, defaults to True
* retry_files_save_path, defaults to ~/.ansible-retry
This change was adapted from PR #5515.
* Roles can now be given a friendly name as third field in role spec csv
* Roles can be installed from URL (not just from archived SCMs)
* Integration tests to demonstrate this
* Unit tests to ensure that role spec parsing works as expected
Ensure that if meta/main.yml has dependencies containing URLs
these are correctly resolved at playbook runtime.
Update role_name during run time so that output of e.g. task names
look sensible
(serial_pct)/100.0 is a typo - the parentheses should be around the entire percentage calculation, which makes much more sense when multiplying by the number of hosts (luckily BODMAS prevailed).
Prevents a "not a legal parameter" error for play ds structures that
may be passed into the Play() constructor, which really is mainly a
concern during tests.
Since the filter list contains hosts, passing that back to list_hosts()
in the inventoy causes issues when the hostname is an IPv6 address
(with :'s interpreted as group unions). Since we already have the list
of hosts, we should not need to pass that back through inventory a
second time.
Fixes#7446
Since some of the vars contained in there may be used in the templating
of other variables. This also reverts e83a494 which originally fixed
issue #6979 but broke some other variable related things.
Fixes#6979Fixes#7321Fixes#7332
I made a typo in a playbook and was great by:
ERROR: expecting dict; got: None
The issue was a single - on the last line of a playbook.
With the name of the file, I was able to see right away where the
error was.
This can be illustrated using this playbook:
- command: id
delegate_to: "{{ remote_server }}"
user: "{{ remote_user }}"
The error is to use 'user' instead of 'remote_user', but the error message
do not really mention it, so it can be a bit hard to spot.
This small change allows for individual setup actions to populate the SETUP_CACHE and not cause a subsequent facts-gathering when not needed. This follows the standard of other facts modules as laid out in #1206 and implemented in fedfd18774. It allows to test of the setup module has already been run even when gather_facts was explicitely disabled.
This occurred when a hash would be passed in via extra args and the
hash variable behavior was set to 'merge', which resulted in the
variable from extra args replacing the playbook variable.
this variable has the 'current host list' to be executed over in the
play. Useful when using --limit to not iterate over hosts not included
in play in templates or with_items.
Signed-off-by: Brian Coca <briancoca+dev@gmail.com>
Operate on that play attribute to make things faster for larger
inventories. Instead of making a round trip through inventory.list_hosts
and working through some lengthy list comprehensions over and over
again, calculate the potenital hosts for a play once, then reduce from
it the unavailable hosts when necessary.
Also moves how the %fail is done. The host count is a play level count
of available hosts, which then is compared after each task to the
current number of available hosts for the play. This used to get a new
count every task which was also time expensive.
The _list_available_hosts call can be lengthy, and in the case where
gather_facts is disabled the call is pointless. So re-arrange the logic
to return early from _do_setup_step when gather_facts is false.
Users of these features should use "when:" as documented at docs.ansible.com.
Similarly, include + with_items has been removed. The solution is to loop
inside the task files, see with_nested / with_together, etc.
Using ANSIBLE_ROLE_PATH environment variable or role_path in ansible.cfg
can configure paths where roles will be searched for
extra paths will only be used as a backup once regular locations are exhausted
Previously, includes had to receive variables via a special 'vars'
field. With this patch, the include syntax becomes a more natural
datastructure without special fields and is more akin to the way
role includes/dependencies work.
Tested with the following playbook:
---
- hosts: localhost
connection: local
tasks:
- { include: inc1.yml, a: 1 }
- include: inc2.yml
b: 2
- include: inc3.yml
with_items:
- x
- y
- z
Fixes#3481
Still compatible with user: but deprecating it so we can have
a matching remote_user: in tasks, cannot be user: because of the
module of the same name. #3932
Signed-off-by: Brian Coca <briancoca+dev@gmail.com>
Reported by Rumen:
TASK: [fail FAIL] *************************************************************
skipping: [hostname.com]
failed: [hostname.com] => {"failed": true}
msg: Failed as requested from task
Traceback (most recent call last):
File "/usr/local/bin/ansible-playbook", line 268, in <module>
sys.exit(main(sys.argv[1:]))
File "/usr/local/bin/ansible-playbook", line 208, in main
pb.run()
File "/Library/Python/2.7/site-packages/ansible/playbook/__init__.py", line 262, in run
if not self._run_play(play):
File "/Library/Python/2.7/site-packages/ansible/playbook/__init__.py", line 580, in _run_play
if (hosts_count - len(host_list)) > int((play.max_fail_pct)/100.0 * hosts_count):
TypeError: object of type 'NoneType' has no len()
The play was just checking for the presence of the keyword in the
YAML datastructure, and not the value of the field, so doing something
like variable substitution was always causing the play to be accelerated
* Default variables are now fed directly into roles, just like the
other variables, so that roles see their unique values rather
than those set at the global level.
* Role dependency duplicates are now determined by checking the params used
when specifying them as dependencies rather than just on the name of the
role. For example, the following would be included twice without having
to specify "allow_duplicates: true":
dependencies:
- { role: foo, x: 1 }
- { role: foo, x: 2 }
