Some debuggers are easier to work with when we do everything in a single
process. This debug option caters to that at the expense of being
different from what Ansible will actually do to invoke a module.
When we document this we should be clear that this shouldn't be used for
general purpose debugging and that some modules may show strange
"errors" when used with this. Those won't be considered real bugs as
it's not how ansible really invokes the modules.
* Run the module as a script from the wrapper instead of executing in the same process.
Fixes cornercases where the module could potentially be executed twice
if we import and then run the main() function without calling sys.exit()
somewhere.
Also fixes problem with concurrent.futures() hanging. Not sure
precisely how this is being triggered but it is related to invoking the
main() function outside of an if __name__ == '__main__' conditional.
* Fix for python-2.6
* Ziploader proof of concept (jimi-c)
* Cleanups to proof of concept ziploader branch:
* python3 compatible base64 encoding
* zipfile compression (still need to enable toggling this off for
systems without zlib support in python)
* Allow non-wildcard imports (still need to make this recusrsive so that
we can have module_utils code that imports other module_utils code.)
* Better tracebacks: module filename is kept and module_utils directory
is kept so that tracebacks show the real filenames that the errors
appear in.
* Make sure we import modules that are used into the module_utils files that they are used in.
* Set ansible version in a more pythonic way for ziploader than we were doing in module replacer
* Make it possible to set the module compression as an inventory var
This may be necessary on systems where python has been compiled without
zlib compression.
* Refactoring of module_common code:
* module replacer only replaces values that make sense for that type of
file (example: don't attempt to replace python imports if we're in
a powershell module).
* Implement configurable shebang support for ziploader wrapper
* Implement client-side constants (for SELINUX_SPECIAL_FS and SYSLOG)
via environment variable.
* Remove strip_comments param as we're never going to use it (ruins line
numbering)
* Don't repeat ourselves about detecting REPLACER
* Add an easy way to debug
* Port test-module to the ziploader-aware modify_module()
* strip comments and blank lines from the wrapper so we send less over the wire.
* Comments cleanup
* Remember to output write the module line itself in powershell modules
* for line in lines strips the newlines so we have to add them back in
* Can be configured in the ansible.cfg for tasks/handlers individually
* If an included filename contains no vars or loops, it will be expanded
in-place as if it were marked as static
* Unit tests exposed a problem where nested blocks did not correctly
hit rescue/always portions of parent blocks
* Cleaned up logic in PlayIterator
* Unfortunately fixing the above exposed a potential problem in the
block integration tests, where a failure in an "always" section may
always lead to a failed state and the termination of execution
beyond that point, so certain parts of the block integration test
were disabled.
main_q is not used anywhere in the codebase.
It is created in TaskQueueManager._initialize_processes, bundled with rslt_q
into TaskQueueManger._workers, later unwrapped in StrategyBase but not used.
This queue is closed in TaskQueueManger._cleanup_processes.
Historically, it is passed as a init parameter into WorkerProcess,
introduced in 62d7956, but this behavior is changed in 120b9a7.
Signed-off-by: 夏恺(Xia Kai) <xiaket@gmail.com>
- now workers passes queue to task_executor so it can send back events per item and on retry attempt
- updated result class to pass along events to strategy
- base strategy updated to forward new events to callback
- callbacks now remove 'items' on final result but process them directly when invoked per item
- new callback method to deal with retry attempt messages (also now obeys nolog)
- updated tests to match new signature of task_executor
fixes#14558fixes#14072
* Fixes bug where the task was not marked as failed if the number of
retries were exceeded (#14461)
* Reorganizing logic to be a bit cleaner, and so retrie messages are
shown before sleeping (which makes way more sense)
Fixes#14461Fixes#14580
* Make sure dep chains are checked recursively for nested blocks
* Fixing iterator is_failed() check to make sure we're not in a
rescue block before returning True
* Use is_failed() to test whether a host should be added to the TQM
failed_hosts list
* Use is_failed() when compiling the list of hosts left to iterate
over in both the linear and free strategies
Fixes#14222
- moved to base cli class to handle centrally and duplicate less code
- now avoids duplication and reiteration of signal handler by reassigning it
- left note on how to do non-graceful in case we add in future
as I won't remember everything i did here and don't want to 'relearn' it.
- adhoc now terminates gracefully
- avoid race condition on terminations by ignoring errors if
worker might have been reaped between checking if active and termination call
- ansible-playbook now properly exits on sigint/term
- adhoc and playbook now give exceptions that we should not normally capture
and rely on top level finally to reap children
- handle systemexit breaks in workers
- added debug to see at which frame we exit
partial fix for #14346
* Fixed a bug in PlayIterator when ITERATING_ALWAYS, where the block
was advanced but the incorrect data structure elements were cleared
* Cleaned up the logic of is_failed() in PlayIterator
* Fixed a bug in the free strategy which had not been updated to use
the base strategy _execute_meta() method
* Stopped strategies from using is_failed() to determine if tasks should
still be fetched for a host
Fixes#14040
The dep chain for roles created during the compile step had bugs, in
which the dep chain was overwriten and the original tasks in the role
were not assigned a dep chain. This lead to problems in determining
whether roles had already run when in a "diamond" structure, and in
some cases roles were not correctly getting variables from parents.
Fixes#14046
* Don't re-use the existing connection if the remote_addr field of
the play context has changed
* When overriding variables in PlayContext (from task/variables),
don't set the same attribute based on a different variable name
if we had already previously set it from another variable name
Fixes#13880
* Relocate the assignment of the host address to the remote_addr field
in the play context, which was only done when the connection was created
(it's now done after the post_validate() is called on the play context)
* Make the assignment of the play context to the connection an else, since
it's not required if the connection is not reused
If this isnt updated, the _connection is reused, and thus has an outdated _play_context
This results in outdated `success_key` and `prompt` causing issues if sudo is run in a loop
Refer to the issue #13763 for more debugging and details
pushed it to use the existing propmpt from display and moved the vars prompt code there also for uniformity
changed vars_prompt to check extra vars vs the empty play.vars to restore 1.9 behaviour
sipmlified the code as it didn't need to check for syntax again (tqm is made none prior based on that)
fixes#13770
Still is a warning as we don't want to repeat it multiple times nor additional callbacks to stop ansible execution.
hopefully we can avoid shipping w/o exceptions in the default/minimal callbacks...
Also added feature that now allows for 'preformated' strings passed to warning
* Added additional methods to the iterator code to assess host failures
while also taking into account the block rescue/always states
* Fixed bugs in the free strategy, where results were not always being
processed after being collected
* Added some prettier printing to the state output from iterator
Fixes#13699
Because the fail_state is potentially non-zero in these block sections,
the prior logic led to included tasks not being inserted at all.
Related issue: #13605
* Saving of the registered variable was occuring after the tests for
changed/failed_when.
* Each of the above fields and until were being post_validated too early,
so variables which were not defined at that time were causing task
failures.
Fixes#13591
* Move self._tqm.load_callbacks() earlier to ensure that v2_on_playbook_start can fire
* Pass the playbook instance to v2_on_playbook_start
* Add a _file_name instance attribute to the playbook
The code isn't sophisticated enough to understand lists and dicts yet.
This mirrors how 1.9.x handled non-string items so its not a regression.
One portion of a fix for #12976
* Properly mark hosts with failures in includes as failed
* Don't send callbacks until we're sure we're done, and also fix how
we increment stats so failures don't show up as ok's
* Fix a bug in the include file logic where a failed include could lead
to an infinite loop in the task iteration logic
Fixes#12933
Simplifies logic and prevents us from accidentally post_validating
an include that would otherwise be skipped due to tags causing a
problem because of potentially missing variables.
Fixes#12793
I inadvertently introduced it in
ca826508d9 and didn't notice, because
there are no unit tests for playbook_executor.py. Sorry!
(The "from ansible.errors import *" was used *only* to get the 'os'
module, which makes go "what?")
Also does some reorganization/cleanup on the magic vars/delegated
variable generation portions of VariableManager to make the above
possible.
Fixes#12633
corrected output from default callback
added new tests for no_log loops
updated makefile test to check for both positive and negative occurrences of no_log
Due to the way we're now calculating delegate_to, if that value is based
on a loop variable ('item') we need to calculate all of the possible
delegated_to variables for that loop.
Fixes#12499
Now, instead of relying on hostvars on the executor side, we compile
the vars for the delegated to host in a special internal variable and
have the PlayContext object look for things there when applying task/
var overrides, which is much cleaner and takes advantage of the code
already dealing with all of the magic variable variations.
Fixes#12127Fixes#12079
* Clearing interpreter settings from variables, so those set for the
original host aren't incorrectly applied to the delegated to host
* Fixed incorrect string for remote user in delegated hosts hostvars
* Properly looking for multiple possiblities in the delegated-to hosts
hostvars (ansible_ssh_host vs. ansible_host)
The earlier distinction was never used; .ipv6_address was always a copy
of .ipv4_address, and the latter was always used to set the remote_addr
field in the PlayContext.
Also uses the canonical ansible_host/ansible_port names when setting the
address and port from variables.
needed for winrm, disabled closing connections in ssh to avoid issues with that persistance, need to normalize all this in future
This reverts commit 23a22397bf.
* When iterating over a child state, a failure should be propagated
up so parent blocks don't continue iterating
* Make sure a child state exists before trying to search it
Fixes#12210
Replace .iteritems() with six.iteritems() everywhere except in
module_utils (because there's no 'six' on the remote host). And except
in lib/ansible/galaxy/data/metadata_template.j2, because I'm not sure
six is available there.
* Speed up serialization of hostvars by simply using the internal
dictionary used for cached lookups
* Use blocking gets/puts on queues instead of spin locking
* Merge sequential implicitly created blocks
Fixes bugs related to creating Templar() objects on the fly, where
the shared loader objects (serialized to TaskExecutor) aren't used
so information loaded into plugin loaders after forking is lost.
Fixes#11815
- Add support for inserting module args into PowerShell modules. Fixes#11661.
- Support Windows paths containing spaces. Applies changes from #10727 to v2. Fixes#9999. Should also fixansible/ansible-modules-core#944 and ansible/ansible-modules-core#1007.
- Change how execution policy is set for running remote scripts. Applies changes from #11092 to v2. Also fixesansible/ansible-modules-core#1776.
- Use codepage 65001 (UTF-8) for WinRM connection instead of default (CP437), convert command to UTF-8 and results from UTF-8. Replaces changes from #10024. Fixes#11198.
- Close WinRM connection when task completes.
- Use win_stat, win_file and win_copy modules instead of stat, file and copy when called from within other action plugins (only when using WinRM+PowerShell).
- Unquote Windows path arguments before passing to win_stat, win_file, win_copy and slurp modules (only when using WinRM/PowerShell).
- Check for win_ping module to determine if core modules are missing (only when using WinRM/PowerShell).
- Add stdout_lines to result from running low level commands (so stdout_lines is available when using raw/script).
- Update copy action plugin to use shell functions for joining paths and checking for trailing slash.
- Update fetch action plugin to unquote source path when using Windows paths.
- Add win_copy and win_template action plugins that inherit from copy and template.
- Support running .bat and .cmd scripts using default system encoding instead of UTF-8.
- Always send PowerShell commands as base64-encoded blobs to allow for running simple PowerShell commands via raw.
- Support running modules on Windows with interpreters other than PowerShell.
- Update integration tests to support above changes and test unicode fixes.
- Add test for win_user error from ansible/ansible-modules-core#1241 (fixed by ansible/ansible-modules-core#1774).
- Add test for additional win_stat output values (implemented by ansible/ansible-modules-core#1473).
- Add test for OS architecture and name from setup.ps1 (implemented by ansible/ansible-modules-core#1100).
All WinRM integration tests pass for me with these changes.
This is unsafe and we debated re-adding it to the v2/2.0 codebase,
however it is a common-enough feature that we will simply mark it
as deprecated for now and remove it at some point in the future.
Fixes#11718
* added pattern matching to match v1 functionality
* check the task name, not the task+role name for matches
* make sure the input is unicode
Fixes#11692
* Prompt had to be moved up, as it needed to be done before the first
templating of the play occurs, otherwise vars_files won't be templated
properly
* Fixed a bug related to an earlier fix of vars_files incorporating
extra vars to do the templating of the file name
Fixes#11404
Also making PlayContext a child class of the Playbook Base class,
which gives it access to all of the FieldAttribute code to ensure
field values are correctly typed after post_validation
Fixes#11381
Moving the make_sudo_cmd() calls back up to the action level so that connection
plugins don't have to know about it at all, and moving some of the become data
(prompt and success_key) into the ConnectionInformation object so they don't
need to be passed around needlessly.
There was a confusion between the valid values for defaults.gathering
(explicit/implicit/smart) and a play's gather_facts setting (boolean),
which resulted in gathering=explicit being ignored.
* Moving connection creation until after the task is post_validated,
to make sure all fields are properly templated (#11230)
* Fixing problems related to the connection method and remote address
lookup on the delegated-to host
Fixes#11230
If the syntax var={{something}} is used, that can be templated too
early in the post_validation, leading the debug module to fail when it
tries to template the same value in turn.
