* Fix tmpdir on non root become
- also avoid exception if tmpdir and remote_tmp are None
- give 'None' on deescalation so tempfile will fallback to it's default behaviour
and use system dirs
- fix issue with bad tempdir (not existing/not createable/not writeable)
i.e nobody and ~/.ansible/tmp
- added tests for blockfile case
* Revert "Temporarily revert c119d54"
This reverts commit 5c614a59a6.
* changes based on PR feedback and changelog fragment
* changes based on the review
* Fix tmpdir when makedirs failed so we just use the system tmp
* Let missing remote_tmp fail
If remote_tmp is missing then there's something more basic wrong in the
communication from the controller to the module-side. It's better to
be alerted in this case than to silently ignore it.
jborean and I have independently checked what happens if the user sets
ansible_remote_tmp to empty string and !!null and both cases work fine.
(null is turned into a default value controller-side. empty string
triggers the warning because it is probably not a directory that the
become user is able to use).
* Create base class for network-style connections
* clean up some differences
* Move NetworkConnectionBase
* Tweak netconf for tests
* Tweak when network_os is checked to avoid failing tests
* Pull back exec_command
* fix minor issues with debug and item labels
- no more `item=None`, we always have a label now
- debug should only show expected information, either msg= or the var in var=
- also fixed method name, deprecated misleading _get_item
* Refactor ios cliconf plugin and ios_config module
* Refactor ios cliconf plugin to support generic network_config module
* Refactor ios_config module to work with cliconf api's
* Enable command and response logging in cliconf pulgin
* cliconf api documentation
* Fix unit test and other minor changes
* Doc update
* Fix CI failure
* Add default flag related changes
* Minor changes
* redact input command logging by default
* All instances of local connection should use _remote_is_local=True. Fixes#40551
* Switch to instance attribute for synchronize
* Add test that shows that synchronize _remote_is_local addresses tmpdir building
* Fix setting the cache when refresh_cache or --flush-cache are used
* Use jsonify function that handles datetime objects in jsonfile cache plugin
* Don't access self._options directly
* Add initial integration tests for aws_ec2 inventory plugin
* Add CI alias
* Fix and add a few more unit tests
* Add integration tests for constructed
* Fix typo
* Use inventory config templates
* Collect all instances that are not terminated by default
* Create separate playbook for setting up the VPC, subnet, security group, and finding an image for the host
Create a separate playbook for removing the resources
* Allow easier grouping by region and add an example
* use a unified json encode/decode that can handle unsafe and vault
* winrm: add better exception handling for krb5 auth with pexpect
* Added changelog fragment
* Added exception handler in case kinit path isn't valid, added test cases
* fixed for Python 2 compatibility
* win_updates: add scheduled tasks back in for older hosts
* Fixed up typo in category name error message
* Fixed up some minor issues after merge
* added changelog fragment
* Default to become but add override to use scheduled tasks
* Added basic unit tests for win_updates
* fix minor typos
* add pytest_cache to gitignore
* onepassword lookup plugin
* fix linter/style test complaints
* second pass at making pycodestyle happy
* use json module instead of jq
* update copyrights, license & version added
* fix python2 compatibility
* doh. fix spacing issue.
* use standard ansible exception
* remove potentially problematic stdin argument
* actually call assertion method
* add support for top-level fields
* make vault uuids pedantically consistent in fixture
* fix new style issues
* ability specify section & correct case handling
* improve error handling
* add onepassword_raw plugin
* Add maintainer info
* Move common code to module_utils/onepassword.py
* Load raw data JSON data for easier use in Ansible
* Put OnePass class back inside lookup plugin
There is no good place for sharing code across lookups currently.
* Remove debugging code in unit tests
* Patche proper module in raw unit tests
* Add changelog entry
Co-authored-by: Scott Buchanan <sbuchanan@ri.pn>
* WIP Pull persistent connection parameters via get_option
* Fix pep8
* Add use_persistent_connection setting to paramiko_ssh plugin
* Add vars section to persistent_command_timeout setting and prevail provider values over config manager
* Use persistent_command_timeout on network_cli instead of timeout
* Fix unit tests
If we don't call loader to get network_cli, then _load_name is never
set and we get KeyError.
* Pull persistent_command_timeout via config manager for ios connection local
* Pull persistent_command_timeout via config manager on connection local
* create module tmpdir based on remote_tmp
* Source remote_tmp from controller if possible
* Fixed sanity test and not use lambda
* Added expansion of env vars to the remote tmp
* Fixed sanity issues
* Added note around shell remote_tmp option
* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
* Adding slx_command module and supporting module_utils.
This commit adds the slx_command module and tests as well as the
required slxos module_utils.
* Update copyright in header
* Adding missing module init
* Cleaning up shebangs/licensing.
* Incorporating feedback
Removing reference to `waitfor` alias in `slxos_command` module.
Adding `Extreme Networks` to `short_description` of `slxos_command` module.
* Adding cliconf tests
* Fixing 3.X tests
* Adding docstrings to test methods for slxos cliconf tests
* Adding slxos terminal tests
* Adding slxos module_utils tests
* Adding Extreme Networks team members to BOTMETA.yml
This offers an optimization that allows loading larger
inventories of various structure by improving the
scaling laws involved for adding hosts and groups.
The primary speed benefit is the elimination of duplicate
recusion from traversing converging paths.
* aws ssm parameter lookup test case - fails demonstrating no exception when parameter missing
* aws ssm parameter lookup - fail in case parameter doesn't exist
* aws ssm parameter lookup test case - failing case for nice return from path lookup
* aws ssm parameter lookup - convert incoming taglist to a key-value dictionary
* aws ssm parameter lookup - pep8 / style clean up
* aws_ssm lookup plugin rewrite for more standard interface
* aws_ssm module and lookup - introduce integration test and fix:
* aws_ssm module and lookup - error case integraton test and many PEP8 and other cleanups
* aws ssm parameter lookup - Various fixes in response to review + recursive fix & test
* aws ssm parameter lookup - more in response to review - shertel/abadger
* aws ssm parameter lookup unit test - move to mocker according to abadger
* aws ssm parameter lookup - integrate with new documentation fragment
* aws ssm parameter lookup - accept either aws_profile or boto_profile
* aws ssm parameter lookup - eliminate lookup document fragment until env vars are fixed later
* Imported lookup plugin from Role
* Plugin cleanup, including:
* Use existing Python YAML parsing
* Remove environment variables as connection options
* Added initial debugging information
* Reworked the lookup plugin using the Python Request library. As it's available through Ansible, it makes communication with Conjur much more straight forward.
* Removed un-used libraries
* Fixed linting issues
* Standardized output on `format` and insure it works for 2.6, 2.7, and 3.x.
* Use quote_plus from the six library for improved python 2/3 behavior.
* Refactored identity & configuration to prefer user's file. This also includes a refactor to remove an un-needed dictionary merge method.
* Removed `requests` in favor of `ansible.module_utils.urls`.
* Refactored netrc loading to warn if host is not present.
* Tests and a refactor to support easier testing.
* Added reference to website
* Fixed two linting errors
* Fixed an extra line found by linting
* Updated file write to use binary to insure config files are written correctly
* Resolved linting issues
* Refactored config & identity loading to take advantage of plugin options
* Cleanup a bunch of small items caught by linting
* Removed extra line caught by linting
* Swapped in pytest and added some tests with mocked network responses
* Pushing to see if this approach works better...
* Refactored be open_url mocking based on feedback
* Fixed a couple linting issues & refactored mocking into each method to attempt to resolve a failing test
* Use a generic MagicMock for python 2.6
* Fixes doc typo
require -> required
* Use `type: path` in identity_file and config_file
Also removes `expanduser` calls below (which will now be called automatically on
paths.)
* Defines maintainers for conjur_variable plugin
* BOTMETA.yml:
** defines $team_cyberark_conjur as maintainers of Conjur Variable plugin
** adds myself and @jvanderhoof to that team
* Adds URLs to relevant documentation for Conjur Variable lookup plugin
* Clarifies "the server," "the machine" -> "controlling host"
The machine identity used is that of the Ansible controlling host, not any
server being provisioned or instructed. This documentation change aims to make
that relationship clear.
* Adds response code to exception message on authentication failure
* Enhances exception messages to specify the controlling host
These error messages are less likely to confuse a user as to which machine is
associated with the files, identities, and configurations being described.
* Adds ANSIBLE_METADATA for Conjur variable lookup plugin
aws_ec2 is an inventory plugin intended to replace uses of the `contrib/inventory/ec2.py`
inventory script.
For advanced grouping, please use the `constructed` inventory plugin.
* allow shells to have per host options, remote_tmp
added language to shell
removed module lang setting from general as plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e
- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action
add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control
* Mostly revert expand_user as it's not quite working.
This was an additional feature anyhow.
Kept the use of pwd as a fallback but moved it to a second ssh
connection. This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.
(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)
* fixed script and other action plugins
ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
* Cache tasks as they are queued instead of en masse
This also moves the task caching from the PlayIterator to the
StrategyBase class, where it makes more sense (and makes it easier
to not have to change the strategy class methods leading to an API
change).
Fixes#31673
* Cleaning up unit tests due to 502ca780
* Provide a way to explicitly invoke the debugger with in the debug strategy
* Merge the debugger strategy into StrategyBase
* Fix some logic, pin to a single result
* Make redo also continue
* Make sure that if the debug closure doesn't need to process the result, that we still return it
* Fix failing tests for the strategy
* Clean up messages from debugger and exit code to match bin/ansible
* Move the FieldAttribute higher, to apply at different levels
* make debugger a string, expand logic
* Better host state rollbacks
* More explicit debugger prompt
* ENABLE_TASK_DEBUGGER should be boolean, and better docs
* No bare except, add pprint, alias h, vars to task_vars
* _validate_debugger can ignore non-string, that can be caught later
* Address issue if there were no previous tasks/state, and use the correct key
* Update docs for changes to the debugger
* Guard against a stat going negative through use of decrement
* Add a few notes about using the debugger on the free strategy
* Add changelog entry for task debugger
* Add a few versionadded indicators and a note about vars -> task_vars
* Change cast_list_to_dict to more generic rekey_on_member
cast_list_to_dict was taking an arbitrary data format in and returning
an arbitrary data format out. Rework this to be a more generic function
which creates a dict of dicts based on a member of the dict.
Remove cast_dict_to_list since rekey_on_member handles the use cases we
know about and cast_dict_to_list suffers from the same problems as
cast_list_to_dict. If this is still needed we could think about filters
we could add to do this in a short jinja2 pipeline.
* Fix bare excepts (bare excepts even catch sys.exit())
* Fix _clean_result for debug callback with 1 char var names
The check in _clean_results was removing any keys that happened
to be one of the chars in ('invocation') instead of the string
'invocation'. This was meant to be a tuple but there was no comma
so the for iterated the string instead of the tuple.
Introduced in 9dba580204fcc7d95d4ca2ea2ae4ce23fe28c2ed
Update unit test to catch this.
Fixes#33723
* Use .pop() to remove invocation from results dict
In base callback _clean_results, simplify the way the
'invocation' item is removed.
Add some more unit tests.
* add type manipulation
* update for py3
* add tests for list to dict
* Add dict to list tests
* Update tests to search for regex in response
* pep8 clean up
* update Exception for py3
* update test to be py2/3 compat
* update for py26 compat
* potential fix for py3 and py26
* potential fix for py3 and py26 take2
* add new line to kick off shippable
* remove cache file created
* fix filter name
* add space for shipable
* Adding aliases for field attributes and renaming async attribute
As of Python 3.7, the use of async raises an error, whereas before the use
of the reserved word was ignored. This adds an alias field for field attrs
so that both async and async_val (interally) work. This allows us to be
backwards-compatible with 3rd party plugins that may still reference Task.async,
but for the core engine to work on Py3.7+.
* Remove files fixed for 'async' usage from the python 3.7 skip list
* Allow the user to circumvent adding -tt on ssh commands to help aid in
debugging ssh related problems.
* Move config to the plugin
* Set version_added
* Change yaml section to "connection"
* Fix ssh unit tests
* Add new filter to parse xml output for network use cases
Fixes#31026
* Add parse_xml filter
* Add documentation for parse_xml filter
* Edited for clarity.
* Fix review comment and add unit tests
* Fix unit test CI failure
* Fix CI issues
* Fix unit test failures
* Fix review comments
* More copy edits.
* included inventory and callback in new config
allow inventory to be configurable
updated connection options settings
also updated winrm to work with new configs
removed now obsolete set_host_overrides
added notes for future bcoca, current one is just punting, it's future's problem
updated docs per feedback
added remove group/host methods to inv data
moved fact cache from data to constructed
cleaner/better options
fix when vars are added
extended ignore list to config dicts
updated paramiko connection docs
removed options from base that paramiko already handles
left the look option as it is used by other plugin types
resolve delegation
updated cache doc options
fixed test_script
better fragment merge for options
fixed proxy command
restore ini for proxy
normalized options
moved pipelining to class
updates for host_key_checking
restructured mixins
* fix typo
* implements jsonrpc message passing for ansible-connection
* implements more generic mechanism for persistent connections
* starts persistent connection in task_executor if enabled and supported
* supports using network_cli as top level connection plugin
* enhances logging for persistent connection to stdout
* Update action plugins
* Fix Python3 RPC
* Fix Junos bytes<-->str issues
* supports using netconf as top level connection plugin
* Error message when running netconf on an unsupported platform
* Update tests
* Fix `authorize: yes` for `connection: local`
* Handle potentially JSON data in terminal
* Add clarifying detail if possible on ConnectionError
* better cleanup on task results display
callbacks get 'clean' copy of result objects
moved cleanup into result object itself
removed now redundant callback cleanup
moved no_log tests
* moved import as per feedback
* Ansible Config part2
- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code
skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution
* killed extra space with extreeme prejudice
cause its an affront against all that is holy that 2 spaces touch each other!
shippable timing out on some images, but merging as it passes most
* [password] _random_password -> random_password and moved to util/encrypt.py
* [passwordstore] Use built-in random_password instead of pwgen utility
* [passwordstore] Add integration tests
1fe67f9 introduced retries to the ssh connection put file and fetch
file. Unfortunately, that change broke the smart transport because it
started raising exceptions instead of returning from _run(). This
breakage is documented in #23711.
An attempt to fix it was made at #23717 but the first attempt was
objected to as needing to touch too much code. The second attmept was
objected to as smart was forced to encapsulate retries (thus retrying
a sftp "rety" times before trying scp "retry" times and then finally
moving onto piped). This third attempt has retries encapsulate smart.
So each sub-transport is tried once and if all three fail, another retry
attempt is made which tries each of the three again.
Fixes#23711Fixes#23717
* add first, last and next usable
* add usable ip filters
* add size usable, range usable and wildcard
* add ip prefix and netmask filter
* add network formatting and check if ip in subnet
* clean up order, add comments
* fix pep8
* update format by index
* clean up and updates from jmcgill298
* test/: PEP8 compliancy
- Make PEP8 compliant
* Python3 chokes on casting int to bytes (#24952)
But if we tell the formatter that the var is a number, it works
* eos python3 changes
* changes to convert response from byte to text
* Add dellos6 python3 changes
Make `execute_command` arguments and its
return value complaint to PY3 changes
made in PR #24431
* Fix py3 prompt issue for invalid show command
* Fix review comments
* Add generic fix for error prompt in py3
* Fix CI issue
* Fix network_cli unit test failure
* draft new inventory plugin arch, yaml sample
- split classes, moved out of init
- extra debug statements
- allow mulitple invenotry files
- dont add hosts more than once
- simplified host vars
- since now we can have multiple, inventory_dir/file needs to be per host
- ported yaml/script/ini/virtualbox plugins, dir is 'built in manager'
- centralized localhost handling
- added plugin docs
- leaner meaner inventory (split to data + manager)
- moved noop vars plugin
- added 'postprocessing' inventory plugins
- fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them
- grouphost_vars loading as inventory plugin (postprocessing)
- playbook_dir allways full path
- use bytes for file operations
- better handling of empty/null sources
- added test target that skips networking modules
- now var manager loads play group/host_vars independant from inventory
- centralized play setup repeat code
- updated changelog with inv features
- asperioribus verbis spatium album
- fixed dataloader to new sig
- made yaml plugin more resistant to bad data
- nicer error msgs
- fixed undeclared group detection
- fixed 'ungrouping'
- docs updated s/INI/file/ as its not only format
- made behaviour of var merge a toggle
- made 'source over group' path follow existing rule for var precedence
- updated add_host/group from strategy
- made host_list a plugin and added it to defaults
- added advanced_host_list as example variation
- refactored 'display' to be availbe by default in class inheritance
- optimized implicit handling as per @pilou's feedback
- removed unused code and tests
- added inventory cache and vbox plugin now uses it
- added _compose method for variable expressions in plugins
- vbox plugin now uses 'compose'
- require yaml extension for yaml
- fix for plugin loader to always add original_path, even when not using all()
- fix py3 issues
- added --inventory as clearer option
- return name when stringifying host objects
- ajdust checks to code moving
* reworked vars and vars precedence
- vars plugins now load group/host_vars dirs
- precedence for host vars is now configurable
- vars_plugins been reworked
- removed unused vars cache
- removed _gathered_facts as we are not keeping info in host anymore
- cleaned up tests
- fixed ansible-pull to work with new inventory
- removed version added notation to please rst check
- inventory in config relative to config
- ensures full paths on passed inventories
* implicit localhost connection local
Fix for persistent connection plugin on Python3. Note that fixes are also needed to each terminal plugin. This PR only fixes the ios terminal (as proof that this approach is workable.) Future PRs can address the other terminal types.
* On Python3, pickle needs to work with byte strings, not text strings.
* Set the pickle protocol version to 0 because we're using a pty to feed data to the connection plugin. A pty can't have control characters. So we have to send ascii only. That means
only using protocol=0 for pickling the data.
* ansible-connection isn't being used with py3 in the bug but it needs
several changes to work with python3.
* In python3, closing the pty too early causes no data to be sent. So
leave stdin open until after we finish with the ansible-connection
process.
* Fix typo using traceback.format_exc()
* Cleanup unnecessary StringIO, BytesIO, and to_bytes calls
* Modify the network_cli and terminal plugins for py3 compat. Lots of mixing of text and byte strings that needs to be straightened out to be compatible with python3
* Documentation for the bytes<=>text strategy for terminal plugins
* Update unittests for more bytes-oriented internals
Fixes#24355
Ansible will now automatically retry a connection if SSH returns an error:
mux_client_hello_exchange: write packet: Broken pipe
This is probably a bug in SSH, but because it's safe to retry this
connection there is no need for Ansible to fail because of it.
* Update module_utils.six to latest
We've been held back on the version of six we could use on the module
side to 1.4.x because of python-2.4 compatibility. Now that our minimum
is Python-2.6, we can update to the latest version of six in
module_utils and get rid of the second copy in lib/ansible/compat.
* Move retry logic into _ssh_retry decorator, and apply to exec_command, put_file and fetch_file
* Update tests to reflect change
* Move _ssh_retry to _run, and update tests to reflect
* piped should use exec_command instead of removed _exec_command
* Rework tests to support selectors instead of select.select
* fixes error where eos would close the cli shell
* fixes network_cli connection plugin to check before calling open_shell()
* fixes json commands being sent over eapi
* removes unneeded supports_multiplexing var
* refactors terminal_prompts_re to terminal_stdout_re
* refactors terminal_errors_re to terminal_stderr_re
* updates network_cli unit test cases
The python 'future' module provides a 'builtins' package
to emulate the py3 'builtins' modules. If installed, the
unit tests that reference builtins.__import__ fail because
the future 'builtins' is imported and it is missing __import__.
Use ansible.compat.six.moves.builtins instead of importing
'builtins' or __builtin__
Fixes#14996
- centralized skipping
- also fixed module name broken by previous refactor
- let action modules handle async processing
- moved async into base action class's module exec
- action plugins can now run final action as async
- actually skip copy if base skips
- fixed normal for new paths
- ensure internal stat is never async
- default poll to 10 as per docs
- added hint for callback fix on poll
- restructured late tmp, now a pipeline query
- moving action handler to connection as networking does
- fixed network assumption invocation is always passed
- centralized key cleanup, normalized internal var
- _supress_tmpdir_delete now in _ansible_xxx and gets removed from results
- delay internal key removal till after we use em
- nicer tmp removing, using existing methods
- moved cleanup tmp flag to mking tmp func
At the moment, this change will use EPoll on Linux, KQueue on *BSDs,
etc, so it should alleviate problems with too many open file
descriptors.
* Bundle a copy of selectors2 so that we have the selectors API everywhere.
* Add licensing information to selectors2 file so it's clear what the
licensing terms and conditions are.
* Exclude the bundled copy of selectors2 from our boilerplate code-smell test
* Rewrite ssh_run tests to attempt to work around problem with mocking
select on shippable
Fixes#14143
