1
0
Fork 0
mirror of https://github.com/ansible-collections/community.general.git synced 2024-09-14 20:13:21 +02:00
Commit graph

228 commits

Author SHA1 Message Date
mikedlr
d31ded47fb Aws ssm multiple fixes (#35569)
* aws ssm parameter lookup test case - fails demonstrating no exception when parameter missing

* aws ssm parameter lookup - fail in case parameter doesn't exist

* aws ssm parameter lookup test case - failing case for nice return from path lookup

* aws ssm parameter lookup - convert incoming taglist to a key-value dictionary

* aws ssm parameter lookup - pep8 / style clean up

* aws_ssm lookup plugin rewrite for more standard interface

* aws_ssm module and lookup - introduce integration test and fix:

* aws_ssm module and lookup - error case integraton test and many PEP8 and other cleanups

* aws ssm parameter lookup - Various fixes in response to review + recursive fix & test

* aws ssm parameter lookup - more in response to review - shertel/abadger

* aws ssm parameter lookup unit test - move to mocker according to abadger

* aws ssm parameter lookup - integrate with new documentation fragment

* aws ssm parameter lookup - accept either aws_profile or boto_profile

* aws ssm parameter lookup - eliminate lookup document fragment until env vars are fixed later
2018-02-06 17:41:46 -05:00
Jason Vanderhoof
7c8e365dff Conjur Lookup Plugin (#34280)
* Imported lookup plugin from Role

* Plugin cleanup, including:
* Use existing Python YAML parsing
* Remove environment variables as connection options
* Added initial debugging information

* Reworked the lookup plugin using the Python Request library.  As it's available through Ansible, it makes communication with Conjur much more straight forward.

* Removed un-used libraries

* Fixed linting issues

* Standardized output on `format` and insure it works for 2.6, 2.7, and 3.x.

* Use quote_plus from the six library for improved python 2/3 behavior.

* Refactored identity & configuration to prefer user's file. This also includes a refactor to remove an un-needed dictionary merge method.

* Removed `requests` in favor of `ansible.module_utils.urls`.

* Refactored netrc loading to warn if host is not present.

* Tests and a refactor to support easier testing.

* Added reference to website

* Fixed two linting errors

* Fixed an extra line found by linting

* Updated file write to use binary to insure config files are written correctly

* Resolved linting issues

* Refactored config & identity loading to take advantage of plugin options

* Cleanup a bunch of small items caught by linting

* Removed extra line caught by linting

* Swapped in pytest and added some tests with mocked network responses

* Pushing to see if this approach works better...

* Refactored be open_url mocking based on feedback

* Fixed a couple linting issues & refactored mocking into each method to attempt to resolve a failing test

* Use a generic MagicMock for python 2.6

* Fixes doc typo

require -> required

* Use `type: path` in identity_file and config_file

Also removes `expanduser` calls below (which will now be called automatically on
paths.)

* Defines maintainers for conjur_variable plugin

* BOTMETA.yml:
** defines $team_cyberark_conjur as maintainers of Conjur Variable plugin
** adds myself and @jvanderhoof to that team

* Adds URLs to relevant documentation for Conjur Variable lookup plugin

* Clarifies "the server," "the machine" -> "controlling host"

The machine identity used is that of the Ansible controlling host, not any
server being provisioned or instructed. This documentation change aims to make
that relationship clear.

* Adds response code to exception message on authentication failure

* Enhances exception messages to specify the controlling host

These error messages are less likely to confuse a user as to which machine is
associated with the files, identities, and configurations being described.

* Adds ANSIBLE_METADATA for Conjur variable lookup plugin
2018-01-23 10:04:57 -06:00
Toshio Kuratomi
8c7bd8beb4 Set the tempdir on the shell mock object so that it is json serializable 2018-01-22 23:36:34 -08:00
Sloane Hertel
b94198f9de [cloud] New inventory plugin: aws_ec2 (#34416)
aws_ec2 is an inventory plugin intended to replace uses of the `contrib/inventory/ec2.py`
inventory script. 

For advanced grouping, please use the `constructed` inventory plugin.
2018-01-22 18:49:57 -05:00
Matt Martz
29c1d5cb5d Ensure that the become password is written on py3 in the ssh connection plugin. Fixes #34727 2018-01-18 09:32:22 -08:00
Brian Coca
bbd6b8bb42 Temporary (#31677)
* allow shells to have per host options, remote_tmp

added language to shell
removed module lang setting from general as  plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e

- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action

add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control

* Mostly revert expand_user as it's not quite working.

This was an additional feature anyhow.

Kept the use of pwd as a fallback but moved it to a second ssh
connection.  This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.

(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)

* fixed script and other action plugins

ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
2018-01-15 21:15:04 -08:00
James Cammarata
b107e397cb
Cache tasks as they are queued instead of en masse (#34752)
* Cache tasks as they are queued instead of en masse

This also moves the task caching from the PlayIterator to the
StrategyBase class, where it makes more sense (and makes it easier
to not have to change the strategy class methods leading to an API
change).

Fixes #31673

* Cleaning up unit tests due to 502ca780
2018-01-15 09:49:58 -06:00
Matt Martz
d1846425db
Provide a way to explicitly invoke the debugger (#34006)
* Provide a way to explicitly invoke the debugger with in the debug strategy

* Merge the debugger strategy into StrategyBase

* Fix some logic, pin to a single result

* Make redo also continue

* Make sure that if the debug closure doesn't need to process the result, that we still return it

* Fix failing tests for the strategy

* Clean up messages from debugger and exit code to match bin/ansible

* Move the FieldAttribute higher, to apply at different levels

* make debugger a string, expand logic

* Better host state rollbacks

* More explicit debugger prompt

* ENABLE_TASK_DEBUGGER should be boolean, and better docs

* No bare except, add pprint, alias h, vars to task_vars

* _validate_debugger can ignore non-string, that can be caught later

* Address issue if there were no previous tasks/state, and use the correct key

* Update docs for changes to the debugger

* Guard against a stat going negative through use of decrement

* Add a few notes about using the debugger on the free strategy

* Add changelog entry for task debugger

* Add a few versionadded indicators and a note about vars -> task_vars
2018-01-09 13:50:07 -06:00
Matt Martz
aece0818c4
Don't cache duplicate plugin names (#34420)
* Don't cache duplicate plugin names. Fixes #33484

* Add tests for duplicate plugin filenames, to showcase what happens
2018-01-04 16:47:23 -06:00
Pierre-Louis Bonicoli
737920cd89 YAML inventory unit test: add tests
Relates: #33878
2017-12-20 16:59:53 -05:00
Pierre-Louis Bonicoli
35a4fe8546 YAML inventory unit test: add tests 2017-12-20 16:59:53 -05:00
Pierre-Louis Bonicoli
8dff0b091e Fix YAML inventory unit test 2017-12-20 16:59:53 -05:00
John R Barker
cfb15b0272 Revert "YAML inventory unit test: fix test inventory format (#33828)" (#33869)
This reverts commit dfb2f346d8.

https://github.com/ansible/ansible/pull/33828
2017-12-13 14:19:19 -05:00
Pilou
dfb2f346d8 YAML inventory unit test: fix test inventory format (#33828)
* Fix YAML inventory unit test

* YAML inventory unit test: add checks

* YAML: add hosts without any group to ungrouped
2017-12-13 15:30:59 +00:00
Toshio Kuratomi
155f36bbd8
Rekey on member (#33836)
* Change cast_list_to_dict to more generic rekey_on_member

cast_list_to_dict was taking an arbitrary data format in and returning
an arbitrary data format out.  Rework this to be a more generic function
which creates a dict of dicts based on a member of the dict.

Remove cast_dict_to_list since rekey_on_member handles the use cases we
know about and cast_dict_to_list suffers from the same problems as
cast_list_to_dict.  If this is still needed we could think about filters
we could add to do this in a short jinja2 pipeline.

* Fix bare excepts (bare excepts even catch sys.exit())
2017-12-12 19:02:15 -08:00
Adrian Likins
a57d6a4206 Use pop in cb clean results (#33779)
* Fix _clean_result for debug callback with 1 char var names

The check in _clean_results was removing any keys that happened
to be one of the chars in ('invocation') instead of the string
'invocation'. This was meant to be a tuple but there was no comma
so the for iterated the string instead of the tuple.

Introduced in 9dba580204fcc7d95d4ca2ea2ae4ce23fe28c2ed

Update unit test to catch this.

Fixes #33723

* Use .pop() to remove invocation from results dict

In base callback _clean_results, simplify the way the
'invocation' item is removed.

Add some more unit tests.
2017-12-11 17:41:12 -05:00
Ganesh Nalawade
95044d69c7
Fix unit test failure for netconf connection plugin (#33547) 2017-12-05 12:04:48 +05:30
Toshio Kuratomi
e499bccbaa
Port some tests away from nose as examples (#33437)
We don't need to use both nose and pytest.  Once we get rid of all uses
of nose we can remove the extra dependency
2017-11-30 14:04:09 -08:00
Nathaniel Case
a1da3dc997
Auth to become (#33319)
* terminal.on_(de)?authorize -> terminal.on_(un)?become

* Update network_cli

* Readd on_(de)?authorize as wrappers

* Update tests
2017-11-28 16:28:58 -05:00
Ken Celenza
748107d369 Type manipulation (#28446)
* add type manipulation

* update for py3

* add tests for list to dict

* Add dict to list tests

* Update tests to search for regex in response

* pep8 clean up

* update Exception for py3

* update test to be py2/3 compat

* update for py26 compat

* potential fix for py3 and py26

* potential fix for py3 and py26 take2

* add new line to kick off shippable

* remove cache file created

* fix filter name

* add space for shipable
2017-11-24 15:26:29 -05:00
James Cammarata
d8ae4dfbf2 Adding aliases for field attributes and renaming async attribute (#33141)
* Adding aliases for field attributes and renaming async attribute

As of Python 3.7, the use of async raises an error, whereas before the use
of the reserved word was ignored. This adds an alias field for field attrs
so that both async and async_val (interally) work. This allows us to be
backwards-compatible with 3rd party plugins that may still reference Task.async,
but for the core engine to work on Py3.7+.

* Remove files fixed for 'async' usage from the python 3.7 skip list
2017-11-22 12:35:58 -08:00
jctanner
218987eac1
ANSIBLE_SSH_USETTY configuration option (#33148)
* Allow the user to circumvent adding -tt on ssh commands to help aid in
debugging ssh related problems.
* Move config to the plugin
* Set version_added
* Change yaml section to "connection"
* Fix ssh unit tests
2017-11-22 11:19:43 -05:00
Ganesh Nalawade
0ddf092ae3
Add new filter to parse xml output for network use cases (#31562)
* Add new filter to parse xml output for network use cases

Fixes #31026
*  Add parse_xml filter
*  Add documentation for parse_xml filter

* Edited for clarity.

* Fix review comment and add unit tests

* Fix unit test CI failure

* Fix CI issues

* Fix unit test failures

* Fix review comments

* More copy edits.
2017-11-21 12:16:18 +05:30
Brian Coca
23b1dbacaf
Config continued (#31024)
* included inventory and callback in new config

allow inventory to be configurable
updated connection options settings
also updated winrm to work with new configs
removed now obsolete set_host_overrides
added notes for future bcoca, current one is just punting, it's future's problem
updated docs per feedback
added remove group/host methods to inv data
moved fact cache from data to constructed
cleaner/better options
fix when vars are added
extended ignore list to config dicts
updated paramiko connection docs
removed options from base that paramiko already handles
left the look option as it is used by other plugin types
resolve delegation
updated cache doc options
fixed test_script
better fragment merge for options
fixed proxy command
restore ini for proxy
normalized options
moved pipelining to class
updates for host_key_checking
restructured mixins

* fix typo
2017-11-16 13:49:57 -05:00
Frederic Lepied
35f79370e1 mock ncclient import in test_connection.py (#32786) 2017-11-10 11:53:04 -05:00
Nathaniel Case
9c0275a879
Connection plugins network_cli and netconf (#32521)
* implements jsonrpc message passing for ansible-connection

* implements more generic mechanism for persistent connections
* starts persistent connection in task_executor if enabled and supported
* supports using network_cli as top level connection plugin
* enhances logging for persistent connection to stdout

* Update action plugins

* Fix Python3 RPC

* Fix Junos bytes<-->str issues

* supports using netconf as top level connection plugin

* Error message when running netconf on an unsupported platform
* Update tests

* Fix `authorize: yes` for `connection: local`

* Handle potentially JSON data in terminal

* Add clarifying detail if possible on ConnectionError
2017-11-09 15:04:40 -05:00
Brian Coca
db749de5b8 namespace facts
updated action plugins to use new guranteed facts
updated tests to new data clean
added cases for ansible_local and some docstrings
2017-11-09 09:48:14 -05:00
Chris Meyers
cf938e9992 tests for InventoryModule error conditions (#31381)
* tests for InventoryModule error conditions

* modified unicode in tests to ahear to Ansible best practices

* flake8 fixes
2017-10-16 18:52:44 -04:00
Brian Coca
01b6c7c9c6 better cleanup on task results display (#27175)
* better cleanup on task results display

callbacks get 'clean' copy of result objects
moved cleanup into result object itself
removed now redundant callback cleanup
moved no_log tests

* moved import as per feedback
2017-10-16 09:44:11 -04:00
Hervé Beraud
db70eeb913 Feature/test inventory (#30707)
* [test] move inventory test to right path
* [feat] add unit test for yaml inventory plugin
2017-10-02 23:01:25 -07:00
Matt Clay
dbf9634b1b Enable more pylint rules and fix reported issues. (#30543)
* Enable pylint invalid-encoded-data test.
* Enable pylint anomalous-unicode-escape-in-string test.
2017-09-19 00:45:35 -07:00
Toshio Kuratomi
75249e311e Fix set theory filters for python3
The set theory filters need to use lists rather than generators on python3.

Also add unit tests for most of the mathstuff filters

Fixes #26494
2017-09-09 17:38:39 -07:00
Nicolas Porcel
ba420c47cd synchronize: Use private_key and ssh_user with delegate_to (Fixes #16215) (#21686) 2017-08-24 21:09:15 -04:00
Brian Coca
f921369445 Ansible Config part2 (#27448)
* Ansible Config part2

- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code

skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution

* killed extra space with extreeme prejudice

cause its an affront against all that is holy that 2 spaces touch each other!

shippable timing out on some images, but merging as it passes most
2017-08-15 16:38:59 -04:00
3onyc
554496c404 [passwordstore] Use builtin _random_password function instead of pwgen (#25843)
* [password] _random_password -> random_password and moved to util/encrypt.py
* [passwordstore] Use built-in random_password instead of pwgen utility
* [passwordstore] Add integration tests
2017-08-14 15:19:40 -07:00
Toshio Kuratomi
3edac559d3 the smart transport is broken by ssh retry code
1fe67f9 introduced retries to the ssh connection put file and fetch
file.  Unfortunately, that change broke the smart transport because it
started raising exceptions instead of returning from _run().  This
breakage is documented in #23711.

An attempt to fix it was made at #23717 but the first attempt was
objected to as needing to touch too much code.  The second attmept was
objected to as smart was forced to encapsulate retries (thus retrying
a sftp "rety" times before trying scp "retry" times and then finally
moving onto piped).  This third attempt has retries encapsulate smart.
So each sub-transport is tried once and if all three fail, another retry
attempt is made which tries each of the three again.

Fixes #23711
Fixes #23717
2017-08-14 08:08:00 -07:00
Toshio Kuratomi
802c9efaa3 Disable abstract-class-instantiated for smoketests
These tests are actually checking that the classes will error out if
they are still abstracted and instantiated
2017-07-25 15:58:23 -07:00
Gabor Lekeny
7eab802669 SSH fails with '"parsed": false' error message
Fixes: #15436
2017-07-20 08:09:42 -07:00
Ken Celenza
31b6ac896d Kc update ip filter (#26566)
* add first, last and next usable

* add usable ip filters

* add size usable, range usable and wildcard

* add ip prefix and netmask filter

* add network formatting and check if ip in subnet

* clean up order, add comments

* fix pep8

* update format by index

* clean up and updates from jmcgill298
2017-07-12 17:17:58 +01:00
Ganesh Nalawade
3bd8dbb53d Add support for netconf network discovery (#25435)
* Add support for netconf network discovery

* Fix unit test failure
2017-06-28 11:07:38 +05:30
Matt Davis
36ad934156 re-enable non-pipelined mode for Powershell (#25012)
* fixes #23986
* fixes 3rd-party Windows connection plugins that don't support pipelining (eg awsrun)
2017-06-26 22:58:09 -07:00
Abhijeet Kasurde
b89cb95609 Fix spelling mistakes (comments only) (#25564)
Original Author : klemens <ka7@github.com>

Taking over previous PR as per
https://github.com/ansible/ansible/pull/23644#issuecomment-307334525

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2017-06-12 07:55:19 +01:00
Ganesh Nalawade
6215922889 Add support for cliconf and netconf plugin (#25093)
* ansible-connection refactor and action plugin changes
* Add cliconf plugin for eos, ios, iosxr, junos, nxos, vyos
* Add netconf plugin for junos
* Add jsonrpc support
* Modify network_cli and netconf connection plugin
* Fix py3 unit test failure
* Fix review comment
* Minor fixes
* Fix ansible-connection review comments
* Fix CI issue
* platform_agnostic related changes
2017-06-06 13:56:25 +05:30
Dag Wieers
4efec414e7 test/: PEP8 compliancy (#24803)
* test/: PEP8 compliancy

- Make PEP8 compliant

* Python3 chokes on casting int to bytes (#24952)

But if we tell the formatter that the var is a number, it works
2017-05-30 18:05:19 +01:00
Ganesh Nalawade
825d9df5ea Add eos changes for Python3 (#24600)
* eos python3 changes

* changes to convert response from byte to text

* Add dellos6 python3 changes
Make `execute_command` arguments and its
return value complaint to PY3 changes
made in PR #24431

* Fix py3 prompt issue for invalid show command
* Fix review comments
* Add generic fix for error prompt in py3
* Fix CI issue
* Fix network_cli unit test failure
2017-05-24 19:40:38 +05:30
Brian Coca
8f97aef1a3 Transition inventory into plugins (#23001)
* draft new inventory plugin arch, yaml sample

 - split classes, moved out of init
 - extra debug statements
 - allow mulitple invenotry files
 - dont add hosts more than once
 - simplified host vars
 - since now we can have multiple, inventory_dir/file needs to be per host
 - ported yaml/script/ini/virtualbox plugins, dir is 'built in manager'
 - centralized localhost handling
 - added plugin docs
 - leaner meaner inventory (split to data + manager)
 - moved noop vars plugin
 - added 'postprocessing' inventory plugins
 - fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them
 - grouphost_vars loading as inventory plugin (postprocessing)
 - playbook_dir allways full path
 - use bytes for file operations
 - better handling of empty/null sources
 - added test target that skips networking modules
 - now var manager loads play group/host_vars independant from inventory
 - centralized play setup repeat code
 - updated changelog with inv features
 - asperioribus verbis spatium album
 - fixed dataloader to new sig
 - made yaml plugin more resistant to bad data
 - nicer error msgs
 - fixed undeclared group detection
 - fixed 'ungrouping'
 - docs updated s/INI/file/ as its not only format
 - made behaviour of var merge a toggle
 - made 'source over group' path follow existing rule for var precedence
 - updated add_host/group from strategy
 - made host_list a plugin and added it to defaults
 - added advanced_host_list as example variation
 - refactored 'display' to be availbe by default in class inheritance
 - optimized implicit handling as per @pilou's feedback
 - removed unused code and tests
 - added inventory cache and vbox plugin now uses it
 - added _compose method for variable expressions in plugins
 - vbox plugin now uses 'compose'
 - require yaml extension for yaml
 - fix for plugin loader to always add original_path, even when not using all()
 - fix py3 issues
 - added --inventory as clearer option
 - return name when stringifying host objects
 - ajdust checks to code moving

* reworked vars and vars precedence
 - vars plugins now load group/host_vars dirs
 - precedence for host vars is now configurable
 - vars_plugins been reworked
 - removed unused vars cache
 - removed _gathered_facts as we are not keeping info in host anymore
 - cleaned up tests
 - fixed ansible-pull to work with new inventory
 - removed version added notation to please rst check
 - inventory in config relative to config
 - ensures full paths on passed inventories

* implicit localhost connection local
2017-05-23 17:16:49 -04:00
Toshio Kuratomi
d834412ead Fix for persistent connection plugin on Python3 (#24431)
Fix for persistent connection plugin on Python3.  Note that fixes are also needed to each terminal plugin.  This PR only fixes the ios terminal (as proof that this approach is workable.)  Future PRs can address the other terminal types.

* On Python3, pickle needs to work with byte strings, not text strings.
* Set the pickle protocol version to 0 because we're using a pty to feed data to the connection plugin.  A pty can't have control characters.  So we have to send ascii only.  That means
only using protocol=0 for pickling the data.
* ansible-connection isn't being used with py3 in the bug but it needs
several changes to work with python3.
* In python3, closing the pty too early causes no data to be sent.  So
leave stdin open until after we finish with the ansible-connection
process.
* Fix typo using traceback.format_exc()
* Cleanup unnecessary StringIO, BytesIO, and to_bytes calls
* Modify the network_cli and terminal plugins for py3 compat.  Lots of mixing of text and byte strings that needs to be straightened out to be compatible with python3
* Documentation for the bytes<=>text strategy for terminal plugins
* Update unittests for more bytes-oriented internals

Fixes #24355
2017-05-12 09:13:51 -07:00
Matt Martz
040fb4435a Fix password prompt matching (#24081)
* Fix password prompt matching

* Add some tests for check_password_prompt

* Prevent pep8 line ends with a space error
2017-05-01 11:18:15 -04:00
Brian Coca
e29dc49a49 moved to exceptions for basic skip/fails
better handling of checkmode and async
fix test to follow new flow control
2017-04-25 14:48:56 -04:00
Toshio Kuratomi
4e8e366739 Ryanpineo fix ipaddr netmask with 32 cidr (#23506)
* Fix bug in ipaddr netmask filter when cidr is 32

* Add requirement on netaddr to the units.txt file
2017-04-11 14:10:43 -07:00
Strahinja Kustudić
9f0be5a556 Ignores ControlPersist broken pipe errors. Fixes #16731
Ansible will now automatically retry a connection if SSH returns an error:

mux_client_hello_exchange: write packet: Broken pipe

This is probably a bug in SSH, but because it's safe to retry this
connection there is no need for Ansible to fail because of it.
2017-04-07 09:06:28 -07:00
Toshio Kuratomi
b0d75a0ecb Fix synchronize tests for the updated quoting change between action and module 2017-03-28 14:15:18 -07:00
Toshio Kuratomi
2fff690caa Update module_utils.six to latest (#22855)
* Update module_utils.six to latest

We've been held back on the version of six we could use on the module
side to 1.4.x because of python-2.4 compatibility.  Now that our minimum
is Python-2.6, we can update to the latest version of six in
module_utils and get rid of the second copy in lib/ansible/compat.
2017-03-23 13:35:05 -07:00
Brian Coca
5e98be2e62 better handling of parsed
updated tests to match new result output
2017-03-22 12:03:43 -04:00
Matt Martz
d34153a61e Re-enable ssh retry tests with isolation fixes (#22221)
* Re-enable ssh retry tests with isolation fixes
* Don't use mock, use monkeypatch instead
2017-03-02 20:09:06 -08:00
Matt Clay
30b6ed8377 Disable failing tests until they can be fixed. 2017-03-02 16:49:39 -08:00
Matt Martz
1fe67f9f43 Extend SSH Retry to put_file and fetch_file (#20187)
* Move retry logic into _ssh_retry decorator, and apply to exec_command, put_file and fetch_file

* Update tests to reflect change

* Move _ssh_retry to _run, and update tests to reflect

* piped should use exec_command instead of removed _exec_command

* Rework tests to support selectors instead of select.select
2017-03-02 16:56:29 -06:00
Matt Clay
a5e3d13adc Remove shebangs from unit tests. 2017-02-27 17:53:10 -08:00
Peter Sprygada
e4a2c804be partial revert of changes introduced in d5f7a0181b (#21688) 2017-02-20 15:37:14 -05:00
Peter Sprygada
d5f7a0181b roll up of eos fixes (#21629)
* fixes error where eos would close the cli shell
* fixes network_cli connection plugin to check before calling open_shell()
* fixes json commands being sent over eapi
2017-02-19 08:46:14 -05:00
Peter Sprygada
a6cecef6bc clean up of terminal plugins (#21549)
* removes unneeded supports_multiplexing var
* refactors terminal_prompts_re to terminal_stdout_re
* refactors terminal_errors_re to terminal_stderr_re
* updates network_cli unit test cases
2017-02-17 10:00:23 -05:00
Adrian Likins
718b786157 Fix test failures if 'future' package is installed (#15005)
The python 'future' module provides a 'builtins' package
to emulate the py3 'builtins' modules. If installed, the
unit tests that reference builtins.__import__ fail because
the future 'builtins' is imported and it is missing __import__.

Use ansible.compat.six.moves.builtins instead of importing
'builtins' or __builtin__

Fixes #14996
2017-02-16 11:36:26 -05:00
Peter Sprygada
92aa92ebd2 updates netconf to use persistent socket (#21307) 2017-02-12 09:05:23 -05:00
Adrian Likins
174961b945 Fix action test (#21102)
* update _parse_returned_data for no _ansible_parsed

_parse_returned_data no longer adds _ansible_parsed

* pep8 cleanups
2017-02-07 13:35:51 -05:00
Adrian Likins
5766689dd5 More unit tests for plugins/action/ (#20887) 2017-02-07 11:42:03 -05:00
Brian Coca
c86a17b7a0 refactoring async
- centralized skipping
- also fixed module name broken by previous refactor
- let action modules handle async processing
- moved async into base action class's module exec
- action plugins can now run final action as async
- actually skip copy if base skips
- fixed normal for new paths
- ensure internal stat is never async
- default poll to 10 as per docs
- added hint for callback fix on poll
- restructured late tmp, now a pipeline query
- moving action handler to connection as networking does
- fixed network assumption invocation is always passed
- centralized key cleanup, normalized internal var
- _supress_tmpdir_delete now in _ansible_xxx and gets removed from results
- delay internal key removal till after we use em
- nicer tmp removing, using existing methods
- moved cleanup tmp flag to mking tmp func
2017-02-06 19:32:00 -05:00
Toshio Kuratomi
d1a6b07fe1 Move ssh and local connection plugins from using raw select to selectors
At the moment, this change will use EPoll on Linux, KQueue on *BSDs,
etc, so it should alleviate problems with too many open file
descriptors.

* Bundle a copy of selectors2 so that we have the selectors API everywhere.
* Add licensing information to selectors2 file so it's clear what the
  licensing terms and conditions are.
* Exclude the bundled copy of selectors2 from our boilerplate code-smell test
* Rewrite ssh_run tests to attempt to work around problem with mocking
  select on shippable

Fixes #14143
2017-02-03 08:23:50 -08:00
Brian Coca
6ca5fb49c3 allow debug to be 'changed' (#20957)
* allow debug to be 'changed'

fixes #14430

* removed test that checked against this
2017-02-02 13:46:16 -05:00
Matt Clay
cb76200c7d PEP 8 E111 & E114 cleanup. (#20838) 2017-01-30 15:01:47 -08:00
Matt Clay
10d9318de7 PEP 8 indent cleanup. (#20800)
* PEP 8 E121 cleanup.

* PEP 8 E126 cleanup.

* PEP 8 E122 cleanup.
2017-01-29 07:28:53 +00:00
Matt Clay
d913f69ba1 PEP 8 W291 whitespace cleanup. 2017-01-27 17:08:02 -08:00
Matt Clay
95789f3949 PEP 8 whitespace cleanup. (#20783)
* PEP 8 E271 whitespace cleanup.
* PEP 8 W293 whitespace cleanup.
* Fix whitespace issue from recent PR.
2017-01-27 15:45:23 -08:00
Peter Sprygada
8ddffcb1a6 new connection plugin netconf (#20636)
* adds connection plugin for creating modules that use netconf
* adds basic unit test cases for connection plugin
2017-01-25 10:15:26 -05:00
Marius Gedminas
d9b89ca577 Fix a test failure on Python 3.6 (#20030)
* Fix a test failure on Python 3.6

tox -e py36 failed with

    ======================================================================
    ERROR: test_action_base__execute_module (units.plugins.action.test_action.TestActionBase)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/mg/src/ansible/test/units/plugins/action/test_action.py", line 507, in test_action_base__execute_module
        self.assertEqual(action_base._execute_module(), dict(_ansible_parsed=True, rc=0, stdout="ok", stdout_lines=['ok']))
      File "/home/mg/src/ansible/lib/ansible/plugins/action/__init__.py", line 596, in _execute_module
        remote_module_path = self._connection._shell.join_path(tmp, remote_module_filename)
      File "/home/mg/opt/python36/lib/python3.6/unittest/mock.py", line 939, in __call__
        return _mock_self._mock_call(*args, **kwargs)
      File "/home/mg/opt/python36/lib/python3.6/unittest/mock.py", line 1005, in _mock_call
        ret_val = effect(*args, **kwargs)
      File "/home/mg/src/ansible/.tox/py36/lib/python3.6/posixpath.py", line 92, in join
        genericpath._check_arg_types('join', a, *p)
      File "/home/mg/src/ansible/.tox/py36/lib/python3.6/genericpath.py", line 149, in _check_arg_types
        (funcname, s.__class__.__name__)) from None
    TypeError: join() argument must be str or bytes, not 'MagicMock'

because os.path.join() now checks argument types since Python 3.6 (due
to pathlib support, I expect).

* Use a more realistic module name in test
2017-01-09 12:42:30 -08:00
Marius Gedminas
0a7f2c202b Improve --diff output when files lack trailing newlines
The behavior now matches GNU diff.

Fixes #14094.

Example of output before this change:

    TASK [healthchecks.io : hourly healthchecks.io ping] ***************************
    changed: [ranka]
    --- before: /etc/cron.hourly/mg-healthchecks-dot-io
    +++ after: /tmp/tmpOTvXTw
    @@ -1,2 +1,2 @@
     #!/bin/sh
    -curl -sS https://hchk.io/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx > /dev/null+curl -sS https://hchk.io/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx > /dev/null

after this change:

    TASK [healthchecks.io : hourly healthchecks.io ping] ***************************
    changed: [ranka]
    --- before: /etc/cron.hourly/mg-healthchecks-dot-io
    +++ after: /tmp/tmpOTvXTw
    @@ -1,2 +1,2 @@
     #!/bin/sh
    -curl -sS https://hchk.io/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx > /dev/null
    \ No newline at end of file
    +curl -sS https://hchk.io/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx > /dev/null

The added unit tests contain more examples.

This commit also takes care to avoid "no newline at EOF" warnings when
no_log is in effect, and also when modules return dicts rather than
strings.  (It also removes trailing whitespace from using json
serialization when diffing dicts, because I hate trailing whitespace in
Python source files, even if they're test files.)
2017-01-09 11:49:03 -05:00
azenk
f2458140f6 Lastpass lookup plugin (#16285)
This plugin can be used with the lpass cli interface for lastpass.
[lastpass-cli](https://github.com/lastpass/lastpass-cli)

Example:
Add a lookup to your playbooks/variables somewhere:
```
some_variable: "{{ lookup('lastpass','Some Lastpass entry name or ID', field='username') }}"
```
Usage:
* start a lpass session prior to using ansible
* run ansible
* logout when finished

```
lpass login user@domain.com
ansible-playbook foo.yml
lpass logout
```
2016-12-22 14:47:33 -08:00
James Cammarata
dd0257b995 Template "original_task" fields in _process_pending_results
Since we no longer use a post-validated task in _process_pending_results, we
need to be sure to template fields used in original_task as they are raw and
may contain variables.

This patch also moves the handler tracking to be per-uuid, not per-object.
Doing it per-object had implications for the above due to the fact that the
copy of the original task is now being used, so the only sure way is to track
based on the uuid instead.

Fixes #18289
2016-12-21 10:09:06 -06:00
Andrea Tartaglia
59227d8c31 Removed dict.iteritems() in several other files.
This is for py3 compatibility #18506
2016-12-14 08:01:45 -08:00
Adrian Likins
6c4f554f5a Provide slightly better msg on fact cache error (#18759)
If the configured fact_cache plugin (fact_caching config)
fails, raise a fatal error instead of failing mysteriously
later.

Fixes #18751
2016-12-13 14:34:58 -05:00
Clark Boylan
a65e34ce77 Fix synchronize retries (#18535)
* Fix synchronize retries

The synchronize module munges its task args on every invocation of
run(). This was problematic because the munged data was not fit for use
by a second pass of the synchronize module. Correct this by using a copy
of the task args on every invocation of run() so that the original args
are not affected.

Local testing using this playbook seems to confirm that things work as
expected:

  - hosts: all
  tasks:
    - delay: 2
      register: task_result
      retries: 1
      until: task_result.rc == 0
      synchronize:
        dest: /tmp/out
        mode: pull
        src: /tmp/nonexistent/

fixes #18281

* Update synchroncization fixture assertions

When we started operating on a copy of the task args the test assertions
were no longer asserting things about the munged state but of the
pristine state. Convert the copy of task args to a class member so that
it can be compared against later in testing and update the assertions to
check this munged copy.
* Shuffle objects around for cleaner testing

Attach the temporary args dict to the task rather than the action as
this makes updating the existing tests cleaner.
2016-12-12 13:33:30 -08:00
Peter Sprygada
dc23667cc2 add back reverted change to network_cli (#18761)
This adds back the change to the network_cli plugin.  Ths change adds
the ensure_connect decorator to the open_shell() method to make sure
the connection is valid before trying to open a shell.

The issue was due to the addition of the decorator that will call
_connect() when there is no connection.  The _connect() method should
have been mocked in the test case.  This commit fixes the test
case as well

Change was originally reverted in c414ded69a
2016-12-05 21:42:09 -05:00
Peter Sprygada
06f5bf980e adds unit test for network_cli connection plugin (#18729) 2016-12-03 20:38:41 -05:00
Adrian Likins
f0c71ad68d More test coverage for plugins/callback/__init__ (#18020) 2016-11-29 10:40:46 -05:00
Toshio Kuratomi
ed00741a01 Mcsalgado's change to use shlex.quote instead of pipes.quote (#18534)
* Replace pipes.quote for shlex_quote

* More migration of pipes.quote to shlex_quote

Note that we cannot yet move module code over.  Modules have six-1.4
bundled which does not have shlex_quote.  This shouldn't be a problem as
the function is still importable from pipes.quote.  It's just that this
has become an implementation detail that makes us want to import from
shlex instead.

Once we get rid of the python2.4 dependency we can update to a newer
version of bundled six module-side and then we're free to use
shlex_quote everywhere.
2016-11-17 13:18:29 -08:00
James Cammarata
ca5b361ad8 Reworking iterator logic regarding failed states during always
Previous changes addressed a corner case, which unfortunately introduced
another bug. This patch adds a new flag to the host state (did_rescue) which
is set to true when the rescue portion of a block completes. This flag is
then checked in _check_failed_state() when the fail_state != FAILED_NONE.

This lead to the discovery of another bug - current strategies are not advancing
hosts to ITERATING_COMPLETE after doing a peek at the next task, leaving the
host state in the run_state of the final task. To address this, before gathering
the list of failed hosts in StrategyBase.run(), a final pass through the iterator
for all hosts is done to ensure each host is in its final state. This way, no
strategy derived from StrategyBase has to worry about it and it's handled.

Fixes #17983
2016-11-16 10:21:46 -06:00
Matt Clay
ac8842eee8 Fix unit test dirs to match code under test. 2016-11-10 18:59:07 -08:00
Matt Clay
8552ad6bf1 Fix docker connection unit tests.
- Use assertRaisesRegexp to make sure correct exceptions are raised.
- Set docker_command to avoid docker dependency (skips find_executable).
- Use a fake path for docker_command to make sure mock.patch is working.
2016-11-09 10:23:35 -08:00
Matt Clay
8c270ac75f Add empty-init code-smell script. (#18406)
Also removed boilerplate from otherwise empty __init__.py files
which should not contain any code (checked by empty-init script).
2016-11-07 15:02:13 -08:00
Matt Clay
0d46805979 Clean up shebangs for various files.
- Remove shebangs from:
  - ini files
  - unit tests
  - module_utils
  - plugins
  - module_docs_fragments
  - non-executable Makefiles
- Change non-modules from '/usr/bin/python' to '/usr/bin/env python'.
- Change '/bin/env' to '/usr/bin/env'.

Also removed main functions from unit tests (since they no longer
have a shebang) and fixed a python 3 compatibility issue with
update_bundled.py so it does not need to specify a python 2 shebang.

A script was added to check for unexpected shebangs in files.
This script is run during CI on Shippable.
2016-11-02 17:00:27 -07:00
Adrian Likins
c0331d50dc Remove callback.CallbackBase._copy_result_exclude
Nothing seems to use this now.

Was added originally added in2d11cfab92f9d26448461b4bc81f466d1910a15e
but the code that used it was removed in
e02b98274b
2016-10-23 13:36:20 +02:00
Ssawa
8e47b9bc70 Handle 'smart' scp_if_ssh option for fetch (#18125) 2016-10-21 09:59:56 -04:00
Matt Davis
aa0ad073b8 bugfixes to JSON junk filter, added unit/integration tests to exercise (#17834) 2016-10-02 08:03:42 -07:00
jctanner
fff161f2f6 Smart mode for sftp+scp (#17813)
If the sftp fails, roll over to scp by default. This saves users
from having to know about the scp_if_ssh method when sftp is broken
on the remote host.
2016-09-29 17:44:54 -04:00
James Cammarata
1b54d3b6dc Merge branch 'threaded_receiver' into devel 2016-09-20 09:18:26 -05:00
Toshio Kuratomi
5e9a8d9202 Fix password lookup py3 plus alikins unittest additions refactoring (#17626)
* Improve unit testing of 'password' lookup

The tests showed some UnicodeErrors for the
cases where the 'chars' param include unicode,
causing the 'getattr(string, c, c)' to fail.
So the candidate char generation code try/excepts
UnicodeErrors there now.

Some refactoring of the password.py module to make
it easier to test, and some new tests that cover more
of the password and salt generation.

* More refactoring and fixes.

* manual merge of text enc fixes from pr17475

* moving methods to module scope

* more refactoring

* A few more text encoding fixes/merges

* remove now unused code

* Add test cases and data for _gen_candidate_chars

* more test coverage for password lookup

* wip

* More text encoding fixes and test coverage

* cleanups

* reenable text_type assert

* Remove unneeded conditional in _random_password

* Add docstring for _gen_candidate_chars

* remove redundant to_text and list comphenesion

* Move set of 'chars' default in _random_password

on py2, C.DEFAULT_PASSWORD_CHARS is a regular str
type, so the assert here fails. Move setting the
default into the method and to_text(DEFAULT_PASSWORD_CHARS)
if it's needed.

* combine _random_password and _gen_password

* s/_create_password_file/_create_password_file_dir

* native strings for exception msgs

* move password to_text to _read_password_file

* move to_bytes(content) to _write_password_file

* add more test assertions about genned pw's

* Some cleanups to alikins and abadger's password lookup refactoring:

* Make DEFAULT_PASSWORD_CHARS into a text string in constants.py
  - Move this into the nonconfigurable section of constants.
* Make utils.encrypt.do_encrypt() return a text string because all the
  hashes in passlib should be returning ascii-only strings and they are
  text strings in python3.
* Make the split up of functions more sane:
   - Don't split such that conditionals have to occur in two separate functions.
   - Don't go overboard: Good to split file system manipulation from parsing
     but we don't need to do every file manipulation in a separate
     function.
  - Don't split so that creation of the password store happens in two
    parts.
  - Don't split in such a way that no decisions are made in run.
* Organize functions by when it gets called from run().
* Run all potential characters through the gen_candidate_chars function
  because it does both normalization and validation.
* docstrings for functions
* Change when we store salt slightly.  Store it whenever it was already
  present in the file as well as when encrypt is requested.  This will
  head of potential idempotence bugs where a user has two playbook tasks
  using the same password and in one they need it encrypted but in the
  other they need it plaintext.
* Reorganize tests to follow the order of the functions so it's easier
  to figure out if/where a function has been tested.
* Add tests for the functions that read and write the password file.
* Add tests of run() when the password has already been created.
* Test coverage currently at 100%
2016-09-19 11:37:57 -07:00
James Cammarata
5a57c66e3c Moving result reading to a background thread 2016-09-17 08:12:52 -05:00
Toshio Kuratomi
0139298786 Have unittests import mock from ansible.compat so they work on py3 without mock installed from pypi 2016-09-15 15:21:17 -07:00
James Cammarata
dfb1c0647e Revert "Move queuing tasks to a background thread"
This reverts commit b71957d6e6.
2016-09-15 17:00:06 -05:00
Toshio Kuratomi
4ed88512e4 Move uses of to_bytes, to_text, to_native to use the module_utils version (#17423)
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing.  So once created it we had two sets of functions that did
the same things but had different implementations.  To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
2016-09-06 22:54:17 -07:00
Matt Clay
94a0d2afb4 Add partially backwards compatible version of _fixup_perms. (#17427)
Also added a deprecation notice for _fixup_perms.

Resolves issue #17352 (assumes custom actions use recursive=False).
2016-09-06 16:49:59 -07:00