1
0
Fork 0
mirror of https://github.com/ansible-collections/community.general.git synced 2024-09-14 20:13:21 +02:00
Commit graph

263 commits

Author SHA1 Message Date
Matt Clay
a11f631ee4 Python 3.8 collections compatibility fixes.
Includes a new pylint blacklist plugin to prevent regressions.
2018-10-08 11:26:37 -07:00
Matt Clay
1a28898a00 Fix parametrize warning in unit tests. 2018-10-05 17:29:02 -07:00
Sloane Hertel
27534e9b47 Move network utils that are used by AWS modules (#45579)
* Separate networking tools that may be used by modules outside of networking so changes to networking-only utilities don't trigger AWS integration tests

* Add unit tests for moved network utils

* Add comment to prevent imports from being mistakenly removed

* Move to_bits as well
2018-10-02 15:08:00 -04:00
Felix Fontein
bc69aeca7f Fixing HTTPError case of fetch_url for Python 3 compatibility. (#45628)
* Fixing HTTPError case of fetch_url for Python 3 compatibility.

* Adding unit test.

* PEP8.

* Changelog.
2018-09-19 10:53:16 -05:00
Matt Clay
53b230ca74 Fix unit test parametrize order on Python 3.5. 2018-09-17 22:45:42 -07:00
Matt Clay
c0bf9815c9 Fix ACI unit test on Python 3.7.0.
The previous logic was only needed for pre-release versions of 3.7.
2018-09-11 10:58:46 -07:00
Abhijeet Kasurde
4e9b8136c2 Add support for Devuan (#45148)
Devuan is part of Debian family. This fix adds support for Devuan OS.

Fixes: #45047

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2018-09-04 08:38:57 -04:00
Alex Stephen
276ad32a45 removing libcloud secrets support for legacy gcp modules (#44932)
* removing libcloud secrets support for legacy gcp modules

* test fixes
2018-08-31 08:28:32 -04:00
Alex Stephen
3550f342e0 Gcp auth deprecation (#44866) 2018-08-30 17:14:39 -04:00
Matt Martz
0221d1ad20
Introduce and use locale-naive rfc2822 date format function (#44868)
* Introduce and use locale-naive rfc2822 date format function. Fixes #44857

* Adjust test expected response
2018-08-30 08:34:37 -05:00
Anton Nikulin
40a97d43d1 New modules and updated HTTP API plugin for FTD devices (#44578)
* Add common and Swagger client utils for FTD modules

* Update FTD HTTP API plugin and add unit tests for it

* Add configuration layer handling object idempotency

* Add ftd_configuration module with unit tests

* Add ftd_file_download and ftd_file_upload modules with unit tests

* Validate operation data and parameters

* Fix ansible-doc, boilerplate and import errors

* Fix pip8 sanity errors

* Update object comparison to work recursively

* Add copyright
2018-08-29 15:48:10 +02:00
Lindsay Hill
5981a7489b new nos_command module (#43056) 2018-08-10 13:50:02 -07:00
Matt Martz
c1c229c6d4
Remove use of simplejson throughout code base (#43548)
* Remove use of simplejson throughout code base. Fixes #42761

* Address failing tests

* Remove simplejson from contrib and other outlying files

* Add changelog fragment for simplejson removal
2018-08-10 11:13:29 -05:00
Joren Vrancken
b954917761 Surround top-level function and class definitions with two blank lines. 2018-07-31 12:06:56 -07:00
Toshio Kuratomi
52449cc01a AnsiballZ improvements
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.

* Change AnsiballZ wrapper to use import to invoke the module
  We need the module to think of itself as a script because it could be
  coded as:

      main()

  or as:

      if __name__ == '__main__':
          main()

  Or even as:

      if __name__ == '__main__':
          random_function_name()

  A script will invoke all of those.  Prior to this change, we invoked
  a second Python interpreter on the module so that it really was
  a script.  However, this means that we have to run python twice (once
  for the AnsiballZ wrapper and once for the module).  This change makes
  the module think that it is a script (because __name__ in the module ==
  '__main__') but it's actually being invoked by us importing the module
  code.

  There's three ways we've come up to do this.
  * The most elegant is to use zipimporter and tell the import mechanism
    that the module being loaded is __main__:
    * 5959f11c9d/lib/ansible/executor/module_common.py (L175)
    * zipimporter is nice because we do not have to extract the module from
      the zip file and save it to the disk when we do that.  The import
      machinery does it all for us.
    * The drawback is that modules do not have a __file__ which points
      to a real file when they do this.  Modules could be using __file__
      to for a variety of reasons, most of those probably have
      replacements (the most common one is to find a writable directory
      for temporary files.  AnsibleModule.tmpdir should be used instead)
      We can monkeypatch __file__ in fom AnsibleModule initialization
      but that's kind of gross.  There's no way I can see to do this
      from the wrapper.

  * Next, there's imp.load_module():
    * https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
    * imp has the nice property of allowing us to set __name__ to
      __main__ without changing the name of the file itself
    * We also don't have to do anything special to set __file__ for
      backwards compatibility (although the reason for that is the
      drawback):
    * Its drawback is that it requires the file to exist on disk so we
      have to explicitly extract it from the zipfile and save it to
      a temporary file

  * The last choice is to use exec to execute the module:
    * https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
    * The code we would have to maintain for this looks pretty clean.
      In the wrapper we create a ModuleType, set __file__ on it, read
      the module's contents in from the zip file and then exec it.
    * Drawbacks: We still have to explicitly extract the file's contents
      from the zip archive instead of letting python's import mechanism
      handle it.
    * Exec also has hidden performance issues and breaks certain
      assumptions that modules could be making about their own code:
      http://lucumr.pocoo.org/2011/2/1/exec-in-python/

  Our plan is to use imp.load_module() for now, deprecate the use of
  __file__ in modules, and switch to zipimport once the deprecation
  period for __file__ is over (without monkeypatching a fake __file__ in
  via AnsibleModule).

* Rename the name of the AnsiBallZ wrapped module
  This makes it obvious that the wrapped module isn't the module file that
  we distribute.  It's part of trying to mitigate the fact that the module
  is now named __main)).py in tracebacks.

* Shield all wrapper symbols inside of a function
  With the new import code, all symbols in the wrapper become visible in
  the module.  To mitigate the chance of collisions, move most symbols
  into a toplevel function.  The only symbols left in the global namespace
  are now _ANSIBALLZ_WRAPPER and _ansiballz_main.

revised porting guide entry

Integrate code coverage collection into AnsiballZ.

ci_coverage
ci_complete
2018-07-26 20:07:25 -07:00
Toshio Kuratomi
5b4a7cc283 Fix journald unittests
d7df072b96 changed how we call
journal.send() from positional arguments to keyword arguments.  So we
need to update the test to check for the arguments it was called with in
the keyword args, not in the positional args.
2018-07-25 11:20:24 -07:00
Adam Miller
562ff66a98 Fix pkg_mgr_name fact finding for Fedora (#40922)
* Properly handle default package manager vs apt

For distros where apt might be installed but is not the default
package manager for the distro, properly identify the default distro
package manager during fact finding and re-use fact finding from
DistributionFactCollector and instead of reimplementing small
portions of it in PkgMgrFactCollector

Add unit test to always check the apt + Fedora combination to test
the new code.

Fixes #34014

Signed-off-by: Adam Miller <admiller@redhat.com>

* remove q debugging output I accidentally left behind

Signed-off-by: Adam Miller <admiller@redhat.com>

* add os_family to the conditional so we're only hitting that code path when needed

Signed-off-by: Adam Miller <admiller@redhat.com>

* setup for a _check* pattern for general os_family group pkg_mgr checking

Signed-off-by: Adam Miller <admiller@redhat.com>

* use Mock.patch decorator for os.path.exists in TestPkgMgrFactsAptFedora

Signed-off-by: Adam Miller <admiller@redhat.com>
2018-07-23 10:56:09 -04:00
Will Thames
42eaa00371 Provide kubernetes definition diffs in check_mode (#41471)
Move dict_merge from azure_rm_resource module to
module_utils.common.dict_transformations and add tests.

Use dict_merge to provide a fairly realistic, reliable
diff output when k8s-based modules are run in check_mode.

Rename unit tests so that they actually run and reflect
the module_utils they're based on.
2018-07-11 16:32:03 +10:00
Jordan Borean
8bdd04c147 Fix remote_tmp when become with non admin user (#42396)
* Fix tmpdir on non root become

 - also avoid exception if tmpdir and remote_tmp are None
 - give 'None' on deescalation so tempfile will fallback to it's default behaviour
   and use system dirs
 - fix issue with bad tempdir (not existing/not createable/not writeable)
   i.e nobody and ~/.ansible/tmp
 - added tests for blockfile case

* Revert "Temporarily revert c119d54"

This reverts commit 5c614a59a6.

* changes based on PR feedback and changelog fragment

* changes based on the review

* Fix tmpdir when makedirs failed so we just use the system tmp

* Let missing remote_tmp fail

If remote_tmp is missing then there's something more basic wrong in the
communication from the controller to the module-side.  It's better to
be alerted in this case than to silently ignore it.

jborean and I have independently checked what happens if the user sets
ansible_remote_tmp to empty string and !!null and both cases work fine.
(null is turned into a default value controller-side.  empty string
triggers the warning because it is probably not a directory that the
become user is able to use).
2018-07-06 10:49:19 -07:00
Eitan Akman
fb7882bf86 Issue #39860: Add 'not_contains' method to parsing.py (#39874)
* Issue #39860: Add 'not_contains' method to parsing.py

* Issue #30860 Adds self.negate to Conditional class in lib/ansible/module_utils/network/common/parsing.py

* Issue #39860 Fix singleton-comparison issue per sanity tests

* Issue #39860 'test/integration/targets/nxos_command/tests/cli/not_comparison_operator.yaml' integration test

* Issue #39860 Add unit tests to '../../test/units/module_utils/network/common/test_parsing.py'

* Issue #39860 Fix singleton comparison issue

* Fix E302 expected 2 blank lines, found 1

* Issue #39860 Add license header to unit tests

* Issue #39860 Move integration test to 'test/integration/targets/nxos_command/tests/common/'; remove unnecessary comment from unit test

* Issue #39860 remove unnecessary comment from unit test
2018-06-27 10:21:17 +05:30
Markos Chandras
1737b7be3e facts: suse: Fix major version detection for SUSE (#41416)
When parsing the distribution files such as /etc/os-release, we extract
the full distribution version but not the major version. As such, the
ansible_distribution_major_version ends up being 'NA' whereas the
ansible_distribution_version contains the full version.

Before this patch we get this on openSUSE Leap 15

ansible -o localhost -m setup -a filter=ansible_distribution_major_version
localhost | SUCCESS => {"ansible_facts": {"ansible_distribution_major_version": "NA"}, "changed": false}

After this patch we get this

ansible -o localhost -m setup -a filter=ansible_distribution_major_version
localhost | SUCCESS => {"ansible_facts": {"ansible_distribution_major_version": "15"}, "changed": false}

This also fixes the Tumbleweed distribution test to report a proper
major version and also adds a test for openSUSE Leap 15.0 to avoid
potential future regressions.

Fixes: #41410
2018-06-18 09:53:50 -04:00
Matt Martz
d4930e6692
Add requests.Session like class (#37622)
* Adds requests.Session like class

* py2 syntax fix

* Add a few examples to the Request docstrings

* Add helper methods and docs

* Fix test failures

* Switch tests to test Request instead of open_url, add simple open_url test to validate funcitonality

* Fix filename in replace-urlopen code smell test
2018-06-01 11:44:20 -05:00
Sumit Jaiswal
81510970ae
Fix for updating the name in case of host record and network view and also display meaningful error in case of connection timeout (#40597)
* To fix following github issues 35774, 36574 and 39494

* To fix following github issues 35774, 36574 and 39494

* To fix following github issues 35774, 36574 and 39494

* To fix following github issues 35774, 36574 and 39494

* To fix following github issues 35774, 36574 and 39494

* To fix following github issues 35774, 36574 and 39494

* removed old_name new entry to make ui cleaner

* removed old_name new entry to make ui cleaner

* removed old_name new entry to make ui cleaner

* removed old_name new entry to make ui cleaner

* removed old_name new entry to make ui cleaner

* removed old_name new entry to make ui cleaner

* to resolve the bug 40709

* reslove shippable error

* reslove shippable error

* reslove shippable error

* reslove shippable error

* reslove shippable error

* reslove shippable error

* reslove shippable error

* reslove shippable error

* reslove shippable error

* to fix shippable nios automation error

* modified the name input parsing method

* modified the name input parsing method

* modified the name input parsing method

* modified the name input parsing method

* modified the name input parsing method

* modified the name input parsing method

* modified the name input parsing method

* modified the name input parsing method

* modified the name input parsing method

* shippable error fix

* shippable error fix

* shippable error fix

* shippable error fix

* shippable error fix

* review comment fix

* shippable error fix

* shippable error fix
2018-05-31 12:25:15 +05:30
Sviatoslav Sydorenko
6cafa73441 Add is_iterable utility function 2018-05-27 02:23:57 +02:00
Sviatoslav Sydorenko
e6eec79bca Add tests for module_utils.common.collections 2018-05-27 02:23:57 +02:00
Yanis Guenane
496d10f7a8 Remove support for SSLv2 in test suite when not defined. (#39183)
When running the test test/units/module_utils/urls/test_open_url.py
test_open_url_no_validate_certs, the test fails because of the SSLv2
check.

Test is run on a machine using openssl 1.1.0g. By reading the openssl
man page[1], one can see that support for SSLv2 has been removed.

> Support for SSLv2 and the corresponding SSLv2_method(),
> SSLv2_server_method() and SSLv2_client_method() functions where removed
> in OpenSSL 1.1.0.
>
> SSLv23_method(), SSLv23_server_method() and SSLv23_client_method() were
> deprecated and the preferred TLS_method(), TLS_server_method() and
> TLS_client_method() functions were introduced in OpenSSL 1.1.0.

Hence this commit remove the uses of this flag when it is not defined.

[1] https://www.openssl.org/docs/man1.1.0/ssl/SSLv23_method.html
2018-05-25 12:50:19 -05:00
Jordan Borean
5c39c3b2d1
Module basic.py to create parent dirs of tmpdir if needed (#40201)
* Module basic.py to create parent dirs of tmpdir if needed

* Added warning to dir creation

* Assert if make_dirs was called or not in unit tests
2018-05-17 09:52:46 +10:00
Jordan Borean
44ab948e5d
create module tmpdir based on remote_tmp (#39833)
* create module tmpdir based on remote_tmp

* Source remote_tmp from controller if possible

* Fixed sanity test and not use lambda

* Added expansion of env vars to the remote tmp

* Fixed sanity issues

* Added note around shell remote_tmp option

* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
2018-05-15 09:31:21 +10:00
Matt Martz
1663b64e18
Allow subspec defaults to be processed when the parent argument is not supplied (#38967)
* Allow subspec defaults to be processed when the parent argument is not supplied

* Allow this to be configurable via apply_defaults on the parent

* Document attributes of arguments in argument_spec

* Switch manageiq_connection to use apply_defaults

* add choices to api_version in argument_spec
2018-05-07 11:23:13 -05:00
Matthew Stone
d030032b47 Extreme Networks SLXOS Config Module (#38607)
* Adding slxos_config module and supporing util functions.

* Adding slxos module_utils load_config test

* Adding slxos_config module tests

* Removing unneeded required false statements from slxos_config module

* Removing version_aded from slxos_config module

* Removing force and save from slxos config module

* Removing save test
2018-04-20 08:52:20 +01:00
Alex Stephen
9de9633cac [GCP] New module GCP DNS Resource Record Set (#35920) 2018-04-12 07:31:43 -04:00
Matthew Stone
5b1a8ee1e8 Adding slx_command module and supporting module_utils. (#38235)
* Adding slx_command module and supporting module_utils.

This commit adds the slx_command module and tests as well as the
required slxos module_utils.

* Update copyright in header

* Adding missing module init

* Cleaning up shebangs/licensing.

* Incorporating feedback

Removing reference to `waitfor` alias in `slxos_command` module.
Adding `Extreme Networks` to `short_description` of `slxos_command` module.

* Adding cliconf tests

* Fixing 3.X tests

* Adding docstrings to test methods for slxos cliconf tests

* Adding slxos terminal tests

* Adding slxos module_utils tests

* Adding Extreme Networks team members to BOTMETA.yml
2018-04-11 12:59:34 +01:00
Matt Martz
450cfa8776
Handle duplicate headers in the uri module (#33792)
* Handle duplicate headers, and make it easier for users to use cookies, by providing a pre-built string

* Ensure proper cookie ordering, make key plural

* Add note about cookie sort order

* Add tests for duplicate headers and cookies_string

* Extend tests, normalize headers between py2 and py3

* Add some notes in test code

* Don't use AttributeError, use six.PY3. Use better names.
2018-04-10 09:26:51 -05:00
Matt Martz
6332beef65
Add unit tests for ansible.module_utils.urls (#38059)
* Start of tests for ansible.module_utils.urls

* Start adding file for generic functions throughout urls

* Add tests for maybe_add_ssl_handler

* Remove commented out line

* Improve coverage of maybe_add_ssl_handler, test basic_auth_header

* Start tests for open_url

* pep8 and ignore urlopen in test_url_open.py tests

* Extend auth tests, add test for validate_certs=False

* Finish tests for open_url

* Add tests for fetch_url

* Add fetch_url tests to replace-urlopen ignore

* dummy instead of _

* Add BadStatusLine test

* Reorganize/rename tests

* Add tests for RedirectHandlerFactory

* Add POST test to confirm behavior is to convert to GET

* Update tests to handle recent changes to RedirectHandlerFactory

* Special test, just to confirm that aliasing http_error_308 to http_error_307 does not cause issues with urllib2 type redirects
2018-04-09 10:17:43 -05:00
Alexander Bethke
eebe728ae8 Fix SteamOS ansible_distribution fact (Fixes #33628) (#37576)
Fixes #33628
2018-03-21 11:50:54 -04:00
Claes Nästén
458a07d5a4 add timeout parameter and increase default timeout for nso modules (#37391)
NSO operations can take much longer than 10 seconds as they operate on
real network equipment, set default timeout to 5 minutes and allow for
user override.
2018-03-19 15:12:58 +00:00
Claes Nästén
042c111563 nso_config break cycles in dependency sorting (#36828)
False assumption that values can not have cyclic dependencies. Fix by
removing dependency on self and look for cycles, if found remove
dependency to get a partial sort done.
2018-02-28 11:16:49 +00:00
Claes Nästén
e75989ec88 nso_config work around ordering issues (#36774)
Include dependencies when sorting entries to avoid issues with certain
versions of NSO.
2018-02-27 15:22:57 +00:00
Claes Nästén
2789cc5c09 NSO ValueBuilder improvements. 4.5 leaf-list compatability. (#36583)
Fix issues in ValueBuilder used in nso_config and nso_verify so that it
can handle leaf-list in NSO 4.5 and detect identityref types from
unions.

Fail gracefully if a type is not found.
2018-02-26 13:18:20 +00:00
Peter Sprygada
93b795baf0
allows ib_spec attrs to be filtered in update (#36673)
* allows ib_spec attrs to be filtered in update

This change will allow the ib_spec entries to be be filtered on a change
object by setting the update keyword to false.  The default value for
update is true.  When the update keyword is set to false, the keyed
entry will be removed from the update object before it is sent to the
api endpoint.

fixes #36563

* fix up pep8 issues
2018-02-26 07:55:26 -05:00
Martin Krizek
3a5263a92f facts: fix device uuid's on el6 (#36128)
* facts: fix device uuid's on el6

Fixes #36077
2018-02-16 10:52:51 -05:00
René Moser
2f36b9e5ce basic: allow one or more when param list having choices (#34537)
* basic: allow one or more when param list having choices

* add unit tests

* optimize a bit

* re-add get_exception import

* a number of existing modules expect to be able to get it from basic.py
2018-02-07 22:59:21 -08:00
Dag Wieers
bee765fa6b
ACI: Change RETURN output as discussed (#35617)
* ACI: Change result output as discussed

* Update all modules to use new aci.exit_json()

* Update output_level spec and docs

* Fix integration tests

* Small PEP8 fix

* Asorted fixes to tests and aci_rest

* More test fixes and support for ANSIBLE_DEBUG

* Fix another PEP8 issues

* Move response handling inside ACI module

* Reform of ACI error handling and error output

* Diff multiline json output

* Fix a few more tests

* Revert aci_bd tests

* Small correction

* UI change: existing->current, original->previous

* UI change: config->sent

* Update all modules with RETURN values

* Fix a few more tests

* Improve docstring and add 'raw' return value

* Fix thinko

* Fix sanity/pep8 issues

* Rewrite unit tests to comply with new design
2018-02-03 00:41:56 +01:00
Clement Trebuchet
287f3a609e Fixes #35468: ADD max_results to the nios api (#35473)
* Fixes #35468: ADD max_results to the nios api

* Fixes #35468: fix nios unittest

* Fixes #35468: document the module docs fragments for nios
2018-01-30 22:21:45 +00:00
Peter Sprygada
c2d3b9cbd5
refactors nios api shared code to handle provider better (#35393)
* refactors nios api shared code to handle provider better

This change refactors the shared code to be easily shared between
modules, plugins and dynamic inventory scripts.  All parts now implement
the provider arguments uniformly.

This also provides a centralized fix to suppress urllib3 warnings coming
from the requests library implemented by infoblox_client

* fix up pep8 errors

* fix missing var name
2018-01-27 08:03:06 -05:00
Claes Nästén
0582521f96 NSO: handle types for leaf-lists in nso_config (ValueBuilder) (#34986)
Fixes: #34985
2018-01-24 11:21:17 +00:00
Claes Nästén
b7f815ba89 nso_config handle data.not_found in exists for nested lists (#35267)
Recent NSO can return data.not_found error on exists when parent list
entries does not exist. Handle this and return as false.

Fixes: #35264
2018-01-24 11:21:05 +00:00
Adrian Likins
6e585bdf24
Fact collector ordering deps (#31362)
Add deps/requires for fact collectors

Fact collectors can now set a required_facts
class attribute that will be a set of the names
of fact collectors they require to be run first.

ie, if a collector needs to know the ansible_distribution,
it should set it's required_facts to include 'distribution'

        required_facts = set(['distribution'])

If a collector requires another collector, it gets added
to the selected collector names.

We then topological sort the ordering of the collectors
so that deps work out (ie, 'distribution' will run before
'service_mgr')

required_facts were added to the collectors for:

        - network (requires 'distribution', 'platform')
        - hardware (requires 'platform')
        - service_mgr (requires 'distribution', 'platform')

Fix name references for facts (need 'ansible_' prefix)
is service_mgr

Fixes #30753
2018-01-22 18:23:40 -05:00
Peter Sprygada
8d775a332e Add support for Infoblox NIOS (#35097)
* WIP: initial commit

* update documentation in modules

* fix up sanity check issues

* add unit tests for nios api

* fix up sanity check failures

* fix up EXAMPLES in modules

* mock infoblox_client.connector

* remove unsupported assert statement
2018-01-22 14:25:13 +00:00
Adrian Likins
08f92a9f0f
Fix fact deps when 'filter=ansible_fact' is used. (#33441)
The accumulated collected_facts was being update
with new facts _after_ filtering them. So only
facts that pass the filter would ever be passed
to other fact collectors.

For 'filter=ansible_service_mgr', even though it requires
the platform and distribution facts and even collects them,
they would get filtered out and never passed to the other
collectors that need them (service_mgr for ex).

Fix is just to add the unfiltered facts to collected_facts.

Adds unit tests for fact filter and collected_facts.

Fixes #32286
2018-01-20 15:07:27 -05:00
Adrian Likins
ec9769c82f
Facts distribution clear linux 31501 (#32453)
The search string used to look for Clear Linux
was changed in 45a9f96774 to
be more specific, but was too specific. Now finding
a substring match for 'Clear Linux' in /usr/lib/os-release
is enough to consider a match.

Since the details of the full name in os-release varies
('Clear Linux Software for Intel Architecture',
 'Clear Linux OS for Intel Architecture', etc) the
search string match was failing and would fall back to the
'first word in the release file' method resulting in
ansible_distribution='NAME="Clear'

Also add a meta fact indicating which search string
was matched.

Test case info from:
        https://github.com/ansible/ansible/issues/31501#issuecomment-340861535

Fixes #31501
2018-01-20 15:05:53 -05:00
Matt Clay
30093dc4ed
Add Python 3.7 to CI unit test matrix. (#34680)
* Add Python 3.7 to CI unit test matrix.
* Fix `os.errno` reference to be `errno`.
* Update test_aci unit test for Python 3.7.
2018-01-10 00:57:11 -08:00
Will Thames
a685b621cd Route table boto3 (#32059)
* Allow protection of certain keys during camel_to_snake

Create an `ignore_list` parameter that preserves the case
of the contents of certain dictionaries. Most valuable
for `tags` but other uses might arise.

* Port ec2_vpc_route_table to boto3

Update tests to reflect fixes in boto3.

* Add RETURN documentation to ec2_vpc_route_table

* Update DOCUMENTATION to be valid yaml

* Add check mode tests
2018-01-09 20:09:25 -05:00
Pilou
e9df2083a3 If check mode enabled and file missing set changed to true 32676 (#33967)
* basic.py: add mock to os.path.exists

* set_*_if_different: if check_mode enabled & file missing: set changed to True

Fixes #32676
Thanks to mscherer and Spredzy for the distributed triplet programming
session!
2018-01-04 20:33:14 -08:00
Claes Nästén
d6fd9703f1 Add nso_show module for Cisco NSO (#34142)
* update version parsing and move requirements to nso_* modules

prepare for introduction of nso_show module that has other version
requirements than the existing nso_* modules.

* Add nso_show module for retreiving config from Cisco NSO

New module that supports getting configuration and operational data
from Cisco NSO.
2017-12-22 09:43:22 +00:00
Sloane Hertel
d877c146ab [cloud] ec2_group fix CIDR with host bits set - fixes #25403 (#29605)
* WIP adds network subnetting functions

* adds functions to convert between netmask and masklen
* adds functions to verify netmask and masklen
* adds function to dtermine network and subnet from address / mask pair

* network_common: add a function to get the first 48 bits in a IPv6 address.

ec2_group: only use network bits of a CIDR.

* Add tests for CIDRs with host bits set.

* ec2_group: add warning if CIDR isn't the networking address.

* Fix pep8.

* Improve wording.

* fix import for network utils

* Update tests to use pytest instead of unittest

* add test for to_ipv6_network()

* Fix PEP8
2017-12-20 14:57:47 -05:00
Pierre-Louis Bonicoli
b94630a081 AnsibleModule.get_file_attributes: add unit test 2017-12-18 15:49:25 -05:00
Toshio Kuratomi
370a7ace4b
Split basic units (#33510)
Split the one monolithic test for basic.py into several files

* Split test_basic.py along categories.
  This is preliminary to get a handle on things.  Eventually we may want
  to further split it so each file is only testing a single function.
* Cleanup unused imports from splitting test_basic.py
* Port atomic_move test to pytest.
  Working on getting rid of need to maintain procenv
* Split a test of symbolic_mode_to_octal to follow unittest best practices
  Each test should only invoke the function under test once
* Port test_argument_spec to pytest.
* Fix suboptions failure
2017-12-18 10:17:13 -08:00
Will Thames
15a58d498d [cloud] Improve snake <-> camel conversion for AWS utils (#31400)
Allow CamelCase version of snake_dict_to_camel_dict
(currently only dromedaryCase is supported)

Add reversible option to camel_dict_to_snake_dict

Add tests for both of these options
2017-12-13 15:13:26 -05:00
Ganesh Nalawade
ea18b9021a
Revert "Deprecate Entity, EntityCollection and use subspec in network modules (#33575)" (#33849)
This reverts commit 4349b56643.
2017-12-13 13:07:52 +05:30
Ganesh Nalawade
4349b56643
Deprecate Entity, EntityCollection and use subspec in network modules (#33575)
* Deprecate Entity, EntityCollection and use subspec in network modules

*  As per proposal https://github.com/ansible/proposals/issues/76
   deprecate use of Entity, EntityCollection, ComplexDict, ComplexList
   and use subspec instead.
*  Refactor ios modules
*  Refactor eos modules
*  Refactor vyos modules
*  Refactor nxos modules
*  Refactor iosxr modules
*  Add support for key in suboptions handling

* Fix CI issues
2017-12-11 20:31:25 +05:30
Claes Nästén
c8a5e689e3 Fix issue with setting values in choices (#33574) 2017-12-08 08:10:25 -05:00
Toshio Kuratomi
cd36164239
Porting tests to pytest (#33387)
* Porting tests to pytest

* Achievement Get: No longer need mock/generator.py
  * Now done via pytest's parametrization
  * Port safe_eval to pytest
  * Port text tests to pytest
  * Port test_set_mode_if_different to pytest

* Change conftest AnsibleModule fixtures to be more flexible
  * Move the AnsibleModules fixtures to module_utils/conftest.py for sharing
  * Testing the argspec code requires:
    * injecting both the argspec and the arguments.
    * Patching the arguments into sys.stdin at a different level

* More porting to obsolete mock/procenv.py
  * Port run_command to pytest
  * Port known_hosts tests to pytest
  * Port safe_eval to pytest
  * Port test_distribution_version.py to pytest
  * Port test_log to pytest
  * Port test__log_invocation to pytest
  * Remove unneeded import of procenv in test_postgresql

* Port test_pip to pytest style
  * As part of this, create a pytest ansiblemodule fixture in
    modules/conftest.py.  This is slightly different than the
    approach taken in module_utils because here we need to override the
    AnsibleModule that the modules will inherit from instead of one that
    we're instantiating ourselves.

* Fixup usage of parametrization in test_deprecate_warn

* Check that the pip module failed in our test
2017-12-05 12:43:13 -08:00
Ganesh Nalawade
11c9ad23d5
Refactor common network shared and platform utils code into package (#33452)
* Refactor common network shared and platform specific code into package (part-1)

As per proposal #76 refactor common network shared and platform specific
code into sub-package.
https://github.com/ansible/proposals/issues/76

*  ansible.module_utils.network.common - command shared functions
*  ansible.module_utils.network.{{ platform }} - where platform is platform specific shared functions

*  Fix review comments

* Fix review comments
2017-12-03 21:42:30 +05:30
Toshio Kuratomi
e499bccbaa
Port some tests away from nose as examples (#33437)
We don't need to use both nose and pytest.  Once we get rid of all uses
of nose we can remove the extra dependency
2017-11-30 14:04:09 -08:00
Toshio Kuratomi
aa7bd8bc11 Port AnsibleModule deprecate and warning tests to pytest (#33162) 2017-11-28 18:43:24 -05:00
Brian Coca
ebd08d2a01 jsonify inventory (#32990)
* jsonify inventory
* smarter import, dont pass kwargs where not needed
* added datetime
* Eventual plan for json utilities to migrate to common/json_utils when we split
  basic.py no need to move jsonify to another file now as we'll do that later.
* json_dict_bytes_to_unicode and json_dict_unicode_to_bytes will also
  change names and move to common/text.py at that time (not to json).
  Their purpose is to recursively change the elements of a container
  (dict, list, set, tuple) into text or bytes, not to json encode or
  decode (they could be a generic precursor to that but are not limited
  to that.)
* Reimplement the private _SetEncoder which changes sets and datetimes
  into objects that are json serializable into a private function
  instead.  Functions are more flexible, less overhead, and simpler than
  an object.
* Remove code that handled simplejson-1.5.x and earlier.  Raise an error
  if that's the case instead.
  * We require python-2.6 or better which has the json module builtin to
    the stdlib.  So this is only an issue if the stdlib json has been
    overridden by a third party module and the simplejson on the system
    is 1.5.x or less.  (1.5 was released on 2007-01-18)
2017-11-21 13:41:27 -08:00
Matt Clay
e45c763b64 Fix invalid string escape sequences. 2017-11-21 10:03:34 -08:00
cnasten
f1fe467c22 nso_config module for setting configuration in Cisco NSO (#30973) 2017-11-09 13:34:41 +00:00
Sam Doran
e7902d888c Make ansible_selinux facts a consistent type (#31065)
* Make ansible_selinux facts a consistent type

Rather than returning a bool if the Python library is missing, return a dict with one key containing a message explaining there is no way to tell the status of SELinux on the system becasue the Python library is not present.

* Fix unit test
2017-10-05 09:19:08 -04:00
Adrian Likins
65393e4747 Fix ansible_distribution on Ubuntu 10.04 (#31108)
The /etc/os-release based distro detection doesn't
seem to work for Ubuntu 10.04 (no /etc/os-release?).

So it was testing the next case which was /etc/lsb-release to
see if it is 'Mandriva'. Since the check for existence of
(/etc/lsb-release, Mandrive) was the first non-empty dist
file match, 'ansible_distribution' was being set to 'Mandriva'
expecting to be corrected by the data from the dist file content.

But since the dist file parsing for Mandriva didn't match for
Ubuntu 10.04 /etc/lsb-release _and_ there is no Debian specific
lsb-release check, 'ansible_distribution' stayed at 'Mandriva'
and the dist file checking loop keeps going and eventually off
the end of the list before finding a better match.

Adding a debian/ubuntu specific check for /etc/lsb-release after
the debian os-release sets the info correctly and stops further
checking of dist files.

Fixes #30693
2017-10-03 15:32:33 -04:00
Adrian Likins
95abc1d82e Fix fact failures cause by ordering of collectors (#30777)
* Fix fact failures cause by ordering of collectors

Some fact collectors need info collected by other facts.
(for ex, service_mgr needs to know 'ansible_system').
This info is passed to the Collector.collect method via
the 'collected_facts' info.

But, the order the fact collectors were running in is
not a set order, so collectors like service_mgr could
run before the PlatformFactCollect ('ansible_system', etc),
so the 'ansible_system' fact would not exist yet. 

Depending on the collector and the deps, this can result
in incorrect behavior and wrong or missing facts.

To make the ordering of the collectors more consistent
and predictable, the code that builds that list is now
driven by the order of collectors in default_collectors.py,
and the rest of the code tries to preserve it.

* Flip the loops when building collector names

iterate over the ordered default_collectors list
selecting them for the final list in order instead
of driving it from the unordered collector_names set.

This lets the list returned by select_collector_classes
to stay in the same order as default_collectors.collectors

For collectors that have implicit deps on other fact collectors,
the default collectors can be ordered to include those early.

* default_collectors.py now uses a handful of sub lists of
collectors that can be ordered in default_collectors.collectors.

fixes #30753
fixes #30623
2017-09-28 10:36:22 -04:00
Adrian Likins
3eab636b3f Fix 'distribution' fact for ArchLinux (#30723)
* Fix 'distribution' fact for ArchLinux

Allow empty wasn't breaking out of the process_dist_files
loop, so a empty /etc/arch-release would continue searching
and eventually try /etc/os-release. The os-release parsing
works, but the distro name there is 'Arch Linux' which does
not match the 2.3 behavior of 'Archlinux'

Add a OS_RELEASE_ALIAS map for the cases where we need to get
the distro name from os-release but use an alias.

We can't include 'Archlinux' in SEARCH_STRING because a name match on its keys
but without a match on the content causes a fallback to using the first
whitespace seperated item from the file content as the name.
For os-release, that is in form 'NAME=Arch Linux'

With os-release returning the right name, this also supports the
case where there is no /etc/arch-release, but there is a /etc/os-release

Fixes #30600

* pep8 and comment cleanup
2017-09-25 15:00:31 -04:00
Adrian Likins
12404f470a Fix pkg_mgr fact on OpenBSD (#30725)
* Fix pkg_mgr fact on OpenBSD

Add a OpenBSDPkgMgrFactCollector that hardcodes pkg_mgr
to 'openbsd_pkg'. The ansible collector will choose the
OpenBSD collector if the system is OpenBSD and the 'Generic'
one otherwise.

This removes PkgMgrFactCollectors depenency on the
'system' fact being in collected_facts, which also
avoids ordering issues (if the pkg mgr fact is collected
before the system fact...)

Fixes #30623
2017-09-22 14:22:05 -04:00
Matt Clay
7714dcd04e Enable more pylint rules and fix reported issues. (#30539)
* Enable pylint unreachable test.
* Enable pylint suppressed-message test.
* Enable pylint redundant-unittest-assert test.
* Enable pylint bad-open-mode test.
* Enable pylint signature-differs test.
* Enable pylint unnecessary-pass test.
* Enable pylint unnecessary-lambda test.
* Enable pylint raising-bad-type test.
* Enable pylint logging-not-lazy test.
* Enable pylint logging-format-interpolation test.
* Enable pylint useless-else-on-loop test.
2017-09-18 23:20:32 -07:00
Dag Wieers
6a6f88d035 Clean up ACI error text (#30501)
* Clean up ACI error text
* Get rid of explicit unicode strings, use PY2
2017-09-18 11:51:58 -07:00
Toshio Kuratomi
1fa3fb45bc Update aci tests for new messages from lxml-4.0+ 2017-09-17 19:04:17 -07:00
Matt Clay
68aeaa58a8 Fix dangerous default args. (#29839) 2017-09-12 00:11:13 -07:00
Sloane Hertel
7551e8c921 AWSRetry: allow retrying on additional ClientError exceptions (#28483)
* Added the ability to extend the exception list in CloudRetry

* AWSRetry boto and boto compatible

* Updated tests to reflect boto/boto3

* Added boto to shippable requirements

* Have base_class and added_exceptions default to None in CloudRetry

AWSRetry - only retry on boto3 exceptions and remove boto requirement from tests

* Make requested changes.
2017-08-22 15:31:20 -04:00
Ganesh Nalawade
3b0e9ded91 Fix fallback and boolean check in argspec (#27994)
Fixes #27978

*  Add missing assignment for param key in fallback handling
*  Add check for frozenset type
*  Unit testcase
2017-08-10 12:10:18 -07:00
mzizzi
4648dc9702 [cloud] Add more configurable backoff implementations to CloudRetry/AWSRetry (#27251) 2017-08-08 08:56:46 -04:00
Philippe Dellaert
6d33e59ca1 Fix for subspec options validation issue #27715 (#27728)
* Fix for issue ansible/ansible#27715

* Also fixing mutually exclusive check

* Updating subspec checks
These changes take into account a spec with all features enabled and do
the following tests for subspecs:
1. Test proper specs
2. Test Alias
3. Test missing required param
4. Test mutually exclusive params
5. Test required if params
6. Test required one of params
7. Test required together params
8. Test required if params with a default value
9. Test basis subspec params
10. Test invalid subsec params
2017-08-04 19:10:38 +05:30
Peter Sprygada
7b604368d3 adds new filter plugins for network use cases (#27695)
* adds new filter plugins for network use cases

* adds parse_cli filter
* adds parse_cli_textfsm filter
* adds Template class to network_common
* adds conditional function to network_common

* fix up PEP8 issues
2017-08-04 07:47:12 -04:00
Adrian Likins
27a015f0ad add a 'min' type for gather_subset to collect nothing (#27085)
previously gather_subset=['!all'] would still gather the
min set of facts, and there was no way to collect no facts.

The 'min' specifier in gather_subset is equilivent to
exclude the minimal_gather_subset facts as well.

   gather_subset=['!all', '!min'] will collect no facts

This also lets explicitly added gather_subsets override excludes.

   gather_subset=['pkg_mgr', '!all', '!min'] will collect only the pkg_mgr
fact.
2017-08-02 11:04:01 -04:00
Adrian Likins
17ab546c48 Add 2.0-2.3 facts api compat (ansible_facts(), get_all_facts()) (#27294)
* Add 2.0-2.3 facts api compat (ansible_facts(), get_all_facts())

These are intended to provide compatibilty for modules that
use 'ansible.module_utils.facts.ansible_facts' and
'ansible.module_utils.facts.get_all_facts' from 2.0-2.3 facts
API.

Fixes #25686

Some related changes/fixes needed to provide the compat api:

* rm ansible.constants import from module_utils.facts.compat

Just use a hard coded default for gather_subset/gather_timeout
instead of trying to load it from non existent config if the
module params dont include it.

* include 'external' collectors in compat ansible_facts()

* Add facter/ohai back to the valid collector classes

facter/ohai had  gotten removed from the default_collectors
class used as the default list for all_collector_classes by
setup.py and compat.py

That made gather_subset['facter'] fail.
2017-08-01 12:51:33 -04:00
Ganesh Nalawade
97a34cf008 Add options sub spec validation (#27119)
* Add aggregate parameter validation

aggregate parameter validation will support checking each individual dict
to resolve conditions for aliases, no_log, mutually_exclusive,
required, type check, values, required_together, required_one_of
and required_if conditions in argspec. It will also set default values.

eg:
tasks:
  - name: Configure interface attribute with aggregate
    net_interface:
      aggregate:
        - {name: ge-0/0/1, description: test-interface-1, duplex: full, state: present}
        - {name: ge-0/0/2, description: test-interface-2, active: False}
    register: response
    purge: Yes

Usage:
```
from ansible.module_utils.network_common import AggregateCollection

transform = AggregateCollection(module)
param = transform(module.params.get('aggregate'))
```

Aggregate allows supports for `purge` parameter, it will instruct the module
to remove resources from remote device that hasn’t been explicitly
defined in aggregate. This is not supported by with_* iterators

Also, it improves performace as compared to with_* iterator for network device
that has seperate candidate and running datastore.
For with_* iteration the sequence of operartion is
load-config-1 (candidate db) -> commit (running db) -> load_config-2
(candidate db) -> commit (running db) ...

With aggregate the sequence of operation is
load-config-1 (candidate db) -> load-config-2 (candidate db) -> commit
(running db)

As commit is executed only once per task for aggregate it has
huge perfomance benefit for large configurations.

* Fix CI issues

* Fix review comments

*  Add support for options validation for aliases, no_log,
   mutually_exclusive, required, type check, value check,
   required_together, required_one_of and required_if
   conditions in sub-argspec.
*  Add unit test for options in argspec.
*  Reverted aggregate implementaion.

* Minor change

* Add multi-level argspec support

*  Multi-level argspec support with module's top most
   conditionals options.

* Fix unit test failure

* Add parent context in errors for sub options

* Resolve merge conflict

* Fix CI issue
2017-08-01 09:32:18 -07:00
Will Thames
5f73bdc3bf [cloud] Improve Camel to Snake conversion in EC2 module_utils (#25015)
* Make camel_to_snake work on capitalized plurals

`TargetGroupARNs` should become `target_group_arns`, not
`target_group_ar_ns`

Promote `camel_to_snake` to top layer function but prefix
it with an underscore.

Add tests for improved `_camel_to_snake` function.

Reduce use of `re.compile` as it makes no sense when the
compilation result is not reused.

* Remove unused LooseVersion check

* Fix PLURALs case for camel_to_snake

Also renamed EXPECTED_CAMELIZATION to EXPECTED_SNAKIFICATION
2017-08-01 11:09:22 -04:00
Dag Wieers
97aaf103e8 ACI module_utils library for ACI modules (#27070)
* ACI module_utils library for ACI modules

This PR includes:
- the ACI argument_spec
- an aci_login function
- an experimental aci_request function
- an aci_response function
- included the ACI team

* New prototype using ACIModule

This PR includes:
- A new ACIModule object with various useful methods
2017-07-31 12:44:27 +01:00
Matt Davis
3f1ec6b862 add custom module type validation (#27183)
* Module argument_spec now accepts a callable for the type argument, which is passed through and called with the value when appropriate. On validation/conversion failure, the name of the callable (or its type as a fallback) is used in the error message.
* adds basic smoke tests for custom callable validator functionality
2017-07-26 16:12:50 -07:00
Toshio Kuratomi
47f26ee11c Refactor the tests for _symbolic_mode_to_octal
* Move tests to their own file
* Port to a pytest parametrized test so it's easier to define new tests
* Now that _symbolic_mode_to_octal is a classmethod, we don't have to
  instantiate an AnsibleModule to test it.
* Add tests for #14994, having more than one operator per role and umask
  chmod
2017-07-24 10:08:51 -07:00
Toshio Kuratomi
ff22528b07 Consolidate boolean/mk_boolean conversion functions into a single location
Consolidate the module_utils, constants, and config functions that
convert values into booleans into a single function in module_utils.

Port code to use the module_utils.validate.convert_bool.boolean function
isntead of mk_boolean.
2017-07-17 11:48:05 -07:00
mikedlr
07a4079a81 aws.core in new aws dir in module utils - module with AnsibleAWSModule class and fail_json_aws (#25780)
* aws module utils including AnsibleAWSModule
* fail_json_aws method on AnsibleAWSModule to do fail_json nicely with AWS exceptions
* aws module util - feedback - rename to aws/core.py & improve doc strings
2017-07-11 14:01:35 -07:00
Adrian Likins
0fc0b6f059 Mv AnsibleFactCollector back to module_utils (#26150)
It was in lib/ansible/modules/system/setup.py since it
was the only thing using it, but move it back to module_utils
and add a ansible_collector.get_ansible_collector() to build
a facts collector just like the one used by setup.py

mv test_setup.py -> test_ansible_collector.py
All the code it was testing is now in ansible_collector

rm code to create 'ansible_facts' subkey from namespace

Just leave it up to the caller to do, and just return a
flat dictionary from AnsibleFactCollector.collect()
2017-07-11 10:44:22 -04:00
Brian Coca
2a041d10d2 better backwards compat handling of status
restored 'rc' inspection but only when failed is not specified
removed redundant changed from basic.py as task_executor already adds
removed redundant filters, they are tests
added aliases to tests removed from filters
fixed test to new rc handling
2017-07-05 21:44:00 -04:00
Peter Sprygada
b12ca95824 renames dict_combine to dict_merge in network_common (#26073) 2017-06-25 18:46:41 -04:00
Peter Sprygada
3aa41eda0b adds new common functions for declarative intent modules (#25210)
* adds new common functions for declarative intent modules

* adds Entity and EntityCollection
* adds dict_diff and dict_combine

* update for CI  PEP8 compliance

* more CI PEP8 fixes

* more PEP8 CI clean up

* refactors the lambda assignments into top level classes

this is to be in compliant the PEP8 CI sanity checks

* one last pep8 ci fix
2017-06-16 10:16:20 -04:00
Adrian Likins
56b7483b85 More statvfs info for mount facts rebase 12073 (#25454)
* Add more mount point statvfs info including sizes

Based on https://github.com/ansible/ansible/pull/12073

facts.utils.get_mount_size() now returns a dict of most
of the posix statvfs data, including block_size and inode
counts.

Update the facts.hardware classes that use get_mount_size() to
use the new info by mount_info.update(mount_statvfs_inof) to merge.

* add back unit tests for LinuxHardware mount/fs facts

* add test cases for facts.utils.get_mount_size
2017-06-13 15:31:41 -04:00
Abhijeet Kasurde
b89cb95609 Fix spelling mistakes (comments only) (#25564)
Original Author : klemens <ka7@github.com>

Taking over previous PR as per
https://github.com/ansible/ansible/pull/23644#issuecomment-307334525

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2017-06-12 07:55:19 +01:00
Adrian Likins
dde3dac9f8 Support NetBSD 7.1+ style ifconfig -a output (#25442)
* Support NetBSD 7.1+ style ifconfig -a output

network facts on NetBSD after 7.1 cvs would fail
because of format changes in 'ifconfig -a' output.

update code to support new and old format.

add unit tests for both based on
examples from Bruce V Chiarelli.

* wrap use of interfaces.keys() in list() for py3 compat
* sort interface ids for stability
2017-06-08 17:09:22 -04:00
Adrian Likins
f4128746d3 Cmdline fact uefi 23647 (#25446)
* Fix ansible_cmdline initrd fact for UEFI

UEFI cmdline paths use \ path sep which would
get munged by cmdline fact collection.

* Make CmdLineFactCollector easier to test

extract the parsing of the /proc/cmdline content to
_parse_proc_cmdline()

add a wrapper method for get_file_content _get_proc_cmdline()

Add unit tests of _parse_proc_cmdline based on examples
from issue #23647

Fixes #23647
2017-06-08 16:03:29 -04:00