ansible/community.general
1
0
Fork 0
mirror of https://github.com/ansible-collections/community.general.git synced 2024-09-14 20:13:21 +02:00
Code Releases Activity

Mdd module unit test docs (#31373)

Browse source 
* new documentation for unit testing - especially module unit testing

* unit test documentation reformatting and further fixes

* unit test documentation - point to online coverage reports & fix bad spaces

* Small copy edits.

* First pass copy edit / rewrite. More info needed.

* testing documentation - clean up structure, especially code coverage - reduce repetition

* module unit test documentation - improved introduction

* testing documentation - more fixes from and inspired by review from dharmabumstead

* testing documentation - fixes from mattclay + some other minor tweaks

* More copy edits.

* testing documentation - further fixes from review

* Copy edits

* Copy edits

* More copy edits.
This commit is contained in:
mikedlr 2017-10-19 21:36:57 +01:00 committed by Toshio Kuratomi
parent c02173880a
commit fbbffbabde
4 changed files with 716 additions and 26 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

35
docs/docsite/rst/dev_guide/testing.rst
View file

@ -33,9 +33,14 @@ At a high level we have the following classifications of tests:
* Tests directly against individual parts of the code base.
If you're a developer, one of the most valuable things you can do is look at the GitHub issues list and help fix bugs. We almost always prioritize bug fixing over feature development, so helping to fix bugs is one of the best things you can do.
If you're a developer, one of the most valuable things you can do is look at the GitHub
issues list and help fix bugs. We almost always prioritize bug fixing over feature
development.
Even if you're not a developer, helping to test pull requests for bug fixes and features is still immensely valuable.
Even for non developers, helping to test pull requests for bug fixes and features is still
immensely valuable. Ansible users who understand writing playbooks and roles should be
able to add integration tests and so Github pull requests with integration tests that show
bugs in action will also be a great way to help.
Testing within GitHub & Shippable
@ -47,7 +52,6 @@ Organization
When Pull Requests (PRs) are created they are tested using Shippable, a Continuous Integration (CI) tool. Results are shown at the end of every PR.
When Shippable detects an error and it can be linked back to a file that has been modified in the PR then the relevant lines will be added as a GitHub comment. For example::
The test `ansible-test sanity --test pep8` failed with the following errors:
@ -69,7 +73,6 @@ Then run the tests detailed in the GitHub comment::
ansible-test sanity --test pep8
ansible-test sanity --test validate-modules
If there isn't a GitHub comment stating what's failed you can inspect the results by clicking on the "Details" button under the "checks have failed" message at the end of the PR.
Rerunning a failing CI job
@ -91,10 +94,6 @@ If the issue persists, please contact us in ``#ansible-devel`` on Freenode IRC.
How to test a PR
================
If you're a developer, one of the most valuable things you can do is look at the GitHub issues list and help fix bugs. We almost always prioritize bug fixing over feature development, so helping to fix bugs is one of the best things you can do.
Even if you're not a developer, helping to test pull requests for bug fixes and features is still immensely valuable.
Ideally, code should add tests that prove that the code works. That's not always possible and tests are not always comprehensive, especially when a user doesn't have access to a wide variety of platforms, or is using an API or web service. In these cases, live testing against real equipment can be more valuable than automation that runs against simulated interfaces. In any case, things should always be tested manually the first time as well.
Thankfully, helping to test Ansible is pretty straightforward, assuming you are familiar with how Ansible works.
@ -188,8 +187,26 @@ If the PR does not resolve the issue, or if you see any failures from the unit/i
| some other output
| \```
Code Coverage Online
````````````````````
`The online code coverage reports <https://codecov.io/gh/ansible/ansible>` are a good way
to identify areas for testing improvement in Ansible. By following red colors you can
drill down through the reports to find files which have no tests at all. Adding both
integration and unit tests which show clearly how code should work, verify important
Ansible functions and increase testing coverage in areas where there is none is a valuable
way to help improve Ansible.
The code coverage reports only cover the ``devel`` branch of Ansible where new feature
development takes place. Pull requests and new code will be missing from the codecov.io
coverage reports so local reporting is needed. Most ``ansible-test`` commands allow you
to collect code coverage, this is particularly useful to indicate where to extend
testing. See :doc:`testing_running_locally` for more information.
Want to know more about testing?
================================
If you'd like to know more about the plans for improving testing Ansible then why not join the `Testing Working Group <https://github.com/ansible/community/blob/master/meetings/README.md>`_.
If you'd like to know more about the plans for improving testing Ansible then why not join the
`Testing Working Group <https://github.com/ansible/community/blob/master/meetings/README.md>`_.

19
docs/docsite/rst/dev_guide/testing_running_locally.rst
View file

@ -45,7 +45,19 @@ Use the ``ansible-test shell`` command to get an interactive shell in the same e
Code Coverage
=============
Add the ``--coverage`` option to any test command to collect code coverage data.
Code coverage reports make it easy to identify untested code for which more tests should
be written. Online reports are available but only cover the ``devel`` branch (see
:doc:`testing`). For new code local reports are needed.
Add the ``--coverage`` option to any test command to collect code coverage data. If you
aren't using the ``--tox`` or ``--docker`` options which create an isolated python
environment then you may have to use the ``--requirements`` option to ensure that the
correct version of the coverage module is installed
ansible-test units --coverage apt
ansible-test integration --coverage aws_lambda --tox --requirements
ansible-test coverage html
Reports can be generated in several different formats:
@ -53,4 +65,7 @@ Reports can be generated in several different formats:
* ``ansible-test coverage html`` - HTML report.
* ``ansible-test coverage xml`` - XML report.
To clear data between test runs, use the ``ansible-test coverage erase`` command.
To clear data between test runs, use the ``ansible-test coverage erase`` command. For a full list of features see the online help::
ansible-test coverage --help

133
docs/docsite/rst/dev_guide/testing_units.rst
View file

@ -2,19 +2,24 @@
Unit Tests
**********
Unit tests are small isolated tests that target a specific library or module.
Unit tests are small isolated tests that target a specific library or module. Unit tests
in Ansible are currently the only way of driving tests from python within Ansible's
continuous integration process. This means that in some circumstances the tests may be a
bit wider than just units.
.. contents:: Topics
Available Tests
===============
Unit tests can be found in `test/units <https://github.com/ansible/ansible/tree/devel/test/units>`_, notice that the directory structure matches that of ``lib/ansible/``
Unit tests can be found in `test/units
<https://github.com/ansible/ansible/tree/devel/test/units>`_. Notice that the directory
structure of the tests matches that of ``lib/ansible/``.
Running Tests
=============
Unit tests can be run across the whole code base by doing:
The Ansible unit tests can be run across the whole code base by doing:
.. code:: shell
@ -35,18 +40,22 @@ Or against a specific Python version by doing:
ansible-test units --tox --python 2.7 apt
For advanced usage see the online help::
ansible-test units --help
You can also run tests in Ansible's continuous integration system by opening a pull
request. This will automatically determine which tests to run based on the changes made
in your pull request.
Installing dependencies
=======================
``ansible-test`` has a number of dependencies , for ``units`` tests we suggest using ``tox``
``ansible-test`` has a number of dependencies. For ``units`` tests we suggest using ``tox``.
The dependencies can be installed using the ``--requirements`` argument, which will install all the required dependencies needed for unit tests. For example:
The dependencies can be installed using the ``--requirements`` argument, which will
install all the required dependencies needed for unit tests. For example:
.. code:: shell
@ -58,7 +67,11 @@ The dependencies can be installed using the ``--requirements`` argument, which w
When using ``ansible-test`` with ``--tox`` requires tox >= 2.5.0
The full list of requirements can be found at `test/runner/requirements <https://github.com/ansible/ansible/tree/devel/test/runner/requirements>`_. Requirements files are named after their respective commands. See also the `constraints <https://github.com/ansible/ansible/blob/devel/test/runner/requirements/constraints.txt>`_ applicable to all commands.
The full list of requirements can be found at `test/runner/requirements
<https://github.com/ansible/ansible/tree/devel/test/runner/requirements>`_. Requirements
files are named after their respective commands. See also the `constraints
<https://github.com/ansible/ansible/blob/devel/test/runner/requirements/constraints.txt>`_
applicable to all commands.
Extending unit tests
@ -67,22 +80,98 @@ Extending unit tests
.. warning:: What a unit test isn't
If you start writing a test that requires external services then you may be writing an integration test, rather than a unit test.
If you start writing a test that requires external services then
you may be writing an integration test, rather than a unit test.
Structuring Unit Tests
``````````````````````
Ansible drives unit tests through `pytest <https://docs.pytest.org/en/latest/>`_. This
means that tests can either be written a simple functions which are included in any file
name like ``test_<something>.py`` or as classes.
Here is an example of a function::
#this function will be called simply because it is called test_*()
def test_add()
a = 10
b = 23
c = 33
assert a + b = c
Here is an example of a class::
import unittest:
class AddTester(unittest.TestCase)
def SetUp()
self.a = 10
self.b = 23
# this function will
def test_add()
c = 33
assert self.a + self.b = c
# this function will
def test_subtract()
c = -13
assert self.a - self.b = c
Both methods work fine in most circumstances; the function-based interface is simpler and
quicker and so that's probably where you should start when you are just trying to add a
few basic tests for a module. The class-based test allows more tidy set up and tear down
of pre-requisites, so if you have many test cases for your module you may want to refactor
to use that.
Assertions using the simple ``assert`` function inside the tests will give give full
information on the cause of the failure with a trace-back of functions called during the
assertion. This means that plain asserts are recommended over other external assertion
libraries.
A number of the unit test suites include functions that are shared between several
modules, especially in the networking arena. In these cases a file is created in the same
directory, which is then included directly.
Module test case common code
````````````````````````````
Keep common code as specific as possible within the `test/units/` directory structure. For
example, if it's specific to testing Amazon modules, it should be in
`test/units/modules/cloud/amazon/`. Don't import common unit test code from directories
outside the current or parent directories.
Don't import other unit tests from a unit test. Any common code should be in dedicated
files that aren't themselves tests.
Fixtures files
``````````````
To mock out fetching results from devices, you can use ``fixtures`` to read in pre-generated data.
To mock out fetching results from devices, or provide other complex datastructures that
come from external libraries, you can use ``fixtures`` to read in pre-generated data.
Text files live in ``test/units/modules/network/PLATFORM/fixtures/``
Data is loaded using the ``load_fixture`` method
See `eos_banner test <https://github.com/ansible/ansible/blob/devel/test/units/modules/network/eos/test_eos_banner.py>`_ for a practical example.
See `eos_banner test
<https://github.com/ansible/ansible/blob/devel/test/units/modules/network/eos/test_eos_banner.py>`_
for a practical example.
Code Coverage
`````````````
Most ``ansible-test`` commands allow you to collect code coverage, this is particularly useful when to indicate where to extend testing.
If you are simulating APIs you may find that python placebo is useful. See
doc:`testing_units_modules` for more information.
Code Coverage For New or Updated Unit Tests
```````````````````````````````````````````
New code will be missing from the codecov.io coverage reports (see :doc:`testing`), so
local reporting is needed. Most ``ansible-test`` commands allow you to collect code
coverage; this is particularly useful when to indicate where to extend testing.
To collect coverage data add the ``--coverage`` argument to your ``ansible-test`` command line:
@ -99,7 +188,21 @@ Reports can be generated in several different formats:
* ``ansible-test coverage html`` - HTML report.
* ``ansible-test coverage xml`` - XML report.
To clear data between test runs, use the ``ansible-test coverage erase`` command. For a full list of features see the online help::
To clear data between test runs, use the ``ansible-test coverage erase`` command. See
:doc:`testing_units_running_locally` for more information about generating coverage
reports.
ansible-test coverage --help
.. seealso::
:doc:`testing_units_modules`
Special considerations for unit testing modules
:doc:`testing_running_locally`
Running tests locally including gathering and reporting coverage data
`Python 3 documentation - 26.4. unittest — Unit testing framework <https://docs.python.org/3/library/unittest.html>`_
The documentation of the unittest framework in python 3
`Python 2 documentation - 25.3. unittest — Unit testing framework <https://docs.python.org/3/library/unittest.html>`_
The documentation of the earliest supported unittest framework - from Python 2.6
`pytest: helps you write better programs <https://docs.pytest.org/en/latest/>`_
The documentation of pytest - the framework actually used to run Ansible unit tests

555
docs/docsite/rst/dev_guide/testing_units_modules.rst Normal file
View file

@ -0,0 +1,555 @@
****************************
Unit Testing Ansible Modules
****************************
.. contents:: Topics
Introduction
============
This document explains why, how and when you should use unit tests for Ansible modules.
The document doesn't apply to other parts of Ansible for which the recommendations are
normally closer to the Python standard. There is basic documentation for Ansible unit
tests in the developer guide :doc:`testing_units`. This document should
be readable for a new Ansible module author. If you find it incomplete or confusing,
please open a bug or ask for help on Ansible IRC.
What Are Unit Tests?
====================
Ansible includes a set of unit tests in the :file:`test/unit` directory. These tests primarily cover the
internals but can also can cover Ansible modules. The structure of the unit tests matches
the structure of the code base, so the tests that reside in the :file:`test/unit/modules/` directory
are organized by module groups.
Integration tests can be used for most modules, but there are situations where
cases cannot be verified using integration tests. This means that Ansible unit test cases
may extend beyond testing only minimal units and in some cases will include some
level of functional testing.
Why Use Unit Tests?
===================
Ansible unit tests have advantages and disadvantages. It is important to understand these.
Advantages include:
* Most unit tests are much faster than most Ansible integration tests. The complete suite
of unit tests can be run regularly by a developer on their local system.
* Unit tests can be run by developers who don't have access to the system which the module is
designed to work on, allowing a level of verification that changes to core functions
haven't broken module expectations.
* Unit tests can easily substitute system functions allowing testing of software that
would be impractical. For example, the ``sleep()`` function can be replaced and we check
that a ten minute sleep was called without actually waiting ten minutes.
* Unit tests are run on different Python versions. This allows us to
ensure that the code behaves in the same way on different Python versions.
There are also some potential disadvantages of unit tests. Unit tests don't normally
directly test actual useful valuable features of software, instead just internal
implementation
* Unit tests that test the internal, non-visible features of software may make
refactoring difficult if those internal features have to change (see also naming in How
below)
* Even if the internal feature is working correctly it is possible that there will be a
problem between the internal code tested and the actual result delivered to the user
Normally the Ansible integration tests (which are written in Ansible YAML) provide better
testing for most module functionality. If those tests already test a feature and perform
well there may be little point in providing a unit test covering the same area as well.
When To Use Unit Tests
======================
There are a number of situations where unit tests are a better choice than integration
tests. For example, testing things which are impossible, slow or very difficult to test
with integration tests, such as:
* Forcing rare / strange / random situations that can't be forced, such as specific network
failures and exceptions
* Extensive testing of slow configuration APIs
* Situations where the integration tests cannot be run as part of the main Ansible
continuous integraiton running in Shippable.
Providing quick feedback
------------------------
Example:
A single step of the rds_instance test cases can take up to 20
minutes (the time to create an RDS instance in Amazon). The entire
test run can last for well over an hour. All 16 of the unit tests
complete execution in less than 2 seconds.
The time saving provided by being able to run the code in a unit test makes it worth
creating a unit test when bug fixing a module, even if those tests do not often identify
problems later. As a basic goal, every module should have at least one unit test which
will give quick feedback in easy cases without having to wait for the integration tests to
complete.
Ensuring correct use of external interfaces
-------------------------------------------
Unit tests can check the way in which external services are run to ensure that they match
specifications or are as efficient as possible *even when the final output will not be changed*.
Example:
Package managers are often far more efficient when installing multiple packages at once
rather than each package separately. The final result is the
same: the packages are all installed, so the efficiency is difficult to verify through
integration tests. By providing a mock package manager and verifying that it is called
once, we can build a valuable test for module efficiency.
Another related use is in the situation where an API has versions which behave
differently. A programmer working on a new version may change the module to work with the
new API version and unintentially break the old version. A test case
which checks that the call happens properly for the old version can help avoid the
problem. In this situation it is very important to include version numbering in the test case
name (see `Naming unit tests`_ below).
Providing specific design tests
--------------------------------
By building a requirement for a particular part of the
code and then coding to that requirement, unit tests _can_ sometimes improve the code and
help future developers understand that code.
Unit tests that test internal implementation details of code, on the other hand, almost
always do more harm than good. Testing that your packages to install are stored in a list
would slow down and confuse a future developer who might need to change that list into a
dictionary for efficiency. This problem can be reduced somewhat with clear test naming so
that the future developer immediately knows to delete the test case, but it is often
better to simply leave out the test case altogether and test for a real valuable feature
of the code, such as installing all of the packages supplied as arguments to the module.
How to unit test Ansible modules
================================
There are a number of techniques for unit testing modules. Beware that most
modules without unit tests are structured in a way that makes testing quite difficult and
can lead to very complicated tests which need more work than the code. Effectively using unit
tests may lead you to restructure your code. This is often a good thing and leads
to better code overall. Good restructuring can make your code clearer and easier to understand.
Naming unit tests
-----------------
Unit tests should have logical names. If a developer working on the module being tested
breaks the test case, it should be easy to figure what the unit test covers from the name.
If a unit test is designed to verify compatibility with a specific software or API version
then include the version in the name of the unit test.
As an example, ``test_v2_state_present_should_call_create_server_with_name()`` would be a
good name, ``test_create_server()`` would not be.
Use of Mocks
------------
Mock objects (from https://docs.python.org/3/library/unittest.mock.html) can be very
useful in building unit tests for special / difficult cases, but they can also
lead to complex and confusing coding situations. One good use for mocks would be in
simulating an API. As for 'six', the 'mock' python package is bundled with Ansible (use
'import ansible.compat.tests.mock'). See for example
Ensuring failure cases are visible with mock objects
----------------------------------------------------
Functions like module.fail_json() are normally expected to terminate execution. When you
run with a mock module object this doesn't happen since the mock always returns another mock
from a function call. You can set up the mock to raise an exception as shown above, or you can
assert that these functions have not been called in each test. For example::
module = MagicMock()
function_to_test(module, argument)
module.fail_json.assert_not_called()
This applies not only to calling the main module but almost any other
function in a module which gets the module object.
Mocking of the actual module
----------------------------
The setup of an actual module is quite complex (see `Passing Arguments`_ below) and often
isn't needed for most functions which use a module. Instead you can use a mock object as
the module and create any module attributes needed by the function you are testing. If
you do this, beware that the module exit functions need special handling as mentioned
above, either by throwing an exception or ensuring that they haven't been called. For example::
class AnsibleExitJson(Exception):
"""Exception class to be raised by module.exit_json and caught by the test case"""
pass
#you may also do the same to fail json
module=MagicMock()
module.exit_json.side_effect = AnsibleExitJson(Exception)
with self.assertRaises(AnsibleExitJson) as result:
return = my_module.test_this_function(module, argument)
module.fail_json.assert_not_called()
assert return["changed"] == True
API definition with unit test cases
-----------------------------------
API interaction is usually best tested with the function tests defined in Ansible's
integration testing section, which run against the actual API. There are several cases
where the unit tests are likely to work better.
Defining a module against an API specification
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This case is especially important for modules interacting with web services, which provide
an API that Ansible uses but which are beyond the control of the user.
By writing a custom emulation of the calls that return data from the API, we can ensure
that only the features which are clearly defined in the specification of the API are
present in the message. This means that we can check that we use the correct
parameters and nothing else.
*Example: in rds_instance unit tests a simple instance state is defined*::
def simple_instance_list(status, pending):
return {u'DBInstances': [{u'DBInstanceArn': 'arn:aws:rds:us-east-1:1234567890:db:fakedb',
u'DBInstanceStatus': status,
u'PendingModifiedValues': pending,
u'DBInstanceIdentifier': 'fakedb'}]}
This is then used to create a list of states::
rds_client_double = MagicMock()
rds_client_double.describe_db_instances.side_effect = [
simple_instance_list('rebooting', {"a": "b", "c": "d"}),
simple_instance_list('available', {"c": "d", "e": "f"}),
simple_instance_list('rebooting', {"a": "b"}),
simple_instance_list('rebooting', {"e": "f", "g": "h"}),
simple_instance_list('rebooting', {}),
simple_instance_list('available', {"g": "h", "i": "j"}),
simple_instance_list('rebooting', {"i": "j", "k": "l"}),
simple_instance_list('available', {}),
simple_instance_list('available', {}),
]
These states are then used as returns from a mock object to ensure that the ``await`` function
waits through all of the states that would mean the RDS instance has not yet completed
configuration::
rds_i.await_resource(rds_client_double, "some-instance", "available", mod_mock,
await_pending=1)
assert(len(sleeper_double.mock_calls) > 5), "await_pending didn't wait enough"
By doing this we check that the ``await`` function will keep waiting through
potentially unusual that it would be impossible to reliably trigger through the
integration tests but which happen unpredictably in reality.
Defining a module to work against multiple API versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This case is especially important for modules interacting with many different versions of
software; for example, package installation modules that might be expected to work with
many different operating system versions.
By using previously stored data from various versions of an API we can ensure that the
code is tested against the actual data which will be sent from that version of the system
even when the version is very obscure and unlikely to be available during testing.
Ansible special cases for unit testing
======================================
There are a number of special cases for unit testing the environment of an Ansible module.
The most common are documented below, and suggestions for others can be found by looking
at the source code of the existing unit tests or asking on the Ansible IRC channel or mailing
lists.
Module argument processing
--------------------------
There are two problems with running the main function of a module:
* Since the module is supposed to accept arguments on ``STDIN`` it is a bit difficult to
set up the arguments correctly so that the module will get them as parameters.
* All modules should finish by calling either the ``module.fail_json`` or
``module.exit_json``, but these won't work correctly in a testing environment.
Passing Arguments
-----------------
.. This section should be updated once https://github.com/ansible/ansible/pull/31456 is
closed since the function below will be provided in a library file.
To pass arguments to a module correctly, use a function that stores the
parameters in a special string variable. Module creation and argument processing is
handled through the AnsibleModule object in the basic section of the utilities. Normally
this accepts input on ``STDIN``, which is not convenient for unit testing. When the special
variable is set it will be treated as if the input came on ``STDIN`` to the module.::
import json
from ansible.module_utils._text import to_bytes
def set_module_args(args):
args = json.dumps({'ANSIBLE_MODULE_ARGS': args})
basic._ANSIBLE_ARGS = to_bytes(args)
simply call that function before setting up your module
def test_already_registered(self):
set_module_args({
'activationkey': 'key',
'username': 'user',
'password': 'pass',
})
Handling exit correctly
-----------------------
.. This section should be updated once https://github.com/ansible/ansible/pull/31456 is
closed since the exit and failure functions below will be provided in a library file.
The ``module.exit_json()`` function won't work properly in a testing environment since it
writes error information to ``STDOUT`` upon exit, where it
is difficult to examine. This can be mitigated by replacing it (and module.fail_json) with
a function that raises an exception::
def exit_json(*args, **kwargs):
if 'changed' not in kwargs:
kwargs['changed'] = False
raise AnsibleExitJson(kwargs)
Now you can ensure that the first function called is the one you expected simply by
testing for the correct exception::
def test_returned_value(self):
set_module_args({
'activationkey': 'key',
'username': 'user',
'password': 'pass',
})
with self.assertRaises(AnsibleExitJson) as result:
my_module.main()
The same technique can be used to replace ``module.fail_json()`` (which is used for failure
returns from modules) and for the ``aws_module.fail_json_aws()`` (used in modules for Amazon
Web Services).
Running the main function
-------------------------
If you do want to run the actual main function of a module you must import the module, set
the arguments as above, set up the appropriate exit exception and then run the module::
# This test is based around pytest's features for individual test functions
import pytest
import ansible.modules.module.group.my_modulle as my_module
def test_main_function(monkeypatch):
monkeypatch.setattr(my_module.AnsibleModule, "exit_json", fake_exit_json)
set_module_args({
'activationkey': 'key',
'username': 'user',
'password': 'pass',
})
my_module.main()
Handling calls to external executables
--------------------------------------
Module must use AnsibleModule.run_command in order to execute an external command. This
method needs to be mocked:
Here is a simple mock of AnsibleModule.run_command (taken from test/units/modules/packaging/os/test_rhn_register.py and
test/units/modules/packaging/os/rhn_utils.py)::
with patch.object(basic.AnsibleModule, 'run_command') as run_command:
run_command.return_value = 0, '', '' # successful execution, no output
with self.assertRaises(AnsibleExitJson) as result:
self.module.main()
self.assertFalse(result.exception.args[0]['changed'])
# Check that run_command has been called
run_command.assert_called_once_with('/usr/bin/command args')
self.assertEqual(run_command.call_count, 1)
self.assertFalse(run_command.called)
A Complete Example
------------------
The following example is a complete skeleton that reuses the mocks explained above and adds a new
mock for Ansible.get_bin_path::
import json
from ansible.compat.tests import unittest
from ansible.compat.tests.mock import patch
from ansible.module_utils import basic
from ansible.module_utils._text import to_bytes
from ansible.modules.namespace import my_module
def set_module_args(args):
"""prepare arguments so that they will be picked up during module creation"""
args = json.dumps({'ANSIBLE_MODULE_ARGS': args})
basic._ANSIBLE_ARGS = to_bytes(args)
class AnsibleExitJson(Exception):
"""Exception class to be raised by module.exit_json and caught by the test case"""
pass
class AnsibleFailJson(Exception):
"""Exception class to be raised by module.fail_json and caught by the test case"""
pass
def exit_json(*args, **kwargs):
"""function to patch over exit_json; package return data into an exception"""
if 'changed' not in kwargs:
kwargs['changed'] = False
raise AnsibleExitJson(kwargs)
def fail_json(*args, **kwargs):
"""function to patch over fail_json; package return data into an exception"""
kwargs['failed'] = True
raise AnsibleFailJson(kwargs)
def get_bin_path(self, arg, required=False):
"""Mock AnsibleModule.get_bin_path"""
if arg.endswith('my_command'):
return '/usr/bin/my_command'
else:
if required:
fail_json(msg='%r not found !' % arg)
class TestMyModule(unittest.TestCase):
def setUp(self):
self.mock_module_helper = patch.multiple(basic.AnsibleModule,
exit_json=exit_json,
fail_json=fail_json,
get_bin_path=get_bin_path)
self.mock_module_helper.start()
self.addCleanup(self.mock_module_helper.stop)
def test_module_fail_when_required_args_missing(self):
with self.assertRaises(AnsibleFailJson):
set_module_args({})
self.module.main()
def test_ensure_command_called(self):
set_module_args({
'param1': 10,
'param2': 'test',
})
with patch.object(basic.AnsibleModule, 'run_command') as mock_run_command:
stdout = 'configuration updated'
stderr = ''
rc = 0
mock_run_command.return_value = rc, stdout, stderr # successful execution
with self.assertRaises(AnsibleExitJson) as result:
my_module.main()
self.assertFalse(result.exception.args[0]['changed']) # ensure result is changed
mock_run_command.assert_called_once_with('/usr/bin/my_command --value 10 --name test')
Restructuring modules to enable testing module set up and other processes
-------------------------------------------------------------------------
Often modules have a main() function which sets up the module and then performs other
actions. This can make it difficult to check argument processing. This can be made easier by
moving module configuration and initialization into a separate function. For example::
argument_spec = dict(
# module function variables
state=dict(choices=['absent', 'present', 'rebooted', 'restarted'], default='present'),
apply_immediately=dict(type='bool', default=False),
wait=dict(type='bool', default=False),
wait_timeout=dict(type='int', default=600),
allocated_storage=dict(type='int', aliases=['size']),
db_instance_identifier=dict(aliases=["id"], required=True),
)
def setup_module_object():
module = AnsibleAWSModule(
argument_spec=argument_spec,
required_if=required_if,
mutually_exclusive=[['old_instance_id', 'source_db_instance_identifier',
'db_snapshot_identifier']],
)
return module
def main():
module = setup_module_object()
validate_parameters(module)
conn = setup_client(module)
return_dict = run_task(module, conn)
module.exit_json(**return_dict)
This now makes it possible to run tests against the module initiation function::
def test_rds_module_setup_fails_if_db_instance_identifier_parameter_missing():
# db_instance_identifier parameter is missing
set_module_args({
'state': 'absent',
'apply_immediately': 'True',
})
with self.assertRaises(AnsibleFailJson) as result:
self.module.setup_json
See also ``test/units/module_utils/aws/test_rds.py``
Note that the argument_spec dictionary is visible in a module variable. This has
advantages, both in allowing explicit testing of the arguments and in allowing the easy
creation of module objects for testing.
The same restructuring technique can be valuable for testing other functionality, such as the part of the module which queries the object that the module configures.
Traps for maintaining Python 2 compatibility
============================================
If you use the ``mock`` library from the Python 2.6 standard library, a number of the
assert functions are missing but will return as if successful. This means that test cases should take great care *not* use
functions marked as _new_ in the Python 3 documentation, since the tests will likely always
succeed even if the code is broken when run on older versions of Python.
A helpful development approach to this should be to ensure that all of the tests have been
run under Python 2.6 and that each assertion in the test cases has been checked to work by breaking
the code in Ansible to trigger that failure.
.. seealso::
:doc:`testing_units`
Ansible unit tests documentation
:doc:`testing_running_locally`
Running tests locally including gathering and reporting coverage data
:doc:`developing_modules`
How to develop modules
`Python 3 documentation - 26.4. unittest — Unit testing framework <https://docs.python.org/3/library/unittest.html>`_
The documentation of the unittest framework in python 3
`Python 2 documentation - 25.3. unittest — Unit testing framework <https://docs.python.org/3/library/unittest.html>`_
The documentation of the earliest supported unittest framework - from Python 2.6
`pytest: helps you write better programs <https://docs.pytest.org/en/latest/>`_
The documentation of pytest - the framework actually used to run Ansible unit tests
`Development Mailing List <http://groups.google.com/group/ansible-devel>`_
Mailing list for development topics
`Testing Your Code (from The Hitchhiker's Guide to Python!) <http://docs.python-guide.org/en/latest/writing/tests/>`_
General advice on testing Python code
`Uncle Bob's many videos on YouTube <https://www.youtube.com/watch?v=QedpQjxBPMA&list=PLlu0CT-JnSasQzGrGzddSczJQQU7295D2>`_
Unit testing is a part of the of various philosophies of software development, including
Extreme Programming (XP), Clean Coding. Uncle Bob talks through how to benfit from this
`"Why Most Unit Testing is Waste" http://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf`
An article warning against the costs of unit testing
`'A Response to "Why Most Unit Testing is Waste"' https://henrikwarne.com/2014/09/04/a-response-to-why-most-unit-testing-is-waste/`
An response pointing to how to maintain the value of unit tests