ansible/community.general
1
0
Fork 0
mirror of https://github.com/ansible-collections/community.general.git synced 2024-09-14 20:13:21 +02:00
Code Releases Activity

rewrite of the developer guide, part 1 (#45179)

Browse source 
* rewrite of the developer guide, part 1
This commit is contained in:
Alicia Cozine 2018-09-07 08:57:36 -05:00 committed by GitHub
parent 1325ddbb0b
commit 9a76441c02
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
39 changed files with 1231 additions and 2882 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/docsite/rst/community/other_tools_and_programs.rst
View file

@ -1,3 +1,5 @@
.. _other_tools_and_programs:
########################
Other Tools And Programs
########################

149
docs/docsite/rst/dev_guide/debugging.rst Normal file
View file

@ -0,0 +1,149 @@
.. _debugging:
*****************
Debugging modules
*****************
Debugging (local)
=================
To break into a module running on ``localhost`` and step through with the debugger:
- Set a breakpoint in the module: ``import pdb; pdb.set_trace()``
- Run the module on the local machine: ``$ python -m pdb ./my_new_test_module.py ./args.json``
Debugging (remote)
==================
To debug a module running on a remote target (i.e. not ``localhost``):
#. On your controller machine (running Ansible) set ``ANSIBLE_KEEP_REMOTE_FILES=1`` to tell Ansible to retain the modules it sends to the remote machine instead of removing them after you playbook runs.
#. Run your playbook targeting the remote machine and specify ``-vvvv`` (verbose) to display the remote location Ansible is using for the modules (among many other things).
#. Take note of the directory Ansible used to store modules on the remote host. This directory is usually under the home directory of your ``ansible_ssh_user``, in the form ``~/.ansible/tmp/ansible-tmp-...``.
#. SSH into the remote target after the playbook runs.
#. Navigate to the directory you noted in step 3.
#. Extract the module you want to debug from the zipped file that Ansible sent to the remote host: ``$ python my_test_module.py explode``. Ansible will expand the module into ``./debug-dir``. You can optionally run the zipped file by specifying ``python my_test_module.py``.
#. Navigate to the debug directory: ``$ cd debug-dir``.
#. Modify or set a breakpoint in ``ansible_module_my_test_module.py``.
#. Ensure that the unzipped module is executable: ``$ chmod 755 ansible_module_my_test_module.py``.
#. Run the unzipped module directly, passing the ``args`` file that contains the params that were originally passed: ``$ ./ansible_module_my_test_module.py args``. This approach is good for reproducing behavior as well as modifying the parameters for debugging.
.. _debugging_ansiblemodule_based_modules:
Debugging AnsibleModule-based modules
=====================================
.. tip::
If you're using the :file:`hacking/test-module` script then most of this
is taken care of for you. If you need to do some debugging of the module
on the remote machine that the module will actually run on or when the
module is used in a playbook then you may need to use this information
instead of relying on test-module.
Starting with Ansible 2.1, AnsibleModule-based modules are put together as
a zip file consisting of the module file and the various python module
boilerplate inside of a wrapper script instead of as a single file with all of
the code concatenated together. Without some help, this can be harder to
debug as the file needs to be extracted from the wrapper in order to see
what's actually going on in the module. Luckily the wrapper script provides
some helper methods to do just that.
If you are using Ansible with the :envvar:`ANSIBLE_KEEP_REMOTE_FILES`
environment variables to keep the remote module file, here's a sample of how
your debugging session will start:
.. code-block:: shell-session
$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible localhost -m ping -a 'data=debugging_session' -vvv
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: badger
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595 `" )'
<127.0.0.1> PUT /var/tmp/tmpjdbJ1w TO /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping
<127.0.0.1> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping'
localhost | SUCCESS => {
"changed": false,
"invocation": {
"module_args": {
"data": "debugging_session"
},
"module_name": "ping"
},
"ping": "debugging_session"
}
Setting :envvar:`ANSIBLE_KEEP_REMOTE_FILES` to ``1`` tells Ansible to keep the
remote module files instead of deleting them after the module finishes
executing. Giving Ansible the ``-vvv`` option makes Ansible more verbose.
That way it prints the file name of the temporary module file for you to see.
If you want to examine the wrapper file you can. It will show a small python
script with a large, base64 encoded string. The string contains the module
that is going to be executed. Run the wrapper's explode command to turn the
string into some python files that you can work with:
.. code-block:: shell-session
$ python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping explode
Module expanded into:
/home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/debug_dir
When you look into the debug_dir you'll see a directory structure like this::
├── ansible_module_ping.py
├── args
└── ansible
├── __init__.py
└── module_utils
├── basic.py
└── __init__.py
* :file:`ansible_module_ping.py` is the code for the module itself. The name
is based on the name of the module with a prefix so that we don't clash with
any other python module names. You can modify this code to see what effect
it would have on your module.
* The :file:`args` file contains a JSON string. The string is a dictionary
containing the module arguments and other variables that Ansible passes into
the module to change its behaviour. If you want to modify the parameters
that are passed to the module, this is the file to do it in.
* The :file:`ansible` directory contains code from
:mod:`ansible.module_utils` that is used by the module. Ansible includes
files for any :`module:`ansible.module_utils` imports in the module but not
any files from any other module. So if your module uses
:mod:`ansible.module_utils.url` Ansible will include it for you, but if
your module includes :mod:`requests` then you'll have to make sure that
the python requests library is installed on the system before running the
module. You can modify files in this directory if you suspect that the
module is having a problem in some of this boilerplate code rather than in
the module code you have written.
Once you edit the code or arguments in the exploded tree you need some way to
run it. There's a separate wrapper subcommand for this:
.. code-block:: shell-session
$ python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping execute
{"invocation": {"module_args": {"data": "debugging_session"}}, "changed": false, "ping": "debugging_session"}
This subcommand takes care of setting the PYTHONPATH to use the exploded
:file:`debug_dir/ansible/module_utils` directory and invoking the script using
the arguments in the :file:`args` file. You can continue to run it like this
until you understand the problem. Then you can copy it back into your real
module file and test that the real module works via :command:`ansible` or
:command:`ansible-playbook`.
.. note::
The wrapper provides one more subcommand, ``excommunicate``. This
subcommand is very similar to ``execute`` in that it invokes the exploded
module on the arguments in the :file:`args`. The way it does this is
different, however. ``excommunicate`` imports the :func:`main`
function from the module and then calls that. This makes excommunicate
execute the module in the wrapper's process. This may be useful for
running the module under some graphical debuggers but it is very different
from the way the module is executed by Ansible itself. Some modules may
not work with ``excommunicate`` or may behave differently than when used
with Ansible normally. Those are not bugs in the module; they're
limitations of ``excommunicate``. Use at your own risk.

12
docs/docsite/rst/dev_guide/developing_api.rst
View file

@ -1,18 +1,19 @@
.. _developing_api:
**********
Python API
==========
**********
.. contents:: Topics
.. note:: This API is intended for internal Ansible use. Ansible may make changes to this API at any time that could break backward compatibility with older versions of the API. Because of this, external use is not supported by Ansible.
.. note:: This API is intended for internal Ansible use. Ansible may make changes to this API at any time that could break backward compatibility with older versions of the API. Because of this, external use is not supported by Ansible.
There are several ways to use Ansible from an API perspective. You can use
the Ansible Python API to control nodes, you can extend Ansible to respond to various Python events, you can
write plugins, and you can plug in inventory data from external data sources. This document
gives a basic overview and examples of the Ansible execution and playbook API.
If you would like to use Ansible programmatically from a language other than Python, trigger events asynchronously,
If you would like to use Ansible programmatically from a language other than Python, trigger events asynchronously,
or have access control and logging demands, please see the `Ansible Tower documentation <https://docs.ansible.com/ansible-tower/>`_.
.. note:: Because Ansible relies on forking processes, this API is not thread safe.
@ -20,7 +21,7 @@ or have access control and logging demands, please see the `Ansible Tower docume
.. _python_api_example:
Python API example
------------------
==================
This example is a simple demonstration that shows how to minimally run a couple of tasks::
@ -100,7 +101,7 @@ This example is a simple demonstration that shows how to minimally run a couple
# we always need to cleanup child procs and the structres we use to communicate with them
if tqm is not None:
tqm.cleanup()
# Remove ansible tmpdir
shutil.rmtree(C.DEFAULT_LOCAL_TMP, True)
@ -122,4 +123,3 @@ command line tools (``lib/ansible/cli/``) is `available on Github <https://githu
Mailing list for development topics
`irc.freenode.net <http://irc.freenode.net>`_
#ansible IRC chat channel

4
docs/docsite/rst/dev_guide/developing_core.rst
View file

@ -1,5 +1,6 @@
**********************************
Developing the Ansible Core Engine
==================================
**********************************
Although many of the pieces of the Ansible Core Engine are plugins that can be
swapped out via playbook directives or configuration, there are still pieces
@ -21,4 +22,3 @@ those pieces work together.
The development mailing list
`irc.freenode.net <http://irc.freenode.net>`_
#ansible-devel IRC chat channel

40
docs/docsite/rst/dev_guide/developing_inventory.rst
View file

@ -1,7 +1,8 @@
.. _developing_inventory:
Developing Dynamic Inventory
============================
****************************
Developing dynamic inventory
****************************
.. contents:: Topics
:local:
@ -11,9 +12,9 @@ including cloud sources, using the supplied :ref:`inventory plugins <inventory_p
If the source you want is not currently covered by existing plugins, you can create your own as with any other plugin type.
In previous versions you had to create a script or program that can output JSON in the correct format when invoked with the proper arguments.
You can still use and write inventory scripts, as we ensured backwards compatiblity via the :ref:`script inventory plugin <script_inventory>`
You can still use and write inventory scripts, as we ensured backwards compatibility via the :ref:`script inventory plugin <script_inventory>`
and there is no restriction on the programming language used.
If you choose to write a script, however, you will need to implement some features youself.
If you choose to write a script, however, you will need to implement some features yourself.
i.e caching, configuration management, dynamic variable and group composition, etc.
While with :ref:`inventory plugins <inventory_plugins>` you can leverage the Ansible codebase to add these common features.
@ -21,7 +22,7 @@ While with :ref:`inventory plugins <inventory_plugins>` you can leverage the Ans
.. _inventory_sources:
Inventory sources
-----------------
=================
Inventory sources are strings (i.e what you pass to ``-i`` in the command line),
they can represent a path to a file/script or just be the raw data for the plugin to use.
@ -44,11 +45,10 @@ Here are some plugins and the type of source they use:
+--------------------------------------------+--------------------------------------+
.. _developing_inventory_inventory_plugins:
Inventory Plugins
-----------------
Inventory plugins
=================
Like most plugin types (except modules) they must be developed in Python, since they execute on the controller they should match the same requirements :ref:`control_machine_requirements`.
@ -61,8 +61,8 @@ When using the 'persistent' cache, inventory plugins can also use the configured
.. _developing_an_inventory_plugin:
Developing an Inventory Plugin
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Developing an inventory plugin
------------------------------
The first thing you want to do is use the base class:
@ -91,7 +91,7 @@ For the bulk of the work in the plugin, We mostly want to deal with 2 methods ``
.. _inventory_plugin_verify_file:
verify_file
"""""""""""
^^^^^^^^^^^
This method is used by Ansible to make a quick determination if the inventory source is usable by the plugin. It does not need to be 100% accurate as there might be overlap in what plugins can handle and Ansible will try the enabled plugins (in order) by default.
@ -128,7 +128,7 @@ This method is just to expedite the inventory process and avoid uneccessary pars
.. _inventory_plugin_parse:
parse
"""""
^^^^^
This method does the bulk of the work in the plugin.
@ -194,8 +194,8 @@ For examples on how to implement an inventory plug in, see the source code here:
.. _inventory_source_common_format:
inventory source common format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Inventory source common format
------------------------------
To simplify development, most plugins use a mostly standard configuration file as the inventory source, YAML based and with just one required field ``plugin`` which should contain the name of the plugin that is expected to consume the file.
Depending on other common features used, other fields might be needed, but each plugin can also add it's own custom options as needed.
@ -204,7 +204,7 @@ For example, if you use the integrated caching, ``cache_plugin``, ``cache_timeou
.. _inventory_development_auto:
The 'auto' plugin
^^^^^^^^^^^^^^^^^
-----------------
Since Ansible 2.5, we include the :ref:`auto inventory plugin <auto_inventory>` enabled by default, which itself just loads other plugins if they use the common YAML configuration format that specifies a ``plugin`` field that matches an inventory plugin name, this makes it easier to use your plugin w/o having to update configurations.
@ -212,8 +212,8 @@ Since Ansible 2.5, we include the :ref:`auto inventory plugin <auto_inventory>`
.. _inventory_scripts:
.. _developing_inventory_scripts:
Inventory Scripts
-----------------
Inventory scripts
=================
Even though we now have inventory plugins, we still support inventory scripts, not only for backwards compatibility but also to allow users to leverage other programming languages.
@ -221,7 +221,7 @@ Even though we now have inventory plugins, we still support inventory scripts, n
.. _inventory_script_conventions:
Inventory script conventions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
----------------------------
Inventory scripts must accept the ``--list`` and ``--host <hostname>`` arguments, other arguments are allowed but Ansible will not use them.
They might still be useful for when executing the scripts directly.
@ -264,8 +264,8 @@ Printing variables is optional. If the script does not do this, it should print
.. _inventory_script_tuning:
Tuning the External Inventory Script
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Tuning the external inventory script
------------------------------------
.. versionadded:: 1.3

95
docs/docsite/rst/dev_guide/developing_locally.rst Normal file
View file

@ -0,0 +1,95 @@
.. _using_local_modules_and_plugins:
.. _developing_locally:
**********************************
Adding modules and plugins locally
**********************************
.. contents:: Topics
:local:
The easiest, quickest, and most popular way to extend Ansible is to copy or write a module or a plugin for local use. You can store local modules and plugins on your Ansible control node for use within your team or organization. You can also share a local plugin or module by embedding it in a role and publishing it on Ansible Galaxy. If you've been using roles off Galaxy, you may have been using local modules and plugins without even realizing it. If you're using a local module or plugin that already exists, this page is all you need.
Extending Ansible with local modules and plugins offers lots of shortcuts:
* You can copy other people's modules and plugins.
* If you're writing a new module, you can choose any programming language you like.
* You don't have to clone the main Ansible repo.
* You don't have to open a pull request.
* You don't have to add tests (though we recommend that you do!).
To save a local module or plugin so Ansible can find and use it, drop the module or plugin in the correct "magic" directory. For local modules, use the name of the file as the module name: for example, if the module file is ``~/.ansible/plugins/modules/local_users.py``, use ``local_users`` as the module name.
.. _modules_vs_plugins:
Modules and plugins: what's the difference?
===========================================
If you're looking to add local functionality to Ansible, you may be wondering whether you need a module or a plugin. Here's a quick overview of the differences:
* Modules are reusable, standalone scripts that can be used by the Ansible API, the :command:`ansible` command, or the :command:`ansible-playbook` command. Modules provide a defined interface, accepting arguments and returning information to Ansible by printing a JSON string to stdout before exiting.
* Plugins are shared code that can be used by any module. They provide abilities like cacheing information or copying files that are useful for many modules.
.. _local_modules:
Adding a module locally
=======================
Ansible automatically loads all executable files found in certain directories as modules, so you can create or add a local module in any of these locations:
* any directory added to the ``ANSIBLE_LIBRARY`` environment variable (``$ANSIBLE_LIBRARY`` takes a colon-separated list like ``$PATH``)
* ``~/.ansible/plugins/modules/``
* ``/usr/share/ansible/plugins/modules/``
Once you save your module file in one of these locations, Ansible will load it and you can use it in any local task, playbook, or role.
To confirm that ``my_custom_module`` is available:
* type ``ansible-doc -t module my_custom_module``. You should see the documentation for that module.
To use a local module only in certain playbooks:
* store it in a sub-directory called ``library`` in the directory that contains the playbook(s)
To use a local module only in a single role:
* store it in a sub-directory called ``library`` within that role
.. _distributing_plugins:
.. _local_plugins:
Adding a plugin locally
=======================
Ansible loads plugins automatically too, loading each type of plugin separately from a directory named for the type of plugin. Here's the full list of plugin directory names:
* action_plugins*
* cache_plugins
* callback_plugins
* connection_plugins
* filter_plugins*
* inventory_plugins
* lookup_plugins
* shell_plugins
* strategy_plugins
* test_plugins*
* vars_plugins
You can create or add a local plugin in any of these locations:
* any directory added to the relevant ``ANSIBLE_plugin_type_PLUGINS`` environment variable (these variables, such as ``$ANSIBLE_INVENTORY_PLUGINS`` and ``$ANSIBLE_VARS_PLUGINS`` take colon-separated lists like ``$PATH``)
* the directory named for the correct ``plugin_type`` within ``~/.ansible/plugins/`` - for example, ``~/.ansible/plugins/callback_plugins``
* the directory named for the correct ``plugin_type`` within ``/usr/share/ansible/plugins/`` - for example, ``/usr/share/ansible/plugins/plugin_type/action_plugins``
Once your plugin file is in one of these locations, Ansible will load it and you can use it in a any local module, task, playbook, or role.
To confirm that ``plugins/plugin_type/my_custom_plugin`` is available:
* type ``ansible-doc -t <plugin_type> my_custom_lookup_plugin``. For example, ``ansible-doc -t lookup my_custom_lookup_plugin``. You should see the documentation for that plugin. This works for all plugin types except the ones marked with ``*`` in the list above - see :ref:`ansible-doc` for more details.
To use your local plugin only in certain playbooks:
* store it in a sub-directory for the correct ``plugin_type`` (for example, ``callback_plugins`` or ``inventory_plugins``) in the directory that contains the playbook(s)
To use your local plugin only in a single role:
* store it in a sub-directory for the correct ``plugin_type`` (for example, ``cache_plugins`` or ``strategy_plugins``) within that role
When shipped as part of a role, the plugin will be available as soon as the role is called in the play.

9
docs/docsite/rst/dev_guide/developing_module_utilities.rst
View file

@ -1,14 +1,15 @@
.. _appendix_module_utilities:
**************************
Appendix: Module Utilities
``````````````````````````
**************************
Ansible provides a number of module utilities that provide helper functions that you can use when developing your own modules. The `basic.py` module utility provides the main entry point for accessing the Ansible library, and all Ansible modules must, at minimum, import from basic.py::
Ansible provides a number of module utilities that provide helper functions that you can use when developing your own modules. The ``basic.py`` module utility provides the main entry point for accessing the Ansible library, and all Python Ansible modules must, at minimum, import ``AnsibleModule``::
from ansible.module_utils.basic import *
from ansible.module_utils.basic import AnsibleModule
The following is a list of module_utils files and a general description. The module utility source code lives in the `./lib/module_utils` directory under your main Ansible path - for more details on any specific module utility, please see the source code.
The following is a list of ``module_utils`` files and a general description. The module utility source code lives in the ``./lib/ansible/module_utils`` directory under your main Ansible path - for more details on any specific module utility, please see the source code.
- api.py - Adds shared support for generic API modules.
- azure_rm_common.py - Definitions and utilities for Microsoft Azure Resource Manager template deployments.

93
docs/docsite/rst/dev_guide/developing_modules.rst
View file

@ -1,110 +1,63 @@
.. _developing_modules:
Developing Modules
==================
.. contents:: Topics
.. _module_dev_welcome:
Welcome
```````
This section discusses how to develop, debug, review, and test modules.
Ansible modules are reusable, standalone scripts that can be used by the Ansible API,
or by the :command:`ansible` or :command:`ansible-playbook` programs. They
return information to ansible by printing a JSON string to stdout before
exiting. They take arguments in one of several ways which we'll go into
as we work through this tutorial.
See :ref:`all_modules` for a list of existing modules.
Modules can be written in any language and are found in the path specified
by :envvar:`ANSIBLE_LIBRARY` or the ``--module-path`` command line option or
in the :envvar:`library section of the Ansible configuration file <ANSIBLE_LIBRARY>`.
.. _module_dev_should_you:
Should You Develop A Module?
````````````````````````````
Before diving into the work of creating a new module, you should think about whether you actually *should*
develop a module. Ask the following questions:
****************************
Should you develop a module?
****************************
Developing Ansible modules is easy, but often it isn't necessary. Before you start writing a new module, ask:
1. Does a similar module already exist?
There are a lot of existing modules available, you should check out the list of existing modules at :ref:`modules`
An existing module may cover the functionality you want. Ansible Core includes thousands of modules. Search our :ref:`list of existing modules <all_modules>` to see if there's a module that does what you need.
2. Has someone already worked on a similar Pull Request?
2. Does a Pull Request already exist?
It's possible that someone has already started developing a similar PR. There are a few ways to find open module Pull Requests:
An existing Pull Request may cover the functionality you want. If someone else has already started developing a similar module, you can review and test it. There are a few ways to find open module Pull Requests:
* `GitHub new module PRs <https://github.com/ansible/ansible/labels/new_module>`_
* `All updates to modules <https://github.com/ansible/ansible/labels/module>`_
* `New module PRs listed by directory <https://ansible.sivel.net/pr/byfile.html>`_ search for `lib/ansible/modules/`
If you find an existing PR that looks like it addresses the issue you are trying to solve, please provide feedback on the PR - this will speed up getting the PR merged.
If you find an existing PR that looks like it addresses your needs, please provide feedback on the PR. Community feedback speeds up the review and merge process.
3. Should you use or develop an action plugin instead?
Action plugins get run on the master instead of on the target. For modules like file/copy/template, some of the work needs to be done on the master before the module executes on the target. Action plugins execute first on the master and can then execute the normal module on the target if necessary.
For more information about action plugins, read the :ref:`action plugins documentation <developing_plugins>`.
An action plugin may be the best way to get the functionality you want. Action plugins run on the control node instead of on the managed node, and their functionality is available to all modules. For more information about developing plugins, read the :ref:`developing plugins page <developing_plugins>`.
4. Should you use a role instead?
Check out the :ref:`roles documentation<playbooks_reuse_roles>`.
A combination of existing modules may cover the functionality you want. You can write a role for this type of use case. Check out the :ref:`roles documentation<playbooks_reuse_roles>`.
5. Should you write multiple modules instead of one module?
The following guidelines will help you determine if your module attempts to do too much, and should likely be broken into several smaller modules.
The functionality you want may be too large for a single module. If you want to connect Ansible to a new cloud provider, database, or network platform, you may need to :ref:`develop a related group of modules<developing_modules_in_groups>`.
* Modules should have a concise and well defined functionality. Basically, follow the UNIX philosophy of doing one thing well.
* Modules should not require that a user know all the underlying options of an api/tool to be used. For instance, if the legal values for a required module parameter cannot be documented, that's a sign that the module would be rejected.
* Modules should not require that a user know all the underlying options of an API/tool to be used. For instance, if the legal values for a required module parameter cannot be documented, that's a sign that the module would be rejected.
* Modules should typically encompass much of the logic for interacting with a resource. A lightweight wrapper around an API that does not contain much logic would likely cause users to offload too much logic into a playbook, and for this reason the module would be rejected. Instead try creating multiple modules for interacting with smaller individual pieces of the API.
If your use case isn't covered by an existing module, an open PR, an action plugin, or a role, and you don't need to create multiple modules, then you're ready to start developing a new module. Choose from the topics below for next steps:
.. _developing_modules_all:
How To Develop A Module
```````````````````````
The following topics will discuss how to develop and work with modules:
:doc:`developing_program_flow_modules`
A description of Ansible's module architecture.
:doc:`developing_modules_general`
A general overview of how to develop, debug, and test modules.
:doc:`developing_modules_general_windows`
A general overview of how to develop, debug and test Windows modules.
:doc:`developing_modules_documenting`
How to include in-line documentation in your module.
:doc:`developing_modules_best_practices`
Best practices, recommendations, and things to avoid.
:doc:`developing_modules_checklist`
Checklist for contributing your module to Ansible.
:doc:`testing`
Developing unit and integration tests.
:ref:`developing_python_3`
Adding Python 3 support to modules (all new modules must be Python-2.6 and Python-3.5 compatible).
:doc:`developing_modules_in_groups`
A guide for partners wanting to submit multiple modules.
* I want to :ref:`get started on a new module <developing_modules_general>`.
* I want to review :ref:`tips and conventions for developing good modules <developing_modules_best_practices>`.
* I want to :ref:`write a Windows module <developing_modules_general_windows>`.
* I want :ref:`an overview of Ansible's architecture <developing_program_flow_modules>`.
* I want to :ref:`document my module <developing_modules_documenting>`.
* I want to :ref:`contribute my module back to Ansible Core <developing_modules_checklist>`.
* I want to :ref:`add unit and integration tests to my module <developing_testing>`.
* I want to :ref:`add Python 3 support to my module <developing_python_3>`.
* I want to :ref:`write multiple modules <developing_modules_in_groups>`.
.. seealso::
:ref:`all_modules`
Learn about available modules
:doc:`developing_plugins`
Learn about developing plugins
:doc:`developing_api`
Learn about the Python API for playbook and task execution
`GitHub modules directory <https://github.com/ansible/ansible/tree/devel/lib/ansible/modules>`_
Browse module source code
`Mailing List <https://groups.google.com/group/ansible-devel>`_
Development mailing list
`irc.freenode.net <http://irc.freenode.net>`_
#ansible IRC chat channel

262
docs/docsite/rst/dev_guide/developing_modules_best_practices.rst
View file

@ -1,197 +1,143 @@
.. _developing_modules_best_practices:
.. _module_dev_conventions:
Conventions, Best Practices, and Pitfalls
`````````````````````````````````````````
*******************************
Conventions, tips, and pitfalls
*******************************
As a reminder from the example code above, here are some basic conventions
and guidelines:
.. contents:: Topics
:local:
* If the module is addressing an object, the parameter for that object should be called 'name' whenever possible, or accept 'name' as an alias.
As you design and develop modules, follow these basic conventions and tips for clean, usable code:
* If you have a company module that returns facts specific to your installations, a good name for this module is `site_facts`.
Scoping your module(s)
======================
* Modules accepting boolean status should generally accept 'yes', 'no', 'true', 'false', or anything else a user may likely throw at them. The AnsibleModule common code supports this with "type='bool'".
Especially if you want to contribute your module(s) back to Ansible Core, make sure each module includes enough logic and functionality, but not too much. If you're finding these guidelines tricky, consider :ref:`whether you really need to write a module <module_dev_should_you>` at all.
* Include a minimum of dependencies if possible. If there are dependencies, document them at the top of the module file, and have the module raise JSON error messages when the import fails.
* Each module should have a concise and well-defined functionality. Basically, follow the UNIX philosophy of doing one thing well.
* Do not add ``list`` or ``info`` state options to an existing module - create a new ``_facts`` module.
* Modules should not require that a user know all the underlying options of an API/tool to be used. For instance, if the legal values for a required module parameter cannot be documented, the module does not belong in Ansible Core.
* Modules should encompass much of the logic for interacting with a resource. A lightweight wrapper around a complex API forces users to offload too much logic into their playbooks. If you want to connect Ansible to a complex API, :ref:`create multiple modules <developing_modules_in_groups>` that interact with smaller individual pieces of the API.
* Avoid creating a module that does the work of other modules; this leads to code duplication and divergence, and makes things less uniform, unpredictable and harder to maintain. Modules should be the building blocks. If you are asking 'how can I have a module execute other modules' ... you want to write a role.
* Modules must be self-contained in one file to be auto-transferred by ansible.
Designing module interfaces
===========================
* If packaging modules in an RPM, they only need to be installed on the control machine and should be dropped into /usr/share/ansible. This is entirely optional and up to you.
* If your module is addressing an object, the parameter for that object should be called ``name`` whenever possible, or accept ``name`` as an alias.
* Modules accepting boolean status should accept ``yes``, ``no``, ``true``, ``false``, or anything else a user may likely throw at them. The AnsibleModule common code supports this with ``type='bool'``.
* Avoid ``action``/``command``, they are imperative and not declarative, there are other ways to express the same thing.
* Modules must output valid JSON only. The top level return type must be a hash (dictionary) although they can be nested. Lists or simple scalar values are not supported, though they can be trivially contained inside a dictionary.
General guidelines & tips
=========================
* In the event of failure, a key of 'failed' should be included, along with a string explanation in 'msg'. Modules that raise tracebacks (stacktraces) are generally considered 'poor' modules, though Ansible can deal with these returns and will automatically convert anything unparseable into a failed result. If you are using the AnsibleModule common Python code, the 'failed' element will be included for you automatically when you call 'fail_json'.
* Each module should be self-contained in one file, so it can be be auto-transferred by Ansible.
* Always use the ``hacking/test-module`` script when developing modules - it will warn you about common pitfalls.
* If you have a local module that returns facts specific to your installations, a good name for this module is ``site_facts``.
* Eliminate or minimize dependencies. If your module has dependencies, document them at the top of the module file and raise JSON error messages when dependency import fails.
* Don't write to files directly; use a temporary file and then use the ``atomic_move`` function from ``ansible.module_utils.basic`` to move the updated temporary file into place. This prevents data corruption and ensures that the correct context for the file is kept.
* Avoid creating caches. Ansible is designed without a central server or authority, so you cannot guarantee it will not run with different permissions, options or locations. If you need a central authority, have it on top of Ansible (for example, using bastion/cm/ci server or tower); do not try to build it into modules.
* If you package your module(s) in an RPM, install the modules on the control machine in ``/usr/share/ansible``. Packaging modules in RPMs is optional.
* Return codes from modules are used if 'failed' is missing, 0=success and non-zero=failure.
Python tips
===========
* As results from many hosts will be aggregated at once, modules should return only relevant output. Returning the entire contents of a log file is generally bad form.
* When fetching URLs, use ``fetch_url`` or ``open_url`` from ``ansible.module_utils.urls``. Do not use ``urllib2``, which does not natively verify TLS certificates and so is insecure for https.
* Include a ``main`` function that wraps the normal execution.
* Call your :func:`main` from a conditional so you can import it into unit tests - for example:
* Modules should have a concise and well defined functionality. Basically, follow the UNIX philosophy of doing one thing well.
.. code-block:: python
* Modules should not require that a user know all the underlying options of an api/tool to be used. For instance, if the legal values for a required module parameter cannot be documented, that's a sign that the module would be rejected.
if __name__ == '__main__':
main()
* Modules should typically encompass much of the logic for interacting with a resource. A lightweight wrapper around an API that does not contain much logic would likely cause users to offload too much logic into a playbook, and for this reason the module would be rejected. Instead try creating multiple modules for interacting with smaller individual pieces of the API.
.. _shared_code:
.. _debugging_ansiblemodule_based_modules:
Importing and using shared code
===============================
Debugging AnsibleModule-based modules
`````````````````````````````````````
* Use shared code whenever possible - don't reinvent the wheel. Ansible offers the ``AnsibleModule`` common Python code, plus :ref:`utilities <appendix_module_utilities>` for many common use cases and patterns.
* Import ``ansible.module_utils`` code in the same place as you import other libraries.
* Do NOT use wildcards (*) for importing other python modules; instead, list the function(s) you are importing (for example, ``from some.other_python_module.basic import otherFunction``).
* Import custom packages in ``try``/``except`` and handle them with ``fail_json()`` in ``main()``. For example:
.. tip::
.. code-block:: python
If you're using the :file:`hacking/test-module` script then most of this
is taken care of for you. If you need to do some debugging of the module
on the remote machine that the module will actually run on or when the
module is used in a playbook then you may need to use this information
instead of relying on test-module.
try:
import foo
HAS_LIB=True
except:
HAS_LIB=False
Starting with Ansible-2.1.0, AnsibleModule-based modules are put together as
a zip file consisting of the module file and the various python module
boilerplate inside of a wrapper script instead of as a single file with all of
the code concatenated together. Without some help, this can be harder to
debug as the file needs to be extracted from the wrapper in order to see
what's actually going on in the module. Luckily the wrapper script provides
some helper methods to do just that.
Then in main(), just after the argspec, do
If you are using Ansible with the :envvar:`ANSIBLE_KEEP_REMOTE_FILES`
environment variables to keep the remote module file, here's a sample of how
your debugging session will start:
.. code-block:: python
.. code-block:: shell-session
if not HAS_LIB:
module.fail_json(msg='The foo Python module is required')
$ ANSIBLE_KEEP_REMOTE_FILES=1 ansible localhost -m ping -a 'data=debugging_session' -vvv
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: badger
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595 `" )'
<127.0.0.1> PUT /var/tmp/tmpjdbJ1w TO /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping
<127.0.0.1> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping'
localhost | SUCCESS => {
"changed": false,
"invocation": {
"module_args": {
"data": "debugging_session"
},
"module_name": "ping"
},
"ping": "debugging_session"
}
And document the dependency in the ``requirements`` section of your module's :ref:`documentation_block`.
Setting :envvar:`ANSIBLE_KEEP_REMOTE_FILES` to ``1`` tells Ansible to keep the
remote module files instead of deleting them after the module finishes
executing. Giving Ansible the ``-vvv`` option makes Ansible more verbose.
That way it prints the file name of the temporary module file for you to see.
.. _module_failures:
If you want to examine the wrapper file you can. It will show a small python
script with a large, base64 encoded string. The string contains the module
that is going to be executed. Run the wrapper's explode command to turn the
string into some python files that you can work with:
Handling module failures
========================
.. code-block:: shell-session
When your module fails, help users understand what went wrong. If you are using the ``AnsibleModule`` common Python code, the ``failed`` element will be included for you automatically when you call ``fail_json``. For polite module failure behavior:
$ python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping explode
Module expanded into:
/home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/debug_dir
* Include a key of ``failed`` along with a string explanation in ``msg``. If you don't do this, Ansible will use standard return codes: 0=success and non-zero=failure.
* Don't raise a traceback (stacktrace). Ansible can deal with stacktraces and automatically converts anything unparseable into a failed result, but raising a stacktrace on module failure is not user-friendly.
* Do not use ``sys.exit()``. Use ``fail_json()`` from the module object.
When you look into the debug_dir you'll see a directory structure like this::
Handling exceptions (bugs) gracefully
=====================================
├── ansible_module_ping.py
├── args
└── ansible
├── __init__.py
└── module_utils
├── basic.py
└── __init__.py
* Validate upfront--fail fast and return useful and clear error messages.
* Use defensive programming--use a simple design for your module, handle errors gracefully, and avoid direct stacktraces.
* Fail predictably--if we must fail, do it in a way that is the most expected. Either mimic the underlying tool or the general way the system works.
* Give out a useful message on what you were doing and add exception messages to that.
* Avoid catchall exceptions, they are not very useful unless the underlying API gives very good error messages pertaining the attempted action.
* :file:`ansible_module_ping.py` is the code for the module itself. The name
is based on the name of the module with a prefix so that we don't clash with
any other python module names. You can modify this code to see what effect
it would have on your module.
.. _module_output:
* The :file:`args` file contains a JSON string. The string is a dictionary
containing the module arguments and other variables that Ansible passes into
the module to change its behaviour. If you want to modify the parameters
that are passed to the module, this is the file to do it in.
Creating correct and informative module output
==============================================
* The :file:`ansible` directory contains code from
:mod:`ansible.module_utils` that is used by the module. Ansible includes
files for any :`module:`ansible.module_utils` imports in the module but not
any files from any other module. So if your module uses
:mod:`ansible.module_utils.url` Ansible will include it for you, but if
your module includes :mod:`requests` then you'll have to make sure that
the python requests library is installed on the system before running the
module. You can modify files in this directory if you suspect that the
module is having a problem in some of this boilerplate code rather than in
the module code you have written.
Modules must output valid JSON only. Follow these guidelines for creating correct, useful module output:
Once you edit the code or arguments in the exploded tree you need some way to
run it. There's a separate wrapper subcommand for this:
* Make your top-level return type a hash (dictionary).
* Nest complex return values within the top-level hash.
* Incorporate any lists or simple scalar values within the top-level return hash.
* Do not send module output to standard error, because the system will merge standard out with standard error and prevent the JSON from parsing.
* Capture standard error and return it as a variable in the JSON on standard out. This is how the command module is implemented.
* Never do ``print("some status message")`` in a module, because it will not produce valid JSON output.
* Always return useful data, even when there is no change.
* Be consistent about returns (some modules are too random), unless it is detrimental to the state/action.
* Make returns reusable--most of the time you don't want to read it, but you do want to process it and re-purpose it.
* Return diff if in diff mode. This is not required for all modules, as it won't make sense for certain ones, but please include it when applicable.
* Enable your return values to be serialized as JSON with Python's standard `JSON encoder and decoder <https://docs.python.org/3/library/json.html>`_ library. Basic python types (strings, int, dicts, lists, etc) are serializable.
* Do not return an object via exit_json(). Instead, convert the fields you need from the object into the fields of a dictionary and return the dictionary.
* Results from many hosts will be aggregated at once, so your module should return only relevant output. Returning the entire contents of a log file is generally bad form.
.. code-block:: shell-session
If a module returns stderr or otherwise fails to produce valid JSON, the actual output will still be shown in Ansible, but the command will not succeed.
$ python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping execute
{"invocation": {"module_args": {"data": "debugging_session"}}, "changed": false, "ping": "debugging_session"}
.. _module_conventions:
This subcommand takes care of setting the PYTHONPATH to use the exploded
:file:`debug_dir/ansible/module_utils` directory and invoking the script using
the arguments in the :file:`args` file. You can continue to run it like this
until you understand the problem. Then you can copy it back into your real
module file and test that the real module works via :command:`ansible` or
:command:`ansible-playbook`.
Following Ansible conventions
=============================
.. note::
Ansible conventions offer a predictable user interface across all modules, playbooks, and roles. To follow Ansible conventions in your module development:
The wrapper provides one more subcommand, ``excommunicate``. This
subcommand is very similar to ``execute`` in that it invokes the exploded
module on the arguments in the :file:`args`. The way it does this is
different, however. ``excommunicate`` imports the :func:`main`
function from the module and then calls that. This makes excommunicate
execute the module in the wrapper's process. This may be useful for
running the module under some graphical debuggers but it is very different
from the way the module is executed by Ansible itself. Some modules may
not work with ``excommunicate`` or may behave differently than when used
with Ansible normally. Those are not bugs in the module; they're
limitations of ``excommunicate``. Use at your own risk.
Module Paths
````````````
If you are having trouble getting your module "found" by ansible, be
sure it is in the :envvar:`ANSIBLE_LIBRARY` environment variable.
If you have a fork of one of the ansible module projects, do something like this::
ANSIBLE_LIBRARY=~/ansible-modules-core
And this will make the items in your fork be loaded ahead of what ships with Ansible. Just be sure
to make sure you're not reporting bugs on versions from your fork!
To be safe, if you're working on a variant on something in Ansible's normal distribution, it's not
a bad idea to give it a new name while you are working on it, to be sure you know you're pulling
your version.
Common Pitfalls
```````````````
You should never do this in a module:
.. code-block:: python
print("some status message")
Because the output is supposed to be valid JSON.
Modules must not output anything on standard error, because the system will merge
standard out with standard error and prevent the JSON from parsing. Capturing standard
error and returning it as a variable in the JSON on standard out is fine, and is, in fact,
how the command module is implemented.
If a module returns stderr or otherwise fails to produce valid JSON, the actual output
will still be shown in Ansible, but the command will not succeed.
Don't write to files directly; use a temporary file and then use the `atomic_move` function from `ansible.module_utils.basic` to move the updated temporary file into place. This prevents data corruption and ensures that the correct context for the file is kept.
Avoid creating a module that does the work of other modules; this leads to code duplication and divergence, and makes things less uniform, unpredictable and harder to maintain. Modules should be the building blocks. Instead of creating a module that does the work of other modules, use Plays and Roles instead.
Avoid creating 'caches'. Ansible is designed without a central server or authority, so you cannot guarantee it will not run with different permissions, options or locations. If you need a central authority, have it on top of Ansible (for example, using bastion/cm/ci server or tower); do not try to build it into modules.
Always use the hacking/test-module script when developing modules and it will warn you about these kind of things.
* Use consistent names across modules (yes, we have many legacy deviations - don't make the problem worse!).
* Use consistent parameters (arguments) within your module(s).
* Normalize parameters with other modules - if Ansible and the API your module connects to use different names for the same parameter, add aliases to your parameters so the user can choose which names to use in tasks and playbooks.
* Return facts from ``*_facts`` modules in the ``ansible_facts`` field of the :ref:`result dictionary<common_return_values>` so other modules can access them.
* Implement ``check_mode`` in all ``*_facts`` modules. Playbooks which conditionalize based on fact information will only conditionalize correctly in ``check_mode`` if the facts are returned in ``check_mode``. Usually you can add ``check_mode=True`` when instantiating ``AnsibleModule``.
* Use module-specific environment variables. For example, if you use the helpers in ``module_utils.api`` for basic authentication with ``module_utils.urls.fetch_url()`` and you fall back on environment variables for default values, use a module-specific environment variable like :code:`API_<MODULENAME>_USERNAME` to avoid conflict between modules.
* Keep module options simple and focused - if you're loading a lot of choices/states on an existing option, consider adding a new, simple option instead.
* Keep options small when possible. Passing a large data structure to an option might save us a few tasks, but it adds a complex requirement that we cannot easily validate before passing on to the module.
* If you want to pass complex data to an option, write an expert module that allows this, along with several smaller modules that provide a more 'atomic' operation against the underlying APIs and services. Complex operations require complex data. Let the user choose whether to reflect that complexity in tasks and plays or in vars files.
* Implement declarative operations (not CRUD) so the user can ignore existing state and focus on final state. For example, use ``started/stopped``, ``present/absent``.
* Strive for a consistent final state (aka idempotency). If running your module twice in a row against the same system would result in two different states, see if you can redesign or rewrite to achieve consistent final state. If you can't, document the behavior and the reasons for it.
* Provide consistent return values within the standard Ansible return structure, even if NA/None are used for keys normally returned under other options.
* Follow additional guidelines that apply to families of modules if applicable. For example, AWS modules should follow `the Amazon guidelines <https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/GUIDELINES.md>`_

178
docs/docsite/rst/dev_guide/developing_modules_checklist.rst
View file

@ -1,168 +1,36 @@
.. _developing_modules_checklist:
.. _module_contribution:
===================================
Contributing Your Module to Ansible
===================================
***********************************
Contributing your module to Ansible
***********************************
High-quality modules with minimal dependencies
can be included in Ansible, but modules (just due to the programming
preferences of the developers) will need to be implemented in Python and use
the AnsibleModule common code, and should generally use consistent arguments with the rest of
the program. Stop by the mailing list to inquire about requirements if you like, and submit
a github pull request to the `ansible <https://github.com/ansible/ansible>`_ project.
Included modules will ship with ansible, and also have a chance to be promoted to 'core' status, which
gives them slightly higher development priority (though they'll work in exactly the same way).
If you want to contribute a module to Ansible, you must meet our objective and subjective requirements. Modules accepted into the `main project repo <https://github.com/ansible/ansible>`_ ship with every Ansible installation. However, contributing to the main project isn't the only way to distribute a module - you can embed modules in roles on Galaxy or simply share copies of your module code for :ref:`local use <developing_locally>`.
.. formerly marked with _module_dev_testing:
Contributing to Ansible: objective requirements
===============================================
------------------------------
Contributing Modules Checklist
------------------------------
To contribute a module to Ansible, you must:
The following checklist items are important guidelines for people who want to contribute to the development of modules to Ansible on GitHub. Please read the guidelines before you submit your PR/proposal.
* write your module in either Python or Powershell for Windows
* use the ``AnsibleModule`` common code
* support Python 2.7 and Python 3.5 - if your module cannot support Python 2.7, explain the required minimum Python version and rationale in the requirements section in ``DOCUMENTATION``
* use proper :ref:`Python 3 syntax <developing_python_3>`
* follow `PEP 8 <https://www.python.org/dev/peps/pep-0008/>`_ Python style conventions - see :ref:`testing_pep8` for more information
* license your module with GPL 3
* conform to Ansible's :ref:`formatting and documentation <developing_modules_documenting>` standards
* include comprehensive :ref:`tests <developing_testing>` for your module
* minimize module dependencies
* support :ref:`check_mode <check_mode_dry>` if possible
* The shebang must always be ``#!/usr/bin/python``. This allows ``ansible_python_interpreter`` to work
* Modules must be written to support Python 2.6. If this is not possible, required minimum Python version and rationale should be explained in the requirements section in ``DOCUMENTATION``. In Ansible-2.3 the minimum requirement for modules was Python-2.4.
* Modules must be written to use proper Python-3 syntax. At some point in the future we'll come up with rules for running on Python-3 but we're not there yet. See :doc:`developing_python_3` for help on how to do this.
* Modules must have a metadata section. For the vast majority of new modules,
the metadata should look exactly like this:
Please make sure your module meets these requirements before you submit your PR/proposal. If you have questions, reach out via `Ansible's IRC chat channel <http://irc.freenode.net>`_ or the `Ansible development mailing list <https://groups.google.com/group/ansible-devel>`_.
.. code-block:: python
Contributing to Ansible: subjective requirements
================================================
ANSIBLE_METADATA = {'status': ['preview'],
'supported_by': 'community',
'metadata_version': '1.1'}
Read the complete :ref:`module metadata specification <ansible_metadata_block>` for more information.
* Documentation: Make sure it exists
* Module documentation should briefly and accurately define what each module and option does, and how it works with others in the underlying system. Documentation should be written for broad audience--readable both by experts and non-experts. This documentation is not meant to teach a total novice, but it also should not be reserved for the Illuminati (hard balance).
* Descriptions should always start with a capital letter and end with a full stop. Consistency always helps.
* The `required` setting is only required when true, otherwise it is assumed to be false.
* If `required` is false/missing, `default` may be specified (assumed 'null' if missing). Ensure that the default parameter in docs matches default parameter in code.
* Documenting `default` is not needed for `required: true`.
* Remove unnecessary doc like `aliases: []` or `choices: []`.
* Do not use Boolean values in a choice list . For example, in the list `choices: ['no', 'verify', 'always]`, 'no' will be interpreted as a Boolean value (you can check basic.py for BOOLEANS_* constants to see the full list of Boolean keywords). If your option actually is a boolean, just use `type=bool`; there is no need to populate 'choices'.
* For new modules or options in a module add version_added. The version should match the value of the current development version and is a string (not a float), so be sure to enclose it in quotes.
* Verify that arguments in doc and module spec dict are identical.
* For password / secret arguments no_log=True should be set.
* Requirements should be documented, using the `requirements=[]` field.
* Author should be set, with their name and their github id, at the least.
* Ensure that you make use of `U()` for URLs, `I()` for option names, `C()` for files and option values, `M()` for module names.
* If an optional parameter is sometimes required this need to be reflected in the documentation, e.g. "Required when C(state=present)."
* Verify that a GPL 3 License header is included.
* Does module use check_mode? Could it be modified to use it? Document it. Documentation is everyone's friend.
* Examples--include them whenever possible and make sure they are reproducible.
* Document the return structure of the module. Refer to :ref:`common_return_values` and :ref:`module_documenting` for additional information.
* Predictable user interface: This is a particularly important section as it is also an area where we need significant improvements.
* Name consistency across modules (we've gotten better at this, but we still have many deviations).
* Declarative operation (not CRUD)--this makes it easy for a user not to care what the existing state is, just about the final state. ``started/stopped``, ``present/absent``--don't overload options too much. It is preferable to add a new, simple option than to add choices/states that don't fit with existing ones.
* Keep options small, having them take large data structures might save us a few tasks, but adds a complex requirement that we cannot easily validate before passing on to the module.
* Allow an "expert mode". This may sound like the absolute opposite of the previous one, but it is always best to let expert users deal with complex data. This requires different modules in some cases, so that you end up having one (1) expert module and several 'piecemeal' ones (ec2_vpc_net?). The reason for this is not, as many users express, because it allows a single task and keeps plays small (which just moves the data complexity into vars files, leaving you with a slightly different structure in another YAML file). It does, however, allow for a more 'atomic' operation against the underlying APIs and services.
* Informative responses: Please note, that for >= 2.0, it is required that return data to be documented.
* Always return useful data, even when there is no change.
* Be consistent about returns (some modules are too random), unless it is detrimental to the state/action.
* Make returns reusable--most of the time you don't want to read it, but you do want to process it and re-purpose it.
* Return diff if in diff mode. This is not required for all modules, as it won't make sense for certain ones, but please attempt to include this when applicable).
* Code: This applies to all code in general, but often seems to be missing from modules, so please keep the following in mind as you work.
* Validate upfront--fail fast and return useful and clear error messages.
* Defensive programming--modules should be designed simply enough that this should be easy. Modules should always handle errors gracefully and avoid direct stacktraces. Ansible deals with this better in 2.0 and returns them in the results.
* Fail predictably--if we must fail, do it in a way that is the most expected. Either mimic the underlying tool or the general way the system works.
* Modules should not do the job of other modules, that is what roles are for. Less magic is more.
* Don't reinvent the wheel. Part of the problem is that code sharing is not that easy nor documented, we also need to expand our base functions to provide common patterns (retry, throttling, etc).
* Support check mode. This is not required for all modules, as it won't make sense for certain ones, but please attempt to include this when applicable). For more information, refer to :ref:`check_mode_drift` and :ref:`check_mode_dry`.
* Exceptions: The module must handle them. (exceptions are bugs)
* Give out useful messages on what you were doing and you can add the exception message to that.
* Avoid catchall exceptions, they are not very useful unless the underlying API gives very good error messages pertaining the attempted action.
* Module-dependent guidelines: Additional module guidelines may exist for certain families of modules.
* Be sure to check out the modules themselves for additional information.
* `Amazon <https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/GUIDELINES.md>`_
* Modules should make use of the "extends_documentation_fragment" to ensure documentation available. For example, the AWS module should include::
extends_documentation_fragment:
- aws
- ec2
* The module must not use sys.exit() --> use fail_json() from the module object.
* Import custom packages in try/except and handled with fail_json() in main() e.g.
.. code-block:: python
try:
import foo
HAS_LIB=True
except:
HAS_LIB=False
* The return structure should be consistent, even if NA/None are used for keys normally returned under other options.
* Are module actions idempotent? If not document in the descriptions or the notes.
* Import ``ansible.module_utils`` code in the same place as you import other libraries. In older code, this was done at the bottom of the file but that's no longer needed.
* Do not use wildcards for importing other python modules (ex: ``from ansible.module_utils.basic import *``). This used to be required for code imported from ``ansible.module_utils`` but, from Ansible-2.1 onwards, it's just an outdated and bad practice.
* The module must have a `main` function that wraps the normal execution.
* Call your :func:`main` from a conditional so that it would be possible to
import them into unit tests in the future example
.. code-block:: python
if __name__ == '__main__':
main()
* Try to normalize parameters with other modules, you can have aliases for when user is more familiar with underlying API name for the option
* Being `PEP 8 <https://www.python.org/dev/peps/pep-0008/>`_ compliant is a requirement. See :doc:`testing_pep8` for more information.
* Avoid '`action`/`command`', they are imperative and not declarative, there are other ways to express the same thing
* Do not add `list` or `info` state options to an existing module - create a new `_facts` module.
* If you are asking 'how can I have a module execute other modules' ... you want to write a role
* Return values must be able to be serialized as json via the python stdlib
json library. basic python types (strings, int, dicts, lists, etc) are
serializable. A common pitfall is to try returning an object via
exit_json(). Instead, convert the fields you need from the object into the
fields of a dictionary and return the dictionary.
* When fetching URLs, please use either fetch_url or open_url from ansible.module_utils.urls
rather than urllib2; urllib2 does not natively verify TLS certificates and so is insecure for https.
* facts modules must return facts in the ansible_facts field of the :ref:`result
dictionary<common_return_values>`.
* modules that are purely about fact gathering need to implement check_mode.
they should not cause any changes anyway so it should be as simple as adding
check_mode=True when instantiating AnsibleModule. (The reason is that
playbooks which conditionalize based on fact information will only
conditionalize correctly in check_mode if the facts are returned in
check_mode).
* Basic auth: module_utils.api has some helpers for doing basic auth with
module_utils.urls.fetch_url(). If you use those you may find you also want
to fallback on environment variables for default values. If you do that,
be sure to use non-generic environment variables (like
:code:`API_<MODULENAME>_USERNAME`). Using generic environment variables
like :code:`API_USERNAME` would conflict between modules.
If your module meets our objective requirements, we'll review your code to see if we think it's clear, concise, secure, and maintainable. We'll consider whether your module provides a good user experience, helpful error messages, reasonable defaults, and more. This process is subjective, and we can't list exact standards for acceptance. For the best chance of getting your module accepted into the Ansible repo, follow our :ref:`tips for module development <developing_modules_best_practices>`.
Windows modules checklist
=========================
For a checklist and details on how to write Windows modules please see :doc:`developing_modules_general_windows`
Deprecating and making module aliases
======================================
Starting in 1.8, you can deprecate modules by renaming them with a preceding ``_``, i.e. ``old_cloud.py`` to
``_old_cloud.py``. This keeps the module available, but hides it from the primary docs and listing.
When deprecating a module:
1) Set the ``ANSIBLE_METADATA`` `status` to `deprecated`.
2) In the ``DOCUMENTATION`` section, add a `deprecated` field along the lines of::
deprecated: Deprecated in 2.3. Use M(whatmoduletouseinstead) instead.
3) Add the deprecation to CHANGELOG.md under the ``###Deprecations:`` section.
Alias module names
------------------
You can also rename modules and keep an alias to the old name by using a symlink that starts with _.
This example allows the stat module to be called with fileinfo, making the following examples equivalent::
EXAMPLES = '''
ln -s stat.py _fileinfo.py
ansible -m stat -a "path=/tmp" localhost
ansible -m fileinfo -a "path=/tmp" localhost
'''
For a checklist and details on how to write Windows modules please see :ref:`developing_modules_general_windows`

444
docs/docsite/rst/dev_guide/developing_modules_documenting.rst
View file

@ -1,80 +1,63 @@
.. _developing_modules_documenting:
.. _module_documenting:
Documenting Your Module
=======================
*******************************
Module format and documentation
*******************************
.. contents:: Topics
If you want to contribute your module to Ansible, you must write your module in Python and follow the standard format described below. (Unless you're writing a Windows module, in which case the :ref:`Windows guidelines <developing_modules_general_windows>` apply.) In addition to following this format, you should review our :ref:`submission checklist <developing_modules_checklist>`, :ref:`programming tips <developing_modules_best_practices>`, and :ref:`strategy for maintaining Python 2 and Python 3 compatibility <developing_python_3>`, as well as information about :ref:`testing <developing_testing>` before you open a pull request.
The online module documentation is generated from the modules themselves.
As the module documentation is generated from documentation strings contained in the modules, all modules included with Ansible must have a ``DOCUMENTATION`` string.
This string must be a valid YAML document
which conforms to the schema defined below. You may find it easier to
start writing your ``DOCUMENTATION`` string in an editor with YAML
syntax highlighting before you include it in your Python file.
Every Ansible module written in Python must begin with seven standard sections in a particular order, followed by the code. The sections in order are:
All modules must have the following sections defined in this order:
1. Copyright
2. ANSIBLE_METADATA
3. DOCUMENTATION
4. EXAMPLES
5. RETURN
6. Python imports
.. contents::
:depth: 1
:local:
.. note:: Why don't the imports go first?
Keen Python programmers may notice that contrary to PEP 8's advice we don't put ``imports`` at the top of the file. This is because the ``ANSIBLE_METADATA`` through ``RETURN`` sections are not used by the module code itself; they are essentially extra docstrings for the file. The imports are placed after these special variables for the same reason as PEP 8 puts the imports after the introductory comments and docstrings. This keeps the active parts of the code together and the pieces which are purely informational apart. The decision to exclude E402 is based on readability (which is what PEP 8 is about). Documentation strings in a module are much more similar to module level docstrings, than code, and are never utilized by the module itself. Placing the imports below this documentation and closer to the code, consolidates and groups all related code in a congruent manner to improve readability, debugging and understanding.
.. warning:: Why do some modules have imports at the bottom of the file?
.. warning:: **Copy old modules with care!**
If you look at some existing older modules, you may find imports at the bottom of the file. Do not copy that idiom into new modules as it is a historical oddity due to how modules used to be combined with libraries. Over time we're moving the imports to be in their proper place.
Some older modules in Ansible Core have ``imports`` at the bottom of the file, ``Copyright`` notices with the full GPL prefix, and/or ``ANSIBLE_METADATA`` fields in the wrong order. These are legacy files that need updating - do not copy them into new modules. Over time we're updating and correcting older modules. Please follow the guidelines on this page!
.. _shebang:
Python shebang
==============
Every Ansible module must begin with ``#!/usr/bin/python`` - this "shebang" allows ``ansible_python_interpreter`` to work.
.. _copyright:
Copyright
----------------------
Copyright and license
=====================
The beginning of every module should look about the same. After the shebang,
there should be at least two lines covering copyright and licensing of the
code.
After the shebang, there should be a `copyright line <https://www.gnu.org/licenses/gpl-howto.en.html>`_ with the original copyright holder and a license declaration. The license declaration should be ONLY one line, not the full GPL prefix.:
.. code-block:: python
#!/usr/bin/python
# Copyright: (c) 2018, Terry Jones <terry.jones@example.org>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
Every file should have a copyright line (see `The copyright notice <https://www.gnu.org/licenses/gpl-howto.en.html>`_)
with the original copyright holder. Major additions to the module (for
instance, rewrites) may add additional copyright lines. Any legal questions
need to review the source control history, so an exhaustive copyright header is
not necessary.
The license declaration should be ONLY one line, not the full GPL prefix. If
you notice a module with the full prefix, feel free to switch it to the
one-line declaration instead.
When adding a copyright line after completing a significant feature or rewrite,
add the newer line above the older one, like so:
Major additions to the module (for instance, rewrites) may add additional copyright lines. Any legal review will include the source control history, so an exhaustive copyright header is not necessary. When adding a second copyright line for a significant feature or rewrite, add the newer line above the older one:
.. code-block:: python
#!/usr/bin/python
# Copyright: (c) 2017, [New Contributor(s)]
# Copyright: (c) 2015, [Original Contributor(s)]
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
.. _ansible_metadata_block:
ANSIBLE_METADATA Block
----------------------
ANSIBLE_METADATA block
======================
``ANSIBLE_METADATA`` contains information about the module for use by other tools. At the moment, it informs other tools which type of maintainer the module has and to what degree users can rely on a module's behaviour remaining the same over time.
For new modules, the following block can be simply added into your module
After the shebang, the copyright, and the license, your module file should contain an ``ANSIBLE_METADATA`` section. This section provides information about the module for use by other tools. For new modules, the following block can be simply added into your module:
.. code-block:: python
@ -87,30 +70,8 @@ For new modules, the following block can be simply added into your module
* ``metadata_version`` is the version of the ``ANSIBLE_METADATA`` schema, *not* the version of the module.
* Promoting a module's ``status`` or ``supported_by`` status should only be done by members of the Ansible Core Team.
.. note:: Pre-released metadata version
During development of Ansible-2.3, modules had an initial version of the
metadata. This version was modified slightly after release to fix some
points of confusion. You may occasionally see PRs for modules where the
ANSIBLE_METADATA doesn't look quite right because of this. Module
metadata should be fixed before checking it into the repository.
Version 1.1 of the metadata
+++++++++++++++++++++++++++
Structure
^^^^^^^^^
.. code-block:: python
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'supported_by': 'community',
'status': ['preview', 'deprecated']
}
Fields
^^^^^^
Ansible metadata fields
-----------------------
:metadata_version: An "X.Y" formatted string. X and Y are integers which
define the metadata format version. Modules shipped with Ansible are
@ -119,160 +80,188 @@ Fields
to an existing field. We'll increment X if we remove fields or values
or change the type or meaning of a field.
Current metadata_version is "1.1"
:supported_by: This field records who supports the module.
Default value is ``community``. Values are:
:supported_by: Who supports the module.
Default value is ``community``. For information on what the support level values entail, please see
:ref:`Modules Support <modules_support>`. Values are:
* core
* network
* certified
* community
* curated (Deprecated. Modules in this category should probably be core or
certified instead)
* curated (*deprecated value - modules in this category should be core or
certified instead*)
For information on what the support level values entail, please see
:ref:`Modules Support <modules_support>`.
:status: This field records information about the module that is
important to the end user. It's a list of strings. The default value
is a single element list ["preview"]. The following strings are valid
:status: List of strings describing how stable the module is likely to be. See also :ref:`module_lifecycle`.
The default value is a single element list ["preview"]. The following strings are valid
statuses and have the following meanings:
:stableinterface: This means that the module's parameters are
stable. Every effort will be made not to remove parameters or change
their meaning. It is not a rating of the module's code quality.
:preview: This module is a tech preview. This means it may be
:stableinterface: The module's parameters are stable. Every effort will be made not to remove parameters or change
their meaning. **Not** a rating of the module's code quality.
:preview: The module is in tech preview. It may be
unstable, the parameters may change, or it may require libraries or
web services that are themselves subject to incompatible changes.
:deprecated: This module is deprecated and will no longer be
available in a future release.
:removed: This module is not present in the release. A stub is
:deprecated: The module is deprecated and will be removed in a future release.
:removed: The module is not present in the release. A stub is
kept so that documentation can be built. The documentation helps
users port from the removed module to new modules.
Changes from Version 1.0
++++++++++++++++++++++++
.. _documentation_block:
:metadata_version: Version updated from 1.0 to 1.1
:supported_by: All substantive changes were to potential values of the supported_by field
DOCUMENTATION block
===================
* Added the certified value
* Deprecated the curated value, modules shipped with Ansible will use
certified instead. Third party modules are encouraged not to use this as
it is meaningless within Ansible proper.
* Added the network value
After the shebang, the copyright line, the license, and the ``ANSIBLE_METADATA`` section comes the ``DOCUMENTATION`` block. Ansible's online module documentation is generated from the ``DOCUMENTATION`` blocks in each module's source code. The ``DOCUMENTATION`` block must be valid YAML. You may find it easier to start writing your ``DOCUMENTATION`` string in an :ref:`editor with YAML syntax highlighting <other_tools_and_programs>` before you include it in your Python file. You can start by copying our `example documentation string <https://github.com/ansible/ansible/blob/devel/examples/DOCUMENTATION.yml>`_ into your module file and modifying it. If you run into syntax issues in your YAML, you can validate it on the `YAML Lint <http://www.yamllint.com/>`_ website.
DOCUMENTATION Block
-------------------
Module documentation should briefly and accurately define what each module and option does, and how it works with others in the underlying system. Documentation should be written for broad audience--readable both by experts and non-experts.
* Descriptions should always start with a capital letter and end with a full stop. Consistency always helps.
* Verify that arguments in doc and module spec dict are identical.
* For password / secret arguments no_log=True should be set.
* If an optional parameter is sometimes required, reflect this fact in the documentation, e.g. "Required when C(state=present)."
* If your module allows ``check_mode``, reflect this fact in the documentation.
See an example documentation string in the checkout under `examples/DOCUMENTATION.yml <https://github.com/ansible/ansible/blob/devel/examples/DOCUMENTATION.yml>`_.
Each documentation field is described below. Before committing your module documentation, please test it at the command line and as HTML:
Include it in your module file like this:
* As long as your module file is :ref:`available locally <local_modules>`, you can use ``ansible-doc -t module my_module_name`` to view your module documentation at the command line. Any parsing errors will be obvious - you can view details by adding ``-vvv`` to the command.
* You should also :ref:`test the HTML output <testing_documentation>` of your documentation.
.. code-block:: python
Documentation fields
--------------------
#!/usr/bin/python
# Copyright (c) 2017 [REPLACE THIS]
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
DOCUMENTATION = '''
---
module: modulename
short_description: This is a sentence describing the module
# ... snip ...
'''
The following fields can be used and are all required unless specified otherwise:
All fields in the ``DOCUMENTATION`` block are lower-case. All fields are required unless specified otherwise:
:module:
The name of the module. This must be the same as the filename, without the ``.py`` extension.
* The name of the module.
* Must be the same as the filename, without the ``.py`` extension.
:short_description:
* A short description which is displayed on the :ref:`all_modules` page and ``ansible-doc -l``.
* As the short description is displayed by ``ansible-doc -l`` without the category grouping it needs enough detail to explain its purpose without the context of the directory structure in which it lives.
* Unlike ``description:`` this field should not have a trailing full stop.
* The ``short_description`` is displayed by ``ansible-doc -l`` without any category grouping,
so it needs enough detail to explain the module's purpose without the context of the directory structure in which it lives.
* Unlike ``description:``, ``short_description`` should not have a trailing period/full stop.
:description:
* A detailed description (generally two or more sentences).
* Must be written in full sentences, i.e. with capital letters and fullstops.
* Shouldn't mention the name module.
* Must be written in full sentences, i.e. with capital letters and periods/full stops.
* Shouldn't mention the module name.
:version_added:
The version of Ansible when the module was added.
This is a `string`, and not a float, i.e. ``version_added: "2.1"``
* The version of Ansible when the module was added.
* This is a string, and not a float, i.e. ``version_added: "2.1"``
:author:
Name of the module author in the form ``First Last (@GitHubID)``. Use a multi-line list if there is more than one author.
* Name of the module author in the form ``First Last (@GitHubID)``.
* Use a multi-line list if there is more than one author.
:deprecated:
If a module is deprecated it must be:
* Mentioned in ``CHANGELOG``
* Referenced in the ``porting_guide_x.y.rst``
* File should be renamed to start with an ``_``
* ``ANSIBLE_METADATA`` must contain ``status: ['deprecated']``
* Following values must be set:
* Marks modules that will be removed in future releases. See also :ref:`module_lifecycle`.
:removed_in: A `string`, such as ``"2.9"``, which represents the version of Ansible this module will replaced with docs only module stub.
:why: Optional string that used to detail why this has been removed.
:alternative: Inform users they should do instead, i.e. ``Use M(whatmoduletouseinstead) instead.``.
:options:
One per module argument:
* If the module has no options (for example, it's a ``_facts`` module), all you need is one line: ``options: {}``.
* If your module has options (in other words, accepts arguments), each option should be documented thoroughly. For each module argument/option, include:
:option-name:
* Declarative operation (not CRUD)–this makes it easy for a user not to care what the existing state is, just about the final state, for example `online:`, rather than `is_online:`.
* Declarative operation (not CRUD), to focus on the final state, for example `online:`, rather than `is_online:`.
* The name of the option should be consistent with the rest of the module, as well as other modules in the same category.
:description:
* Detailed explanation of what this option does. It should be written in full sentences.
* Should not list the options values (that's what ``choices:`` is for, though it should explain `what` the values do if they aren't obvious.
* Should not list the possible values (that's what ``choices:`` is for, though it should explain `what` the values do if they aren't obvious).
* If an optional parameter is sometimes required this need to be reflected in the documentation, e.g. "Required when I(state=present)."
* Mutually exclusive options must be documented as the final sentence on each of the options.
:required:
Only needed if true, otherwise it is assumed to be false.
* Only needed if ``true``.
* If missing, we assume the option is not required.
:default:
* If `required` is false/missing, `default` may be specified (assumed 'null' if missing).
* If ``required`` is false/missing, ``default`` may be specified (assumed 'null' if missing).
* Ensure that the default parameter in the docs matches the default parameter in the code.
* The default option must not be listed as part of the description.
* If the option is a boolean value, you can use any of the boolean values recognized by Ansible:
(such as true/false or yes/no). Choose the one that reads better in the context of the option.
:choices:
List of option values. Should be absent if empty.
* List of option values.
* Should be absent if empty.
:type:
* Specifies the data type that option accepts, must match the ``argspec``.
* If an argument is ``type='bool'``, this field should be set to ``type: bool`` and no ``choices`` should be specified.
:aliases:
List of option name aliases; generally not needed.
:version_added:
Only needed if this option was extended after initial Ansible release, i.e. this is greater than the top level `version_added` field.
This is a string, and not a float, i.e. ``version_added: "2.3"``.
:suboptions:
If this option takes a dict, you can define it here. See `azure_rm_securitygroup`, `os_ironic_node` for examples.
:requirements:
List of requirements, and minimum versions (if applicable)
:notes:
Details of any important information that doesn't fit in one of the above sections; for example if ``check_mode`` isn't supported, or a link to external documentation.
:aliases:
* List of optional name aliases.
* Generally not needed.
:version_added:
* Only needed if this option was extended after initial Ansible release, i.e. this is greater than the top level `version_added` field.
* This is a string, and not a float, i.e. ``version_added: "2.3"``.
:suboptions:
* If this option takes a dict, you can define it here.
* See :ref:`azure_rm_securitygroup_module`, :ref:`os_ironic_node_module` for examples.
:requirements:
* List of requirements (if applicable).
* Include minimum versions.
:notes:
* Details of any important information that doesn't fit in one of the above sections.
* For example, whether ``check_mode`` is or is not supported, or links to external documentation.
Linking within module documentation
-----------------------------------
You can link from your module documentation to other module docs, other resources on docs.ansible.com, and resources elsewhere on the internet. The correct formats for these links are:
* ``L()`` for Links with a heading. For example: ``See L(IOS Platform Options guide, ../network/user_guide/platform_ios.html).``
* ``U()`` for URLs. For example: ``See U(https://www.ansible.com/products/tower) for an overview.``
* ``I()`` for option names. For example: ``Required if I(state=present).``
* ``C()`` for files and option values. For example: ``If not set the environment variable C(ACME_PASSWORD) will be used.``
* ``M()`` for module names. For example: ``See also M(win_copy) or M(win_template).``
.. note::
- The above fields are are all in lowercase.
To refer a collection of modules, use ``C(..)``, e.g. ``Refer to the C(win_*) modules.``
- If the module doesn't doesn't have any options (for example, it's a ``_facts`` module), you can use ``options: {}``.
Documentation fragments
-----------------------
If you're writing multiple related modules, they may share common documentation, such as authentication details or file mode settings. Rather than duplicate that information in each module's ``DOCUMENTATION`` block, you can save it once as a fragment and use it in each module's documentation. Shared documentation fragments are contained in a ``ModuleDocFragment`` class in `lib/ansible/utils/module_docs_fragments/ <https://github.com/ansible/ansible/tree/devel/lib/ansible/utils/module_docs_fragments>`_. To include a documentation fragment, add ``extends_documentation_fragment: FRAGMENT_NAME`` in your module's documentation.
For example, all AWS modules should include::
extends_documentation_fragment:
- aws
- ec2
You can find more examples by searching for ``extends_documentation_fragment`` under the Ansible source tree.
.. _examples_block:
EXAMPLES block
--------------
==============
The EXAMPLES section is required for all new modules.
After the shebang, the copyright line, the license, the ``ANSIBLE_METADATA`` section, and the ``DOCUMENTATION`` block comes the ``EXAMPLES`` block. Here you show users how your module works with real-world examples in multi-line plain-text YAML format. The best examples are ready for the user to copy and paste into a playbook. Review and update your examples with every change to your module.
Examples should demonstrate real world usage, and be written in multi-line plain-text YAML format.
Ensure that examples are kept in sync with the options during the PR review and any following code refactor.
As per playbook best practice, a `name:` should be specified.
``EXAMPLES`` string within the module like this::
Per playbook best practices, each example should include a ``name:`` line::
EXAMPLES = '''
- name: Ensure foo is installed
@ -281,18 +270,17 @@ As per playbook best practice, a `name:` should be specified.
state: present
'''
If the module returns facts that are often needed, an example of how to use them can be helpful.
If your module returns facts that are often needed, an example of how to use them can be helpful.
RETURN Block
------------
.. _return_block:
The RETURN section documents what the module returns, and is required for all new modules.
RETURN block
============
For each value returned, provide a ``description``, in what circumstances the value is ``returned``,
the ``type`` of the value and a ``sample``. For example, from the ``copy`` module:
After the shebang, the copyright line, the license, the ``ANSIBLE_METADATA`` section, ``DOCUMENTATION`` and ``EXAMPLES`` blocks comes the ``RETURN`` block. This section documents the information the module returns for use by other modules.
The following fields can be used and are all required unless specified otherwise.
If your module doesn't return anything (apart from the standard returns), this section of your module should read: ``RETURN = ''' # '''``
Otherwise, for each value returned, provide the following fields. All fields are required unless specified otherwise.
:return name:
Name of the returned field.
@ -300,37 +288,18 @@ The following fields can be used and are all required unless specified otherwise
:description:
Detailed description of what this value represents.
:returned:
When this value is returned, such as `always`, on `success`, `always`
When this value is returned, such as ``always``, or ``on success``.
:type:
Data type
Data type.
:sample:
One or more examples.
:version_added:
Only needed if this return was extended after initial Ansible release, i.e. this is greater than the top level `version_added` field.
This is a string, and not a float, i.e. ``version_added: "2.3"``.
:contains:
Optional, if you set `type: complex` you can detail the dictionary here by repeating the above elements.
:return name:
One per return
:description:
Detailed description of what this value represents.
:returned:
When this value is returned, such as `always`, on `success`, `always`
:type:
Data type
:sample:
One or more examples.
:version_added:
Only needed if this return was extended after initial Ansible release, i.e. this is greater than the top level `version_added` field.
This is a string, and not a float, i.e. ``version_added: "2.3"``.
For complex nested returns type can be specified as ``type: complex``.
Example::
Optional. To describe nested return values, set ``type: complex`` and repeat the elements above for each sub-field.
Here are two example ``RETURN`` sections, one with three simple fields and one with a complex nested field::
RETURN = '''
dest:
@ -373,102 +342,15 @@ Example::
constraint: ">= 3.0"
'''
.. note::
.. _python_imports:
If your module doesn't return anything (apart from the standard returns), you can use ``RETURN = ''' # '''``.
Python imports
==============
Python Imports
--------------
Starting with Ansible version 2.2, all new modules are required to use imports in the form:
After the shebang, the copyright line, the license, and the sections for ``ANSIBLE_METADATA``, ``DOCUMENTATION``, ``EXAMPLES``, and ``RETURN``, you can finally add the python imports. All modules must use Python imports in the form:
.. code-block:: python
from module_utils.basic import AnsibleModule
.. warning::
The use of "wildcard" imports such as ``from module_utils.basic import *`` is no longer allowed.
Formatting functions
--------------------
The formatting functions are:
* ``L()`` for Links with a heading
* ``U()`` for URLs
* ``I()`` for option names
* ``C()`` for files and option values
* ``M()`` for module names.
Module names should be specified as ``M(module)`` to create a link to the online documentation for that module.
Example usage::
Or if not set the environment variable C(ACME_PASSWORD) will be used.
...
Required if I(state=present)
...
Mutually exclusive with I(project_src) and I(files).
...
See also M(win_copy) or M(win_template).
...
Time zone names are from the L(tz database,https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)
See U(https://www.ansible.com/products/tower) for an overview.
...
See L(IOS Platform Options guide, ../network/user_guide/platform_ios.html)
.. note::
If you wish to refer a collection of modules, use ``C(..)``, e.g. ``Refer to the C(win_*) modules.``
Documentation fragments
-----------------------
Some categories of modules share common documentation, such as details on how to authenticate options, or file mode settings. Rather than duplicate that information it can be shared using ``docs_fragments``.
These shared fragments are similar to the standard documentation block used in a module, they are just contained in a ``ModuleDocFragment`` class.
All the existing ``docs_fragments`` can be found in ``lib/ansible/utils/module_docs_fragments/``.
To include, simply add in ``extends_documentation_fragment: FRAGMENT_NAME`` into your module.
Examples can be found by searching for ``extends_documentation_fragment`` under the Ansible source tree.
Testing documentation
---------------------
The simplest way to check if your documentation works is to use ``ansible-doc`` to view it. Any parsing errors will be apparent, and details can be obtained by adding ``-vvv``.
If you are going to submit the module for inclusion in the main Ansible repo you should make sure that it renders correctly as HTML.
Put your completed module file into the ``lib/ansible/modules/$CATEGORY/`` directory and then
run the command: ``make webdocs``. The new 'modules.html' file will be
built in the ``docs/docsite/_build/html/$MODULENAME_module.html`` directory.
In order to speed up the build process, you can limit the documentation build to
only include modules you specify, or no modules at all. To do this, run the command:
``MODULES=$MODULENAME make webdocs``. The ``MODULES`` environment variable
accepts a comma-separated list of module names. To skip building
documentation for all modules, specify a non-existent module name, for example:
``MODULES=none make webdocs``.
You may also build a single page of the entire docsite. From ``ansible/docs/docsite`` run ``make htmlsingle rst=[relative path to the .rst file]``, for example: ``make htmlsingle rst=dev_guide/developing_modules_documenting.rst``
To test your documentation against your ``argument_spec`` you can use ``validate-modules``. Note that this option isn't currently enabled in Shippable due to the time it takes to run.
.. code-block:: bash
# If you don't already, ensure you are using your local checkout
source hacking/env-setup
./test/sanity/validate-modules/validate-modules --arg-spec --warnings lib/ansible/modules/your/modules/
.. tip::
If you're having a problem with the syntax of your YAML you can
validate it on the `YAML Lint <http://www.yamllint.com/>`_ website.
For more information in testing, including how to add unit and integration tests, see :doc:`testing`.
The use of "wildcard" imports such as ``from module_utils.basic import *`` is no longer allowed.

139
docs/docsite/rst/dev_guide/developing_modules_general.rst
View file

@ -1,30 +1,23 @@
.. _developing_modules_general:
.. _module_dev_tutorial_sample:
Ansible Module Development Walkthrough
======================================
*******************************************
Ansible module development: getting started
*******************************************
A module is a reusable, standalone script that Ansible runs on your behalf, either locally or remotely. Modules interact with your local machine, an API, or a remote system to perform specific tasks like changing a database password or spinning up a cloud instance. Each module can be used by the Ansible API, or by the :command:`ansible` or :command:`ansible-playbook` programs. A module provides a defined interface, accepting arguments and returning information to Ansible by printing a JSON string to stdout before exiting. Ansible ships with thousands of modules, and you can easily write your own. If you're writing a module for local use, you can choose any programming language and follow your own rules. This tutorial illustrates how to get started developing an Ansible module in Python.
In this section, we will walk through developing, testing, and debugging an Ansible module.
What's covered in this section:
- `Environment setup <#environment-setup>`__
- `New module development <#new-module-development>`__
- `Local/direct module testing <#localdirect-module-testing>`__
- `Playbook module testing <#playbook-module-testing>`__
- `Debugging (local) <#debugging-local>`__
- `Debugging (remote) <#debugging-remote>`__
- `Unit testing <#unit-testing>`__
- Integration testing (coming soon)
- `Communication and development
support <#communication-and-development-support>`__
- `Credit <#credit>`__
.. contents:: Topics
:local:
.. _environment_setup:
Environment setup
=================
Prerequisites Via Apt (Ubuntu)
``````````````````````````````
Prerequisites via apt (Ubuntu)
------------------------------
Due to dependencies (for example ansible -> paramiko -> pynacl -> libffi):
.. code:: bash
@ -32,8 +25,9 @@ Due to dependencies (for example ansible -> paramiko -> pynacl -> libffi):
sudo apt update
sudo apt install build-essential libssl-dev libffi-dev python-dev
Common Environment setup
````````````````````````
Common environment setup
------------------------------
1. Clone the Ansible repository:
``$ git clone https://github.com/ansible/ansible.git``
2. Change directory into the repository root dir: ``$ cd ansible``
@ -52,21 +46,23 @@ Common Environment setup
``$ . venv/bin/activate && . hacking/env-setup``
New module development
======================
Starting a new module
=====================
If you are creating a new module that doesn't exist, you would start
working on a whole new file. Here is an example:
To create a new module:
- Navigate to the directory that you want to develop your new module
in. E.g. ``$ cd lib/ansible/modules/cloud/azure/``
- Create your new module file: ``$ touch my_new_test_module.py``
- Paste this example code into the new module file: (explanation in comments)
1. Navigate to the correct directory for your new module: ``$ cd lib/ansible/modules/cloud/azure/``
2. Create your new module file: ``$ touch my_new_test_module.py``
3. Paste the content below into your new module file. It includes the :ref:`required Ansible format and documentation <developing_modules_documenting>` and some example code.
4. Modify and extend the code to do what you want your new module to do. See the :ref:`programming tips <developing_modules_best_practices>` and :ref:`Python 3 compatibility <developing_python_3>` pages for pointers on writing clean, concise module code.
.. code:: python
#!/usr/bin/python
# Copyright: (c) 2018, Terry Jones <terry.jones@example.org>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['preview'],
@ -130,8 +126,7 @@ working on a whole new file. Here is an example:
from ansible.module_utils.basic import AnsibleModule
def run_module():
# define the available arguments/parameters that a user can pass to
# the module
# define available arguments/parameters a user can pass to the module
module_args = dict(
name=dict(type='str', required=True),
new=dict(type='bool', required=False, default=False)
@ -189,15 +184,19 @@ working on a whole new file. Here is an example:
if __name__ == '__main__':
main()
Local/direct module testing
Exercising your module code
===========================
You may want to test the module on the local machine without targeting a
remote host. This is a great way to quickly and easily debug a module
that can run locally.
Once you've modified the sample code above to do what you want, you can try out your module.
Our :ref:`debugging tips <debugging>` will help if you run into bugs as you exercise your module code.
- Create an arguments file in ``/tmp/args.json`` with the following
content: (explanation below)
Exercising module code locally
------------------------------
If you module does not need to target a remote host, you can quickly and easily exercise you code locally like this:
- Create an arguments file, a basic JSON config file that passes parameters to your module so you can run it. Name the arguments file ``/tmp/args.json`` and add the following content:
.. code:: json
@ -214,21 +213,17 @@ that can run locally.
- Run your test module locally and directly:
``$ python ./my_new_test_module.py /tmp/args.json``
This should be working output that resembles something like the
following:
This should return output something like this:
.. code:: json
{"changed": true, "state": {"original_message": "hello", "new_message": "goodbye"}, "invocation": {"module_args": {"name": "hello", "new": true}}}
The arguments file is just a basic json config file that you can
use to pass the module your parameters to run the module it
Playbook module testing
=======================
Exercising module code in a playbook
------------------------------------
If you want to test your new module, you can now consume it with an
Ansible playbook.
The next step in testing your new module is to consume it with an Ansible playbook.
- Create a playbook in any directory: ``$ touch testmod.yml``
- Add the following to the new playbook file::
@ -247,35 +242,26 @@ Ansible playbook.
- Run the playbook and analyze the output: ``$ ansible-playbook ./testmod.yml``
Debugging (local)
=================
Testing basics
====================
If you want to break into a module and step through with the debugger, locally running the module you can do:
These two examples will get you started with testing your module code. Please review our :ref:`testing <developing_testing>` section for more detailed
information, including instructions for :ref:`testing documentation <testing_documentation>`, adding :ref:`integration tests <testing_integration>`, and more.
- Set a breakpoint in the module: ``import pdb; pdb.set_trace()``
- Run the module on the local machine: ``$ python -m pdb ./my_new_test_module.py ./args.json``
Sanity tests
------------
Debugging (remote)
==================
You can run through Ansible's sanity checks in a container:
In the event you want to debug a module that is running on a remote target (i.e. not localhost), one way to do this is the following:
``$ ansible-test sanity -v --docker --python 2.7 MODULE_NAME``
- On your controller machine (running Ansible) set `ANSIBLE_KEEP_REMOTE_FILES=1` (this tells Ansible to retain the modules it sends to the remote machine instead of removing them)
- Run your playbook targetting the remote machine and specify ``-vvvv`` (the verbose output will show you many things, including the remote location that Ansible uses for the modules)
- Take note of the remote path Ansible used on the remote host
- SSH into the remote target after the completion of the playbook
- Navigate to the directory (most likely it is going to be your ansible remote user defined or implied from the playbook: ``~/.ansible/tmp/ansible-tmp-...``)
- Here you should see the module that you executed from your Ansible controller, but this is the zipped file that Ansible sent to the remote host. You can run this by specifying ``python my_test_module.py`` (not necessary)
- To debug, though, we will want to extract this zip out to the original module format: ``python my_test_module.py explode`` (Ansible will expand the module into ``./debug-dir``)
- Navigate to ``./debug-dir`` (notice that unzipping has caused the generation of ``ansible_module_my_test_module.py``)
- Modify or set a breakpoint in the unzipped module
- Ensure that the unzipped module is executable: ``$ chmod 755 ansible_module_my_test_module.py``
- Run the unzipped module directly passing the args file: ``$ ./ansible_module_my_test_module.py args`` (args is the file that contains the params that were originally passed. Good for repro and debugging)
Note that this example requires Docker to be installed and running. If you'd rather not use a
container for this, you can choose to use ``--tox`` instead of ``--docker``.
Unit testing
============
Unit tests
----------
Unit tests for modules will be appropriately located in ``./test/units/modules``. You must first setup your testing environment. In this example, we're using Python 3.5.
You can add unit tests for your module in ``./test/units/modules``. You must first setup your testing environment. In this example, we're using Python 3.5.
- Install the requirements (outside of your virtual environment): ``$ pip3 install -r ./test/runner/requirements/units.txt``
- To run all tests do the following: ``$ ansible-test units --python 3.5`` (you must run ``. hacking/env-setup`` prior to this)
@ -287,26 +273,23 @@ To run pytest against a single test module, you can do the following (provide th
``$ pytest -r a --cov=. --cov-report=html --fulltrace --color yes
test/units/modules/.../test/my_new_test_module.py``
Going Further
=============
Contributing back to Ansible
============================
If you would like to contribute to the main Ansible repository
by adding a new feature or fixing a bug, `create a fork <https://help.github.com/articles/fork-a-repo/>`_
of the Ansible repository and develop against a new feature
branch using the ``devel`` branch as a starting point.
When you you have a good working code change,
When you you have a good working code change, you can
submit a pull request to the Ansible repository by selecting
your feature branch as a source and the Ansible devel branch as
a target.
If you want to submit a new module to the upstream Ansible repo, be sure
to run through sanity checks first. For example:
``$ ansible-test sanity -v --docker --python 2.7 MODULE_NAME``
Note that this example requires docker to be installed and running. If you'd rather not use a
container for this, you can choose to use ``--tox`` instead of ``--docker``.
If you want to contribute your module back to the upstream Ansible repo,
review our :ref:`submission checklist <developing_modules_checklist>`, :ref:`programming tips <developing_modules_best_practices>`,
and :ref:`strategy for maintaining Python 2 and Python 3 compatibility <developing_python_3>`, as well as
information about :ref:`testing <developing_testing>` before you open a pull request.
The :ref:`Community Guide <ansible_community_guide>` covers how to open a pull request and what happens next.
Communication and development support

1572
docs/docsite/rst/dev_guide/developing_modules_general_OLD._rst
View file

File diff suppressed because it is too large Load diff

29
docs/docsite/rst/dev_guide/developing_modules_general_windows.rst
View file

@ -1,7 +1,8 @@
.. _developing_modules_general_windows:
Windows Ansible Module Development Walkthrough
==============================================
**********************************************
Windows Ansible module development walkthrough
**********************************************
In this section, we will walk through developing, testing, and debugging an
Ansible Windows module.
@ -39,7 +40,7 @@ VirtualBox documentation for installation instructions):
- Vagrant
- VirtualBox
Create a Windows Server in a VM
Create a Windows server in a VM
===============================
To create a single Windows Server 2016 instance, run the following:
@ -55,7 +56,7 @@ for the first time, the Windows VM will run through the sysprep process and
then create a HTTP and HTTPS WinRM listener automatically. Vagrant will finish
its process once the listeners are onlinem, after which the VM can be used by Ansible.
Create an Ansible Inventory
Create an Ansible inventory
===========================
The following Ansible inventory file can be used to connect to the newly
@ -121,7 +122,7 @@ hosts that are defined under the ``domain_children`` key. The host variable
only network adapter while ``vagrant_box`` is the box that will be used to
create the VM.
Provisioning the Environment
Provisioning the environment
============================
To provision the environment as is, run the following:
@ -206,7 +207,7 @@ into PowerShell as well as some Ansible-specific requirements specified by
but are most commonly near the top. They are used to make it easier to state the
requirements of the module without writing any of the checks. Each ``requires``
statement must be on its own line, but there can be multiple requires statements
in one script.
in one script.
These are the checks that can be used within Ansible modules:
@ -228,7 +229,7 @@ can be imported by adding the following line to a PowerShell module:
#Requires -Module Ansible.ModuleUtils.Legacy
This will import the module_util at ``./lib/ansible/module_utils/powershell/Ansible.ModuleUtils.Legacy.psm1``
and enable calling all of its functions.
and enable calling all of its functions.
The following is a list of module_utils that are packaged with Ansible and a general description of what
they do:
@ -241,7 +242,7 @@ they do:
- LinkUtil: Utility to create, remove, and get information about symbolic links, junction points and hard inks.
- SID: Utilities used to convert a user or group to a Windows SID and vice versa.
For more details on any specific module utility and their requirements, please see the `Ansible
For more details on any specific module utility and their requirements, please see the `Ansible
module utilities source code <https://github.com/ansible/ansible/tree/devel/lib/ansible/module_utils/powershell>`_.
PowerShell module utilities can be stored outside of the standard Ansible
@ -360,7 +361,7 @@ Windows integration testing
Integration tests for Ansible modules are typically written as Ansible roles. These test
roles are located in ``./test/integration/targets``. You must first set up your testing
environment, and configure a test inventory for Ansible to connect to.
environment, and configure a test inventory for Ansible to connect to.
In this example we will set up a test inventory to connect to two hosts and run the integration
tests for win_stat:
@ -386,11 +387,11 @@ idempotent and does not report changes. For example:
state: absent
register: remove_file_check
check_mode: yes
- name: get result of remove a file (check mode)
win_command: powershell.exe "if (Test-Path -Path 'C:\temp') { 'true' } else { 'false' }"
register: remove_file_actual_check
- name: assert remove a file (check mode)
assert:
that:
@ -402,11 +403,11 @@ idempotent and does not report changes. For example:
path: C:\temp
state: absent
register: remove_file
- name: get result of remove a file
win_command: powershell.exe "if (Test-Path -Path 'C:\temp') { 'true' } else { 'false' }"
register: remove_file_actual
- name: assert remove a file
assert:
that:
@ -418,7 +419,7 @@ idempotent and does not report changes. For example:
path: C:\temp
state: absent
register: remove_file_again
- name: assert remove a file (idempotent)
assert:
that:

46
docs/docsite/rst/dev_guide/developing_modules_in_groups.rst
View file

@ -1,27 +1,31 @@
.. _developing_modules_in_groups:
*********************************************
Information for submitting a group of modules
=============================================
*********************************************
.. contents:: Topics
:local:
Submitting a group of modules
`````````````````````````````
=============================
This section discusses how to get multiple related modules into Ansible.
This document is intended for both companies wishing to add modules for their own products as well as users of 3rd party products wishing to add Ansible functionality.
It's based on module development best practices that the Ansible core team and community have accumulated.
It's based on module development tips and tricks that the Ansible core team and community have accumulated.
.. include:: shared_snippets/licensing.txt
Before you start coding
```````````````````````
=======================
Although it's tempting to get straight into coding, there are a few things to be aware of first. This list of prerequisites is designed to help ensure that you develop high-quality modules that flow easily through the review process and get into Ansible more quickly.
* Read though all the pages linked off :doc:`developing_modules`; paying particular focus to the :doc:`developing_modules_checklist`.
* For new modules going into Ansible 2.4 we are raising the bar so they must be PEP 8 compliant. See :doc:`testing_pep8` for more information.
* Starting with Ansible version 2.4, all new modules must support Python 2.6 and Python 3.5+. If this is an issue, please contact us (see the "Speak to us" section later in this document to learn how).
* All modules shipped with Ansible must be done so under the GPLv3 license. Files under the ``lib/ansible/module_utils/`` directory should be done so under the BSD license.
* New modules must be PEP 8 compliant. See :doc:`testing_pep8` for more information.
* Starting with Ansible version 2.7, all new modules must :ref:`support Python 2.7+ and Python 3.5+ <developing_python_3>`. If this is an issue, please contact us (see the "Speak to us" section later in this document to learn how).
* Have a look at the existing modules and how they've been named in the :ref:`all_modules`, especially in the same functional area (such as cloud, networking, databases).
* Shared code can be placed into ``lib/ansible/module_utils/``
* Shared documentation (for example describing common arguments) can be placed in ``lib/ansible/utils/module_docs_fragments/``.
@ -30,8 +34,8 @@ Although it's tempting to get straight into coding, there are a few things to be
* Starting with Ansible 2.4 all :ref:`network_modules` MUST have unit tests.
Naming Convention
`````````````````
Naming convention
=================
As you may have noticed when looking under ``lib/ansible/modules/`` we support up to two directories deep (but no deeper), e.g. `databases/mysql`. This is used to group files on disk as well as group related modules into categories and topics the Module Index, for example: :ref:`database_modules`.
@ -47,7 +51,7 @@ Each module should have the above (or similar) prefix; see existing :ref:`all_mo
Speak to us
```````````
===========
Circulating your ideas before coding is a good way to help you set off in the right direction.
@ -57,7 +61,7 @@ We've found that writing a list of your proposed module names and a one or two l
Where to get support
````````````````````
====================
Ansible has a thriving and knowledgeable community of module developers that is a great resource for getting your questions answered.
@ -68,8 +72,8 @@ In the :ref:`ansible_community_guide` you can find how to:
* IRC meetings - Join the various weekly IRC meetings `meeting schedule and agenda page <https://github.com/ansible/community/blob/master/meetings/README.md>`_
Your First Pull Request
````````````````````````
Your first pull request
=======================
Now that you've reviewed this document, you should be ready to open your first pull request.
@ -102,7 +106,7 @@ We have a ``ansibullbot`` helper that comments on GitHub Issues and PRs which sh
Subsequent PRs
``````````````
==============
By this point you first PR that defined the module namespace should have been merged. You can take the lessons learned from the first PR and apply it to the rest of the modules.
@ -116,10 +120,10 @@ Over the years we've experimented with different sized module PRs, ranging from
You can raise up to five PRs at once (5 PRs = 5 new modules) **after** your first PR has been merged. We've found this is a good batch size to keep the review process flowing.
Finally
```````
Maintaining your modules
========================
Now that your modules are integrated there are a few bits of housekeeping to be done
Now that your modules are integrated there are a few bits of housekeeping to be done.
**Bot Meta**
Update `Ansibullbot` so it knows who to notify if/when bugs or PRs are raised against your modules
@ -134,10 +138,10 @@ Review the autogenerated module documentation for each of your modules, found in
If the module documentation hasn't been published live yet, please let a member of the Ansible Core Team know in the ``#ansible-devel`` IRC channel.
New to Git or GitHub
````````````````````
New to git or GitHub
====================
We realise this may be your first use of Git or GitHub. The following guides may be of use:
We realize this may be your first use of Git or GitHub. The following guides may be of use:
* `How to create a fork of ansible/ansible <https://help.github.com/articles/fork-a-repo/>`_
* `How to sync (update) your fork <https://help.github.com/articles/syncing-a-fork/>`_
@ -147,4 +151,4 @@ Please note that in the Ansible Git Repo the main branch is called ``devel`` rat
After your first PR has been merged ensure you "sync your fork" with ``ansible/ansible`` to ensure you've pulled in the directory structure and and shared code or documentation previously created.
As stated in the GitHub documentation, always use feature branches for your PRs, never commit directly into `devel`.
As stated in the GitHub documentation, always use feature branches for your PRs, never commit directly into ``devel``.

171
docs/docsite/rst/dev_guide/developing_plugins.rst
View file

@ -1,23 +1,30 @@
.. _developing_plugins:
.. _plugin_guidelines:
Developing Plugins
==================
******************
Developing plugins
******************
.. contents:: Topics
Plugins are pieces of code that augment Ansible's core functionality. Ansible ships with a number of handy plugins, and you can easily write your own. This section describes the various types of Ansible plugins and how to implement them.
Plugins augment Ansible's core functionality with logic and features that are accessible to all modules. Ansible ships with a number of handy plugins, and you can easily write your own. All plugins must:
.. _plugin_guidelines:
* be written in Python
* raise errors
* return strings in unicode
* conform to Ansible's configuration and documentation standards
General Guidelines
------------------
Once you've reviewed these general guidelines, you can skip to the particular type of plugin you want to develop.
This section lists some things that should apply to any type of plugin you develop.
Writing plugins in Python
=========================
Raising Errors
^^^^^^^^^^^^^^
You must write your plugin in Python so it can be loaded by the ``PluginLoader`` and returned as a Python object that any module can use. Since your plugin will execute on the controller, you must write it in a :ref:`compatible version of Python <control_machine_requirements>`.
In general, errors encountered during execution should be returned by raising AnsibleError() or similar class with a message describing the error. When wrapping other exceptions into error messages, you should always use the ``to_text`` Ansible function to ensure proper string compatibility across Python versions:
Raising errors
==============
You should return errors encountered during plugin execution by raising ``AnsibleError()`` or a similar class with a message describing the error. When wrapping other exceptions into error messages, you should always use the ``to_text`` Ansible function to ensure proper string compatibility across Python versions:
.. code-block:: python
@ -28,40 +35,53 @@ In general, errors encountered during execution should be returned by raising An
except Exception as e:
raise AnsibleError('Something happened, this was original exception: %s' % to_native(e))
Check the different AnsibleError objects and see which one applies the best to your situation.
Check the different `AnsibleError objects <https://github.com/ansible/ansible/blob/devel/lib/ansible/errors/__init__.py>`_ and see which one applies best to your situation.
String Encoding
^^^^^^^^^^^^^^^
Any strings returned by your plugin that could ever contain non-ASCII characters must be converted into Python's unicode type because the strings will be run through jinja2. To do this, you can use:
String encoding
===============
You must convert any strings returned by your plugin into Python's unicode type. Converting to unicode ensures that these strings can run through Jinja2. To convert strings:
.. code-block:: python
from ansible.module_utils._text import to_text
result_string = to_text(result_string)
Plugin Configuration
^^^^^^^^^^^^^^^^^^^^
Plugin configuration & documentation standards
==============================================
Starting with Ansible version 2.4, we are unifying how each plugin type is configured and how they get those settings. Plugins will be able to declare their requirements and have Ansible provide them with a resolved'configuration. Starting with Ansible 2.4 both callback and connection type plugins can use this system.
To define configurable options for your plugin, describe them in the ``DOCUMENTATION`` section of the python file. Callback and connection plugins have declared configuration requirements this way since Ansible version 2.4; most plugin types now do the same. This approach ensures that the documentation of your plugin's options will always be correct and up-to-date. To add a configurable option to your plugin, define it in this format:
Most plugins will be able to use ``self.get_option(<optionname>)`` to access the settings.
These are pre-populated by a ``self.set_options()`` call, for most plugin types this is done by the controller,
but for some types you might need to do this explicitly.
Of course, if you don't have any configurable options, you can ignore this.
.. code-block:: yaml
Plugins that support embedded documentation (see :ref:`ansible-doc` for the list) must now include well-formed doc strings to be considered for merge into the Ansible repo. This documentation also doubles as 'configuration definition' so they will never be out of sync.
options:
option_name:
description: describe this config option
default: default value for this config option
env:
- name: NAME_OF_ENV_VAR
ini:
- section: section_of_ansible.cfg_where_this_config_option_is_defined
key: key_used_in_ansible.cfg
required: True/False
type: boolean/float/integer/list/none/path/pathlist/pathspec/string/tmppath
version_added: X.x
If you inherit from a plugin, you must document the options it takes, either via a documentation fragment or as a copy.
To access the configuration settings in your plugin, use ``self.get_option(<option_name>)``. For most plugin types, the controller pre-populates the settings. If you need to populate settings explicitly, use a ``self.set_options()`` call.
Plugins that support embedded documentation (see :ref:`ansible-doc` for the list) must include well-formed doc strings to be considered for merge into the Ansible repo. If you inherit from a plugin, you must document the options it takes, either via a documentation fragment or as a copy. See :ref:`module_documenting` for more information on correct documentation. Thorough documentation is a good idea even if you're developing a plugin for local use.
Developing particular plugin types
==================================
.. _developing_callbacks:
Callback Plugins
Callback plugins
----------------
Callback plugins enable adding new behaviors to Ansible when responding to events. By default, callback plugins control most of the output you see when running the command line programs.
Callback plugins add new behaviors to Ansible when responding to events. By default, callback plugins control most of the output you see when running the command line programs.
Callback plugins are created by creating a new class with the Base(Callbacks) class as the parent:
To create a callback plugin, create a new class with the Base(Callbacks) class as the parent:
.. code-block:: python
@ -75,7 +95,6 @@ For plugins intended for use with Ansible version 2.0 and later, you should only
For a complete list of methods that you can override, please see ``__init__.py`` in the
`lib/ansible/plugins/callback <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/callback>`_ directory.
The following is a modified example of how Ansible's timer plugin is implemented,
but with an extra option so you can see how configuration works in Ansible version 2.4 and later:
@ -144,41 +163,49 @@ but with an extra option so you can see how configuration works in Ansible versi
# Also note the use of the display object to print to screen. This is available to all callbacks, and you should use this over printing yourself
self._display.display(self._plugin_options['format_string'] % (self._days_hours_minutes_seconds(runtime)))
Note that the CALLBACK_VERSION and CALLBACK_NAME definitions are required for properly functioning plugins for Ansible version 2.0 and later. CALLBACK_TYPE is mostly needed to distinguish 'stdout' plugins from the rest, since you can only load one plugin that writes to stdout.
Note that the ``CALLBACK_VERSION`` and ``CALLBACK_NAME`` definitions are required for properly functioning plugins for Ansible version 2.0 and later. ``CALLBACK_TYPE`` is mostly needed to distinguish 'stdout' plugins from the rest, since you can only load one plugin that writes to stdout.
For example callback plugins, see the source code for the `callback plugins included with Ansible Core <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/callback>`_
.. _developing_connection_plugins:
Connection Plugins
Connection plugins
------------------
Connection plugins allow Ansible to connect to the target hosts so it can execute tasks on them. Ansible ships with many connection plugins, but only one can be used per host at a time.
Connection plugins allow Ansible to connect to the target hosts so it can execute tasks on them. Ansible ships with many connection plugins, but only one can be used per host at a time. The most commonly used connection plugins are the ``paramiko`` SSH, native ssh (just called ``ssh``), and ``local`` connection types. All of these can be used in playbooks and with ``/usr/bin/ansible`` to connect to remote machines.
By default, Ansible ships with several plugins. The most commonly used are the 'paramiko' SSH, native ssh (just called 'ssh'), and 'local' connection types. All of these can be used in playbooks and with /usr/bin/ansible to decide how you want to talk to remote machines.
Ansible version 2.1 introduced the ``smart`` connection plugin. The ``smart`` connection type allows Ansible to automatically select either the ``paramiko`` or ``openssh`` connection plugin based on system capabilities, or the ``ssh`` connection plugin if OpenSSH supports ControlPersist.
The basics of these connection types are covered in the :ref:`intro_getting_started` section.
To create a new connection plugin (for example, to support SNMP, Message bus, or other transports), copy the format of one of the existing connection plugins and drop it into the ``connection_plugins`` directory.
Should you want to extend Ansible to support other transports (SNMP, Message bus, etc) it's as simple as copying the format of one of the existing modules and dropping it into the connection plugins directory.
For example connection plugins, see the source code for the `connection plugins included with Ansible Core <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/connection>`_.
Ansible version 2.1 introduced the 'smart' connection plugin. The 'smart' connection type allows Ansible to automatically select either the 'paramiko' or 'openssh' connection plugin based on system capabilities, or the 'ssh' connection plugin if OpenSSH supports ControlPersist.
.. _developing_filter_plugins:
For examples on how to implement a connection plug in, see the source code here:
`lib/ansible/plugins/connection <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/connection>`_.
Filter plugins
--------------
Filter plugins manipulate data. They are a feature of Jinja2 and are also available in Jinja2 templates used by the ``template`` module. As with all plugins, they can be easily extended, but instead of having a file for each one you can have several per file. Most of the filter plugins shipped with Ansible reside in a ``core.py``.
Filter plugins do not use the standard configuration and documentation system described above.
For example filter plugins, see the source code for the `filter plugins included with Ansible Core <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/filter>`_.
.. _developing_inventory_plugins:
Inventory Plugins
Inventory plugins
-----------------
Inventory plugins were added in Ansible version 2.4. Inventory plugins parse inventory sources and form an in memory representation of the inventory.
Inventory plugins parse inventory sources and form an in-memory representation of the inventory. Inventory plugins were added in Ansible version 2.4.
You can see the details for inventory plugins in the :ref:`developing_inventory` page.
.. _developing_lookup_plugins:
Lookup Plugins
Lookup plugins
--------------
Lookup plugins are used to pull in data from external data stores. Lookup plugins can be used within playbooks both for looping --- playbook language constructs like ``with_fileglob`` and ``with_items`` are implemented via lookup plugins --- and to return values into a variable or parameter.
Lookup plugins pull in data from external data stores. Lookup plugins can be used within playbooks both for looping --- playbook language constructs like ``with_fileglob`` and ``with_items`` are implemented via lookup plugins --- and to return values into a variable or parameter.
Lookup plugins are very flexible, allowing you to retrieve and return any type of data. When writing lookup plugins, always return data of a consistent type that can be easily consumed in a playbook. Avoid parameters that change the returned data type. If there is a need to return a single value sometimes and a complex dictionary other times, write two different lookup plugins.
@ -261,13 +288,24 @@ The following is an example of how this lookup is called::
- debug:
msg: the value of foo.txt is {{ contents }} as seen today {{ lookup('pipe', 'date +"%Y-%m-%d"') }}
For more example lookup plugins, check out the source code for the lookup plugins that are included with Ansible here: `lib/ansible/plugins/lookup <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/lookup>`_.
For example lookup plugins, see the source code for the `lookup plugins included with Ansible Core <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/lookup>`_.
For more usage examples of lookup plugins, see :ref:`Using Lookups<playbooks_lookups>`.
.. _developing_test_plugins:
Test plugins
------------
Test plugins verify data. They are a feature of Jinja2 and are also available in Jinja2 templates used by the ``template`` module. As with all plugins, they can be easily extended, but instead of having a file for each one you can have several per file. Most of the test plugins shipped with Ansible reside in a ``core.py``. These are specially useful in conjunction with some filter plugins like ``map`` and ``select``; they are also available for conditional directives like ``when:``.
Test plugins do not use the standard configuration and documentation system described above.
For example test plugins, see the source code for the `test plugins included with Ansible Core <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/test>`_.
.. _developing_vars_plugins:
Vars Plugins
Vars plugins
------------
Vars plugins inject additional variable data into Ansible runs that did not come from an inventory source, playbook, or command line. Playbook constructs like 'host_vars' and 'group_vars' work using vars plugins.
@ -292,7 +330,7 @@ Most of the work now happens in the ``get_vars`` method which is called from th
The parameters are:
* loader: Ansible's DataLoader. The DataLoader can read files, auto load JSON/YAML and decrypt vaulted data, and cache read files.
* loader: Ansible's DataLoader. The DataLoader can read files, auto-load JSON/YAML and decrypt vaulted data, and cache read files.
* path: this is 'directory data' for every inventory source and the current play's playbook directory, so they can search for data in reference to them. ``get_vars`` will be called at least once per available path.
* entities: these are host or group names that are pertinent to the variables needed. The plugin will get called once for hosts and again for groups.
@ -300,51 +338,8 @@ This ``get vars`` method just needs to return a dictionary structure with the va
Since Ansible version 2.4, vars plugins only execute as needed when preparing to execute a task. This avoids the costly 'always execute' behavior that occurred during inventory construction in older versions of Ansible.
For implementation examples of vars plugins, check out the source code for the vars plugins that are included with Ansible:
`lib/ansible/plugins/vars <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/vars>`_ .
.. _developing_filter_plugins:
Filter Plugins
--------------
Filter plugins are used for manipulating data. They are a feature of Jinja2 and are also available in Jinja2 templates used by the ``template`` module. As with all plugins, they can be easily extended, but instead of having a file for each one you can have several per file. Most of the filter plugins shipped with Ansible reside in a ``core.py``.
See `lib/ansible/plugins/filter <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/filter>`_ for details.
.. _developing_test_plugins:
Test Plugins
------------
Test plugins are for verifying data. They are a feature of Jinja2 and are also available in Jinja2 templates used by the ``template`` module. As with all plugins, they can be easily extended, but instead of having a file for each one you can have several per file. Most of the test plugins shipped with Ansible reside in a ``core.py``. These are specially useful in conjunction with some filter plugins like ``map`` and ``select``; they are also available for conditional directives like ``when:``.
See `lib/ansible/plugins/test <https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/test>`_ for details.
.. _distributing_plugins:
Distributing Plugins
--------------------
Plugins are loaded from the library installed path and the configured plugins directory (check your `ansible.cfg`).
The location can vary depending on how you installed Ansible (pip, rpm, deb, etc) or by the OS/Distribution/Packager.
Plugins are automatically loaded when you have one of the following subfolders adjacent to your playbook or inside a role:
* action_plugins
* lookup_plugins
* callback_plugins
* connection_plugins
* inventory_plugins
* filter_plugins
* strategy_plugins
* cache_plugins
* test_plugins
* shell_plugins
* vars_plugins
When shipped as part of a role, the plugin will be available as soon as the role is called in the play.
For example vars plugins, see the source code for the `vars plugins included with Ansible Core
<https://github.com/ansible/ansible/tree/devel/lib/ansible/plugins/vars>`_.
.. seealso::

92
docs/docsite/rst/dev_guide/developing_program_flow_modules.rst
View file

@ -1,8 +1,9 @@
.. _flow_modules:
.. _developing_program_flow_modules:
===========================
Ansible Module Architecture
===========================
***************************
Ansible module architecture
***************************
This in-depth dive helps you understand Ansible's program flow to execute
modules. It is written for people working on the portions of the Core Ansible
@ -10,9 +11,12 @@ Engine that execute a module. Those writing Ansible Modules may also find this
in-depth dive to be of interest, but individuals simply using Ansible Modules
will not likely find this to be helpful.
.. contents:: Topics
:local:
.. _flow_types_of_modules:
Types of Modules
Types of modules
================
Ansible supports several different types of modules in its code base. Some of
@ -20,7 +24,7 @@ these are for backwards compatibility and others are to enable flexibility.
.. _flow_action_plugins:
Action Plugins
Action plugins
--------------
Action Plugins look like modules to end users who are writing :term:`playbooks` but
@ -42,7 +46,7 @@ into its final location, sets file permissions, and so on.
.. _flow_new_style_modules:
New-style Modules
New-style modules
-----------------
All of the modules that ship with Ansible fall into this category.
@ -55,7 +59,7 @@ connections instead of only one.
.. _flow_python_modules:
Python
^^^^^^
------
New-style Python modules use the :ref:`Ansiballz` framework for constructing
modules. All official modules (shipped with Ansible) use either this or the
@ -73,7 +77,7 @@ values as :term:`JSON`, and various file operations.
.. _flow_powershell_modules:
Powershell
^^^^^^^^^^
----------
New-style powershell modules use the :ref:`module_replacer` framework for
constructing modules. These modules get a library of powershell code embedded
@ -82,7 +86,7 @@ in them before being sent to the managed node.
.. _flow_jsonargs_modules:
JSONARGS
^^^^^^^^
--------
Scripts can arrange for an argument string to be placed within them by placing
the string ``<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>`` somewhere inside of the
@ -133,7 +137,7 @@ only modifies them to change a shebang line if present.
.. _flow_binary_modules:
Binary Modules
Binary modules
--------------
From Ansible 2.2 onwards, modules may also be small binary programs. Ansible
@ -152,7 +156,7 @@ way as :ref:`want JSON modules <flow_want_json_modules>`.
.. _flow_old_style_modules:
Old-style Modules
Old-style modules
-----------------
Old-style modules are similar to
@ -176,7 +180,7 @@ the remote machine.
.. _flow_executor_task_executor:
executor/task_executor
Executor/task_executor
----------------------
The TaskExecutor receives the module name and parameters that were parsed from
@ -189,10 +193,10 @@ to that Action Plugin for further processing.
.. _flow_normal_action_plugin:
Normal Action Plugin
Normal action plugin
--------------------
The ``normal`` Action Plugin executes the module on the remote host. It is
The ``normal`` action plugin executes the module on the remote host. It is
the primary coordinator of much of the work to actually execute the module on
the managed machine.
@ -223,7 +227,7 @@ which lives in :file:`plugins/action/__init__.py`. It makes use of
.. _flow_executor_module_common:
executor/module_common.py
Executor/module_common.py
-------------------------
Code in :file:`executor/module_common.py` takes care of assembling the module
@ -248,8 +252,8 @@ Next we'll go into some details of the two assembler frameworks.
.. _module_replacer:
Module Replacer
^^^^^^^^^^^^^^^
Module Replacer framework
^^^^^^^^^^^^^^^^^^^^^^^^^
The Module Replacer framework is the original framework implementing new-style
modules. It is essentially a preprocessor (like the C Preprocessor for those
@ -310,8 +314,8 @@ substitutions:
.. _Ansiballz:
Ansiballz
^^^^^^^^^
Ansiballz framework
^^^^^^^^^^^^^^^^^^^
Ansible 2.1 switched from the :ref:`module_replacer` framework to the
Ansiballz framework for assembling modules. The Ansiballz framework differs
@ -330,7 +334,7 @@ ansible module.
.. note::
Ansible wraps the zipfile in the Python script for two reasons:
* for compatibility with Python-2.6 which has a less
* for compatibility with Python 2.6 which has a less
functional version of Python's ``-m`` command line switch.
* so that pipelining will function properly. Pipelining needs to pipe the
Python module into the Python interpreter on the remote node. Python
@ -354,7 +358,7 @@ the zipfile as well.
.. _flow_passing_module_args:
Passing args
~~~~~~~~~~~~
------------
In :ref:`module_replacer`, module arguments are turned into a JSON-ified
string and substituted into the combined module file. In :ref:`Ansiballz`,
@ -379,7 +383,7 @@ other code.
.. _flow_internal_arguments:
Internal arguments
^^^^^^^^^^^^^^^^^^
------------------
Both :ref:`module_replacer` and :ref:`Ansiballz` send additional arguments to
the module beyond those which the user specified in the playbook. These
@ -389,7 +393,7 @@ the features are implemented in :py:mod:`ansible.module_utils.basic` but certain
features need support from the module so it's good to know about them.
_ansible_no_log
~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^
This is a boolean. If it's True then the playbook specified ``no_log`` (in
a task's parameters or as a play parameter). This automatically affects calls
@ -402,7 +406,7 @@ to instantiate an `AnsibleModule` and then check the value of
``no_log`` specified in a module's argument_spec are handled by a different mechanism.
_ansible_debug
~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^
This is a boolean that turns on more verbose logging. If a module uses
:py:meth:`AnsibleModule.debug` rather than :py:meth:`AnsibleModule.log` then
@ -414,7 +418,7 @@ should do so by instantiating an `AnsibleModule` and accessing
:attr:`AnsibleModule._debug`.
_ansible_diff
~~~~~~~~~~~~~
^^^^^^^^^^^^^^^
This boolean is turned on via the ``--diff`` command line option. If a module
supports it, it will tell the module to show a unified diff of changes to be
@ -423,13 +427,13 @@ instantiating an `AnsibleModule` and accessing
:attr:`AnsibleModule._diff`.
_ansible_verbosity
~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^
This value could be used for finer grained control over logging. However, it
is currently unused.
_ansible_selinux_special_fs
~~~~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is a list of names of filesystems which should have a special selinux
context. They are used by the `AnsibleModule` methods which operate on
@ -452,7 +456,7 @@ filesystem names. Under Ansiballz it's an actual list.
.. versionadded:: 2.1
_ansible_syslog_facility
~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^
This parameter controls which syslog facility ansible module logs to. It may
be set by changing the ``syslog_facility`` value in :file:`ansible.cfg`. Most
@ -477,7 +481,7 @@ than it did under :ref:`module_replacer` due to how hacky the old way was
.. versionadded:: 2.1
_ansible_version
~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^
This parameter passes the version of ansible that runs the module. To access
it, a module should instantiate an `AnsibleModule` and then retrieve it
@ -489,7 +493,7 @@ from :attr:`AnsibleModule.ansible_version`. This replaces
.. _flow_special_considerations:
Special Considerations
Special considerations
----------------------
.. _flow_pipelining:
@ -528,14 +532,14 @@ Passing arguments via stdin was chosen for the following reasons:
truncation of the parameters if we hit that limit.
.. _ansiblemodule:
.. _flow_ansiblemodule:
AnsibleModule
-------------
.. _argument_spec:
Argument Spec
Argument spec
^^^^^^^^^^^^^
The ``argument_spec`` provided to ``AnsibleModule`` defines the supported arguments for a module, as well as their type, defaults and more.
@ -556,10 +560,10 @@ Example ``argument_spec``:
)
))
This section will discss the behavioral attributes for arguments
This section will discuss the behavioral attributes for arguments:
type
~~~~
""""
``type`` allows you to define the type of the value accepted for the argument. The default value for ``type`` is ``str``. Possible values are:
@ -579,17 +583,17 @@ type
The ``raw`` type, performs no type validation or type casing, and maintains the type of the passed value.
elements
~~~~~~~~
""""""""
``elements`` works in combination with ``type`` when ``type='list'``. ``elements`` can then be defined as ``elements='int'`` or any other type, indicating that each element of the specified list should be of that type.
default
~~~~~~~
"""""""
The ``default`` option allows sets a default value for the argument for the scenario when the argument is not provided to the module. When not specified, the default value is ``None``.
fallback
~~~~~~~~
""""""""
``fallback`` accepts a ``tuple`` where the first argument is a callable (function) that will be used to perform the lookup, based on the second argument. The second argument is a list of values to be accepted by the callable.
@ -600,38 +604,38 @@ Example::
username=dict(fallback=(env_fallback, ['ANSIBLE_NET_USERNAME']))
choices
~~~~~~~
"""""""
``choices`` accepts a list of choices that the argument will accept. The types of ``choices`` should match the ``type``.
required
~~~~~~~~
""""""""
``required`` accepts a boolean, either ``True`` or ``False`` that indicates that the argument is required. This should not be used in combination with ``default``.
no_log
~~~~~~
""""""
``no_log`` indicates that the value of the argument should not be logged or displayed.
aliases
~~~~~~~
"""""""
``aliases`` accepts a list of alternative argument names for the argument, such as the case where the argument is ``name`` but the module accepts ``aliases=['pkg']`` to allow ``pkg`` to be interchangably with ``name``
options
~~~~~~~
"""""""
``options`` implements the ability to create a sub-argument_spec, where the sub options of the top level argument are also validated using the attributes discussed in this section. The example at the top of this section demonstrates use of ``options``. ``type`` or ``elements`` should be ``dict`` is this case.
apply_defaults
~~~~~~~~~~~~~~
""""""""""""""
``apply_defaults`` works alongside ``options`` and allows the ``default`` of the sub-options to be applied even when the top-level argument is not supplied.
In the example of the ``argument_spec`` at the top of this section, it would allow ``module.params['top_level']['second_level']`` to be defined, even if the user does not provide ``top_level`` when calling the module.
removed_in_version
~~~~~~~~~~~~~~~~~~
""""""""""""""""""
``removed_in_version`` indicates which version of Ansible a deprecated argument will be removed in.

357
docs/docsite/rst/dev_guide/developing_python_3.rst
View file

@ -1,106 +1,101 @@
.. _developing_python_3:
====================
********************
Ansible and Python 3
====================
********************
Ansible is pursuing a strategy of having one code base that runs on both
Python-2 and Python-3 because we want Ansible to be able to manage a wide
.. contents:: Topics
:local:
Ansible maintains a single code base that runs on both
Python 2 and Python 3 because we want Ansible to be able to manage a wide
variety of machines. Contributors to Ansible should be aware of the tips in
this document so that they can write code that will run on the same versions
of Python as the rest of Ansible.
Ansible can be divided into three overlapping pieces for the purposes of
porting:
To ensure that your code runs on Python 3 as well as on Python 2, learn the tips and tricks and idioms
described here. Most of these considerations apply to all three types of Ansible code:
1. Controller-side code. This is the code which runs on the machine where you
invoke :command:`/usr/bin/ansible`
2. Modules. This is the code which Ansible transmits over the wire and
invokes on the managed machine.
3. module_utils code. This is code whose primary purpose is to be used by the
modules to perform tasks. However, some controller-side code might use
generic functions from here.
1. controller-side code - code that runs on the machine where you invoke :command:`/usr/bin/ansible`
2. modules - the code which Ansible transmits to and invokes on the managed machine.
3. shared ``module_utils`` code - the common code that's used by modules to perform tasks and sometimes used by controller-side code as well
Much of the knowledge of porting code will be usable on all three of these
pieces but there are some special considerations for some of it as well.
Information that is generally applicable to all three places is located in the
controller-side section.
However, the three types of code do not use the same string strategy. If you're developing a module or some ``module_utils`` code, be sure
to read the section on string strategy carefully.
--------------------------------------------
Minimum Version of Python-3.x and Python-2.x
--------------------------------------------
Minimum version of Python 3.x and Python 2.x
============================================
On the controller we support Python-3.5 or greater and Python-2.7 or greater. Module-side, we
support Python-3.5 or greater and Python-2.6 or greater.
On the controller we support Python 3.5 or greater and Python 2.7 or greater. Module-side, we
support Python 3.5 or greater and Python 2.6 or greater.
Python-3.5 was chosen as a minimum because it is the earliest Python-3 version adopted as the
Python 3.5 was chosen as a minimum because it is the earliest Python 3 version adopted as the
default Python by a Long Term Support (LTS) Linux distribution (in this case, Ubuntu-16.04).
Previous LTS Linux distributions shipped with a Python-2 version which users can rely upon instead
of the Python-3 version.
Previous LTS Linux distributions shipped with a Python 2 version which users can rely upon instead
of the Python 3 version.
For Python 2, the default is for modules to run on at least Python-2.6. This allows
users with older distributions that are stuck on Python-2.6 to manage their
machines. Modules are allowed to drop support for Python-2.6 when one of
their dependent libraries requires a higher version of Python. This is not an
For Python 2, the default is for modules to run on at least Python 2.6. This allows
users with older distributions that are stuck on Python 2.6 to manage their
machines. Modules are allowed to drop support for Python 2.6 when one of
their dependent libraries requires a higher version of Python. This is not an
invitation to add unnecessary dependent libraries in order to force your
module to be usable only with a newer version of Python; instead it is an
acknowledgment that some libraries (for instance, boto3 and docker-py) will
only function with a newer version of Python.
.. note:: Python-2.4 Module-side Support:
.. note:: Python 2.4 Module-side Support:
Support for Python-2.4 and Python-2.5 was dropped in Ansible-2.4. RHEL-5
Support for Python 2.4 and Python 2.5 was dropped in Ansible-2.4. RHEL-5
(and its rebuilds like CentOS-5) were supported until April of 2017.
Ansible-2.3 was released in April of 2017 and was the last Ansible release
to support Python-2.4 on the module-side.
to support Python 2.4 on the module-side.
-----------------------------------
Porting Controller Code to Python 3
-----------------------------------
Developing Ansible code that supports Python 2 and Python 3
===========================================================
Most of the general tips for porting code to be used on both Python-2 and
Python-3 applies to porting controller code. The best place to start learning
to port code is `Lennart Regebro's book: Porting to Python 3 <http://python3porting.com/>`_.
The best place to start learning about writing code that supports both Python 2 and Python 3
is `Lennart Regebro's book: Porting to Python 3 <http://python3porting.com/>`_.
The book describes several strategies for porting to Python 3. The one we're
using is `to support Python 2 and Python 3 from a single code base
<http://python3porting.com/strategies.html#python 2-and-python 3-without-conversion>`_
The book describes several strategies for porting to Python 3. The one we're
using is `to support Python-2 and Python-3 from a single code base
<http://python3porting.com/strategies.html#python-2-and-python-3-without-conversion>`_
Understanding strings in Python 2 and Python 3
----------------------------------------------
Controller String Strategy
==========================
Background
----------
One of the most essential things to decide upon for porting code to Python-3
is what string model to use. Strings can be an array of bytes (like in C) or
Python 2 and Python 3 handle strings differently, so when you write code that supports Python 3
you must decide what string model to use. Strings can be an array of bytes (like in C) or
they can be an array of text. Text is what we think of as letters, digits,
numbers, other printable symbols, and a small number of unprintable "symbols"
(control codes).
In Python-2, the two types for these (:class:`str` for bytes and
In Python 2, the two types for these (:class:`str` for bytes and
:class:`unicode` for text) are often used interchangeably. When dealing only
with ASCII characters, the strings can be combined, compared, and converted
from one type to another automatically. When non-ASCII characters are
introduced, Python starts throwing exceptions due to not knowing what encoding
introduced, Python 2 starts throwing exceptions due to not knowing what encoding
the non-ASCII characters should be in.
Python-3 changes this behavior by making the separation between bytes (:class:`bytes`)
and text (:class:`str`) more strict. Python will throw an exception when
Python 3 changes this behavior by making the separation between bytes (:class:`bytes`)
and text (:class:`str`) more strict. Python 3 will throw an exception when
trying to combine and compare the two types. The programmer has to explicitly
convert from one type to the other to mix values from each.
This change makes it immediately apparent to the programmer when code is
mixing the types inappropriately, rather than working until one of their users
causes an exception by entering non-ASCII input. However, it forces the
programmer to proactively define a strategy for working with strings in their
program so that they don't mix text and byte strings unintentionally.
In Python 3 it's immediately apparent to the programmer when code is
mixing the byte and text types inappropriately, whereas in Python 2, code that mixes those types
may work until a user causes an exception by entering non-ASCII input.
Python 3 forces programmers to proactively define a strategy for
working with strings in their program so that they don't mix text and byte strings unintentionally.
Unicode Sandwich
----------------
Ansible uses different strategies for working with strings in controller-side code, in
:ref: `modules <module_string_strategy>`, and in :ref:`module_utils <module_utils_string_strategy>` code.
.. _controller_string_strategy:
Controller string strategy: the Unicode Sandwich
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In controller-side code we use a strategy known as the Unicode Sandwich (named
after Python-2's :class:`unicode` text type). For Unicode Sandwich we know that
after Python 2's :class:`unicode` text type). For Unicode Sandwich we know that
at the border of our code and the outside world (for example, file and network IO,
environment variables, and some library calls) we are going to receive bytes.
We need to transform these bytes into text and use that throughout the
@ -109,17 +104,18 @@ the outside world we first convert the text back into bytes.
To visualize this, imagine a 'sandwich' consisting of a top and bottom layer
of bytes, a layer of conversion between, and all text type in the center.
Common Borders
--------------
Unicode Sandwich common borders: places to convert bytes to text in controller code
-----------------------------------------------------------------------------------
This is a partial list of places where we have to convert to and from bytes.
It's not exhaustive but gives you an idea of where to watch for problems.
This is a partial list of places where we have to convert to and from bytes
when using the Unicode Sandwich string strategy. It's not exhaustive but
it gives you an idea of where to watch for problems.
Reading and writing to files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In Python-2, reading from files yields bytes. In Python-3, it can yield text.
To make code that's portable to both we don't make use of Python-3's ability
In Python 2, reading from files yields bytes. In Python 3, it can yield text.
To make code that's portable to both we don't make use of Python 3's ability
to yield text but instead do the conversion explicitly ourselves. For example:
.. code-block:: python
@ -153,14 +149,14 @@ Note that we don't have to catch :exc:`UnicodeError` here because we're
transforming to UTF-8 and all text strings in Python can be transformed back
to UTF-8.
Filesystem Interaction
~~~~~~~~~~~~~~~~~~~~~~
Filesystem interaction
^^^^^^^^^^^^^^^^^^^^^^
Dealing with filenames often involves dropping back to bytes because on UNIX-like
systems filenames are bytes. On Python-2, if we pass a text string to these
systems filenames are bytes. On Python 2, if we pass a text string to these
functions, the text string will be converted to a byte string inside of the
function and a traceback will occur if non-ASCII characters are present. In
Python-3, a traceback will only occur if the text string can't be decoded in
Python 3, a traceback will only occur if the text string can't be decoded in
the current locale, but it's still good to be explicit and have code which
works on both versions:
@ -198,30 +194,75 @@ and manipulate in bytes.
all the types are the same (either all bytes or all text). Mixing
bytes and text will cause tracebacks.
Interacting with Other Programs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Interacting with other programs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Interacting with other programs goes through the operating system and
C libraries and operates on things that the UNIX kernel defines. These
interfaces are all byte-oriented so the Python interface is byte oriented as
well. On both Python-2 and Python-3, byte strings should be given to Python's
well. On both Python 2 and Python 3, byte strings should be given to Python's
subprocess library and byte strings should be expected back from it.
One of the main places in Ansible's controller code that we interact with
other programs is the connection plugins' ``exec_command`` methods. These
methods transform any text strings they receive in the command (and arguments
to the command) to execute into bytes and return stdout and stderr as byte strings
to the command) to execute into bytes and return stdout and stderr as byte strings
Higher level functions (like action plugins' ``_low_level_execute_command``)
transform the output into text strings.
Tips, tricks, and idioms to adopt
=================================
.. _module_string_strategy:
Forwards Compatibility Boilerplate
----------------------------------
Module string strategy: Native String
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Use the following boilerplate code at the top of all controller-side modules
to make certain constructs act the same way on Python-2 and Python-3:
In modules we use a strategy known as Native Strings. This makes things
easier on the community members who maintain so many of Ansible's
modules, by not breaking backwards compatibility by
mandating that all strings inside of modules are text and converting between
text and bytes at the borders.
Native strings refer to the type that Python uses when you specify a bare
string literal:
.. code-block:: python
"This is a native string"
In Python 2, these are byte strings. In Python 3 these are text strings. Modules should be
coded to expect bytes on Python 2 and text on Python 3.
.. _module_utils_string_strategy:
Module_utils string strategy: hybrid
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In ``module_utils`` code we use a hybrid string strategy. Although Ansible's
``module_utils`` code is largely like module code, some pieces of it are
used by the controller as well. So it needs to be compatible with modules
and with the controller's assumptions, particularly the string strategy.
The module_utils code attempts to accept native strings as input
to its functions and emit native strings as their output.
In ``module_utils`` code:
* Functions **must** accept string parameters as either text strings or byte strings.
* Functions may return either the same type of string as they were given or the native string type for the Python version they are run on.
* Functions that return strings **must** document whether they return strings of the same type as they were given or native strings.
Module-utils functions are therefore often very defensive in nature.
They convert their string parameters into text (using ``ansible.module_utils._text.to_text``)
at the beginning of the function, do their work, and then convert
the return values into the native string type (using ``ansible.module_utils._text.to_native``)
or back to the string type that their parameters received.
Tips, tricks, and idioms for Python 2/Python 3 compatibility
------------------------------------------------------------
Use forward-compatibility boilerplate
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Use the following boilerplate code at the top of all python files
to make certain constructs act the same way on Python 2 and Python 3:
.. code-block:: python
@ -247,8 +288,8 @@ The ``__future__`` imports do the following:
* `PEP 0238: Division <https://www.python.org/dev/peps/pep-0238>`_
* `PEP 3105: Print function <https://www.python.org/dev/peps/pep-3105>`_
Prefix byte strings with "b\_"
------------------------------
Prefix byte strings with ``b\_``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Since mixing text and bytes types leads to tracebacks we want to be clear
about what variables hold text and what variables hold bytes. We do this by
@ -265,11 +306,11 @@ We do not prefix the text strings instead because we only operate
on byte strings at the borders, so there are fewer variables that need bytes
than text.
Bundled six
-----------
Import Ansible's bundled ``python-six`` library
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The third-party `python-six <https://pythonhosted.org/six/>`_ library exists
to help projects create code that runs on both Python-2 and Python-3. Ansible
to help projects create code that runs on both Python 2 and Python 3. Ansible
includes a version of the library in module_utils so that other modules can use it
without requiring that it is installed on the remote system. To make use of
it, import it like this:
@ -283,10 +324,10 @@ it, import it like this:
Ansible will use a system copy of six if the system copy is a later
version than the one Ansible bundles.
Exceptions
----------
Handle exceptions with ``as``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order for code to function on Python-2.6+ and Python-3, use the
In order for code to function on Python 2.6+ and Python 3, use the
new exception-catching syntax which uses the ``as`` keyword:
.. code-block:: python
@ -296,7 +337,7 @@ new exception-catching syntax which uses the ``as`` keyword:
except ValueError as e:
module.fail_json(msg="Tried to divide by zero: %s" % e)
Do **not** use the following syntax as it will fail on every version of Python-3:
Do **not** use the following syntax as it will fail on every version of Python 3:
.. This code block won't highlight because python2 isn't recognized. This is necessary to pass tests under python 3.
.. code-block:: none
@ -306,25 +347,25 @@ Do **not** use the following syntax as it will fail on every version of Python-3
except ValueError, e:
module.fail_json(msg="Tried to divide by zero: %s" % e)
Octal numbers
-------------
Update octal numbers
^^^^^^^^^^^^^^^^^^^^
In Python-2.x, octal literals could be specified as ``0755``. In Python-3,
In Python 2.x, octal literals could be specified as ``0755``. In Python 3,
octals must be specified as ``0o755``.
String formatting
-----------------
String formatting for controller code
-------------------------------------
str.format() compatibility
~~~~~~~~~~~~~~~~~~~~~~~~~~
Use ``str.format()`` for Python 2.6 compatibility
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Starting in Python-2.6, strings gained a method called ``format()`` to put
Starting in Python 2.6, strings gained a method called ``format()`` to put
strings together. However, one commonly used feature of ``format()`` wasn't
added until Python-2.7, so you need to remember not to use it in Ansible code:
added until Python 2.7, so you need to remember not to use it in Ansible code:
.. code-block:: python
# Does not work in Python-2.6!
# Does not work in Python 2.6!
new_string = "Dear {}, Welcome to {}".format(username, location)
# Use this instead
@ -332,120 +373,40 @@ added until Python-2.7, so you need to remember not to use it in Ansible code:
Both of the format strings above map positional arguments of the ``format()``
method into the string. However, the first version doesn't work in
Python-2.6. Always remember to put numbers into the placeholders so the code
is compatible with Python-2.6.
Python 2.6. Always remember to put numbers into the placeholders so the code
is compatible with Python 2.6.
.. seealso::
Python documentation on `format strings <https://docs.python.org/2/library/string.html#formatstrings>`_
Use percent format with byte strings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In Python-3.x, byte strings do not have a ``format()`` method. However, it
In Python 3.x, byte strings do not have a ``format()`` method. However, it
does have support for the older, percent-formatting.
.. code-block:: python
b_command_line = b'ansible-playbook --become-user %s -K %s' % (user, playbook_file)
.. note:: Percent formatting added in Python-3.5
.. note:: Percent formatting added in Python 3.5
Percent formatting of byte strings was added back into Python3 in 3.5.
This isn't a problem for us because Python-3.5 is our minimum version.
However, if you happen to be testing Ansible code with Python-3.4 or
Percent formatting of byte strings was added back into Python 3 in 3.5.
This isn't a problem for us because Python 3.5 is our minimum version.
However, if you happen to be testing Ansible code with Python 3.4 or
earlier, you will find that the byte string formatting here won't work.
Upgrade to Python-3.5 to test.
Upgrade to Python 3.5 to test.
.. seealso::
Python documentation on `percent formatting <https://docs.python.org/2/library/stdtypes.html#string-formatting>`_
---------------------------
Porting Modules to Python 3
---------------------------
.. _testing_modules_python_3:
Ansible modules are slightly harder to port than normal code from other
projects. A lot of mocking has to go into unit testing an Ansible module so
it's harder to test that your porting has fixed everything or to to make sure
that later commits haven't regressed the Python-3 support.
Testing modules on Python 3
===================================
Module String Strategy
======================
There are a large number of modules in Ansible. Most of those are maintained
by the Ansible community at large, not by a centralized team. To make life
easier on them, it was decided not to break backwards compatibility by
mandating that all strings inside of modules are text and converting between
text and bytes at the borders; instead, we're using a native string strategy
for now.
Native strings refer to the type that Python uses when you specify a bare
string literal:
.. code-block:: python
"This is a native string"
In Python-2, these are byte strings. In Python-3 these are text strings. The
module_utils shipped with Ansible attempts to accept native strings as input
to its functions and emit native strings as their output. Modules should be
coded to expect bytes on Python-2 and text on Python-3.
Tips, tricks, and idioms to adopt
=================================
Python-2.4 Compatible Exception Syntax
--------------------------------------
Until Ansible-2.4, modules needed to be compatible with Python-2.4 as
well. Python-2.4 did not understand the new exception-catching syntax so
we had to write a compatibility function that could work with both
Python-2 and Python-3. You may still see this used in some modules:
.. code-block:: python
from ansible.module_utils.pycompat24 import get_exception
try:
a = 2/0
except ValueError:
e = get_exception()
module.fail_json(msg="Tried to divide by zero: %s" % e)
Unless a change is going to be backported to Ansible-2.3, you should not
have to use this in new code.
Python 2.4 octal workaround
---------------------------
Before Ansible-2.4, modules had to be compatible with Python-2.4.
Python-2.4 did not understand the new syntax for octal literals so we used
the following workaround to specify octal values:
.. code-block:: python
# Can't use 0755 on Python-3 and can't use 0o755 on Python-2.4
EXECUTABLE_PERMS = int('0755', 8)
Unless a change is going to be backported to Ansible-2.3, you should not
have to use this in new code.
-------------------------------------
Porting module_utils code to Python 3
-------------------------------------
module_utils code is largely like module code. However, some pieces of it are
used by the controller as well. Because of this, it needs to be usable with
the controller's assumptions. This is most notable in the string strategy.
Module_utils String Strategy
============================
Module_utils **must** use the Native String Strategy. Functions in
module_utils receive either text strings or byte strings and may emit either
the same type as they were given or the native string for the Python version
they are run on depending on which makes the most sense for that function.
Functions which return strings **must** document whether they return text,
byte, or native strings. Module-utils functions are therefore often very
defensive in nature, converting from potential text or bytes at the
beginning of a function and converting to the native string type at the end.
Ansible modules are slightly harder to code to support Python 3 than normal code from other
projects. A lot of mocking has to go into unit testing an Ansible module, so
it's harder to test that your changes have fixed everything or to to make sure
that later commits haven't regressed the Python 3 support. Review our :ref:`testing <developing_testing>` pages
for more information.

26
docs/docsite/rst/dev_guide/developing_rebasing.rst
View file

@ -1,5 +1,6 @@
Rebasing a Pull Request
```````````````````````
***********************
Rebasing a pull request
***********************
You may find that your pull request (PR) is out-of-date and needs to be rebased. This can happen for several reasons:
@ -8,8 +9,8 @@ You may find that your pull request (PR) is out-of-date and needs to be rebased.
Rebasing the branch used to create your PR will resolve both of these issues.
Configuring Your Remotes
++++++++++++++++++++++++
Configuring your remotes
========================
Before you can rebase your PR, you need to make sure you have the proper remotes configured.
Assuming you cloned your fork in the usual fashion, the ``origin`` remote will point to your fork::
@ -37,8 +38,8 @@ Checking the status of your branch should show you're up-to-date with your fork
Your branch is up-to-date with 'origin/YOUR_BRANCH'.
nothing to commit, working tree clean
Rebasing Your Branch
++++++++++++++++++++
Rebasing your branch
====================
Once you have an ``upstream`` remote configured, you can rebase the branch for your PR::
@ -59,8 +60,8 @@ Once you've rebased, the status of your branch will have changed::
Don't worry, this is normal after a rebase. You should ignore the ``git status`` instructions to use ``git pull``.
We'll cover what to do next in the following section.
Updating Your Pull Request
++++++++++++++++++++++++++
Updating your pull request
==========================
Now that you've rebased your branch, you need to push your changes to GitHub to update your PR.
@ -71,8 +72,13 @@ Since rebasing re-writes git history, you will need to use a force push::
Your PR on GitHub has now been updated. This will automatically trigger testing of your changes.
You should check in on the status of your PR after tests have completed to see if further changes are required.
Getting Help Rebasing
+++++++++++++++++++++
Getting help rebasing
=====================
For help with rebasing your PR, or other development related questions, join us on our #ansible-devel IRC chat channel
on `freenode.net <https://freenode.net>`_.
.. seealso::
:ref:`community_development_process`
Information on roadmaps, opening PRs, Ansibullbot, and more

57
docs/docsite/rst/dev_guide/index.rst
View file

@ -4,26 +4,61 @@ Developer Guide
Welcome to the Ansible Developer Guide!
The purpose of this guide is to document all of the paths available to you for interacting and shaping Ansible with code, ranging from developing modules and plugins to helping to develop the Ansible Core Engine via pull requests.
**Who should use this guide?**
To get started, select one of the following topics.
If you want to extend Ansible by using a custom module or plugin locally, creating a module or plugin, adding functionality to an existing module, or expanding test coverage, this guide is for you. We've included detailed information for developers on how to test and document modules, as well as the prerequisites for getting your module or plugin accepted into the main Ansible repository.
Find the task that best describes what you want to do:
* I'm looking for a way to address a use case:
* I want to :ref:`add a custom plugin or module locally <developing_locally>`.
* I want to figure out if :ref:`developing a module is the right approach <module_dev_should_you>` for my use case.
* I'm ready to start developing:
* I want to :ref:`set up my Python development environment <environment_setup>`.
* I want to :ref:`get started writing a module <developing_modules_general>`.
* I want to :ref:`write a Windows module <developing_modules_general_windows>`.
* I want to :ref:`write a series of related modules <developing_modules_in_groups>` that integrate Ansible with a new product (for example, a database, cloud provider, network platform, etc.).
* I want to refine my code:
* I want to :ref:`debug my module code <debugging>`.
* I want to :ref:`test my module <developing_testing>`.
* I want to :ref:`document my module <module_documenting>`.
* I want to :ref:`make sure my code runs on Python 2 and Python 3 <developing_python_3>`.
* I want to work on other development projects:
* I want to :ref:`write a plugin <developing_plugins>`.
* I want to :ref:`connect Ansible to a new source of inventory <developing_inventory>`.
* I want to :ref:`deprecate an outdated module <deprecating_modules>`.
* I want to :ref:`contribute my module or plugin to Ansible Core <developing_modules_checklist>`.
If you prefer to read the entire guide, here's a list of the pages in order.
.. toctree::
:maxdepth: 2
overview_architecture
developing_locally
developing_modules
developing_program_flow_modules
developing_module_utilities
developing_modules_general
developing_modules_checklist
developing_modules_best_practices
developing_python_3
debugging
developing_modules_documenting
developing_modules_general_windows
developing_modules_in_groups
testing
module_lifecycle
developing_plugins
developing_inventory
developing_core
developing_python_3
developing_program_flow_modules
developing_api
developing_rebasing
testing
repomerge
../reference_appendices/release_and_maintenance
../community/committer_guidelines
./style_guide/index
developing_module_utilities
overview_architecture

46
docs/docsite/rst/dev_guide/module_lifecycle.rst Normal file
View file

@ -0,0 +1,46 @@
.. _module_lifecycle:
**********************************
The lifecycle of an Ansible module
**********************************
Modules in the main Ansible repo have a defined life cycle, from first introduction to final removal. The module life cycle is tied to the `Ansible release cycle <release_cycle>` and reflected in the :ref:`ansible_metadata_block`. A module may move through these four states:
1. When a module is first accepted into Ansible, we consider it in tech preview and mark it ``preview``. Modules in ``preview`` are not stable. You may change the parameters or dependencies, expand or reduce the functionality of ``preview`` modules. Many modules remain ``preview`` for years.
2. If a module matures, we may mark it ``stableinterface`` and commit to maintaining its parameters, dependencies, and functionality. We support (though we cannot guarantee) backwards compatibility for ``stableinterface`` modules, which means their parameters should be maintained with stable meanings.
3. If a module's target API changes radically, or if someone creates a better implementation of its functionality, we may mark it ``deprecated``. Modules that are ``deprecated`` are still available but they are reaching the end of their life cycle. We retain deprecated modules for 4 release cycles with deprecation warnings to help users update playbooks and roles that use them.
4. When a module has been deprecated for four release cycles, we remove the code and mark the stub file ``removed``. Modules that are ``removed`` are no longer shipped with Ansible. The stub file helps users find alternative modules.
.. _deprecating_modules:
Deprecating modules
===================
To deprecate a module, you must:
1. Rename the file so it starts with an ``_``, for example, rename ``old_cloud.py`` to ``_old_cloud.py``. This keeps the module available and marks it as deprecated on the module index pages.
2. Mention the deprecation in the relevant ``CHANGELOG``.
3. Reference the deprecation in the relevant ``porting_guide_x.y.rst``.
4. Update ``ANSIBLE_METADATA`` to contain ``status: ['deprecated']``.
5. Add ``deprecated:`` to the documentation with the following sub-values:
:removed_in: A ``string``, such as ``"2.9"``; the version of Ansible where the module will be replaced with a docs-only module stub. Usually current release +4.
:why: Optional string that used to detail why this has been removed.
:alternative: Inform users they should do instead, i.e. ``Use M(whatmoduletouseinstead) instead.``.
* For an example of documenting deprecation, see this `PR that deprecates multiple modules <https://github.com/ansible/ansible/pull/43781/files>`_.
Changing a module name
======================
You can also rename a module and keep an alias to the old name by using a symlink that starts with _.
This example allows the ``stat`` module to be called with ``fileinfo``, making the following examples equivalent::
EXAMPLES = '''
ln -s stat.py _fileinfo.py
ansible -m stat -a "path=/tmp" localhost
ansible -m fileinfo -a "path=/tmp" localhost
'''

16
docs/docsite/rst/dev_guide/overview_architecture.rst
View file

@ -1,5 +1,6 @@
********************
Ansible Architecture
====================
********************
Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.
@ -10,18 +11,19 @@ It uses no agents and no additional custom security infrastructure, so it's easy
In this section, we'll give you a really quick overview of how Ansible works so you can see how the pieces fit together.
Modules
`````````````````
=======
Ansible works by connecting to your nodes and pushing out small programs, called "Ansible Modules" to them. These programs are written to be resource models of the desired state of the system. Ansible then executes these modules (over SSH by default), and removes them when finished.
Your library of modules can reside on any machine, and there are no servers, daemons, or databases required. Typically you'll work with your favorite terminal program, a text editor, and probably a version control system to keep track of changes to your content.
Plugins
``````````````````
=======
Plugins are pieces of code that augment Ansible's core functionality. Ansible ships with a number of handy plugins, and you can easily write your own.
Inventory
````````````````````
=========
By default, Ansible represents what machines it manages using a very simple INI file that puts all of your managed machines in groups of your own choosing.
@ -45,7 +47,7 @@ Once inventory hosts are listed, variables can be assigned to them in simple tex
Or, as already mentioned, use a dynamic inventory to pull your inventory from data sources like EC2, Rackspace, or OpenStack.
Playbooks
````````````````````
=========
Playbooks can finely orchestrate multiple slices of your infrastructure topology, with very detailed control over how many machines to tackle at a time. This is where Ansible starts to get most interesting.
@ -66,7 +68,7 @@ Here's what a simple playbook looks like::
- content
Extending Ansible with Plug-ins and the API
````````````````````````````````````````````
Extending Ansible with plug-ins and the API
===========================================
Should you want to write your own, Ansible modules can be written in any language that can return JSON (Ruby, Python, bash, etc). Inventory can also plug in to any datasource by writing a program that speaks to that datasource and returns JSON. There's also various Python APIs for extending Ansible's connection types (SSH is not the only transport possible), callbacks (how Ansible logs, etc), and even for adding new server side behaviors.

29
docs/docsite/rst/dev_guide/repomerge.rst
View file

@ -1,29 +0,0 @@
==========
Repo Merge
==========
Background
----------
On Tuesday 6th December 2016, the Ansible Core Team re-merged the module repositories back into `ansible/ansible <https://github.com/ansible/ansible/>`_ in GitHub. The two module repos will be essentially locked, though they will be kept in place for the existing 2.1 and 2.2 dependencies. Once 2.2 moves out of official support (early 2018), these repositories will be fully readonly for all branches. Until then, any issues/PRs opened there will be auto-closed with a note to open it on `ansible/ansible <https://github.com/ansible/ansible/>`_.
Why Are We Doing This (Again...)?
-----------------------------------
For those who've been using Ansible long enough, you know that originally we started with a single repository. The original intention of the core vs. extras split was that core would be better supported/tested/etc. Extras would have been a bit more of a "wild-west" for modules, to allow new modules to make it into the distribution more quickly. Unfortunately this never really worked out, as well as the following:
1. Many modules in the core repo were also essentially "grand-fathered" in, despite not having a good set of tests or dedicated maintainers from the community.
2. The time in queue for modules to be merged into extras was not really any different from the time to merge modules into core.
3. The split introduced a few other problems for contributors such as having to submit multiple related PRs for modules with tests, or for those which rely on action plugins.
4. git submodules are notoriously complicated, even for contributors with decent git experience. The constant need to update git submodule pointers for devel and each stable branch can lead to testing surprises and really buys us nothing in terms of flexibility.
5. Users can already be confused about where to open issues, especially when the problem appears to be with a module but is actually an action plugin (ie. template) or something more fundamental like includes. Having everything back in one repo makes it easier to link issues, and you're always sure to open a bug report in the right place.
Metadata - Support/Ownership and Module Status
----------------------------------------------------------------------
As part of this move, we will be introducing module metadata, which will contain a couple of pieces of information regarding modules:
1. Support Status: This field indicates who supports the module, whether it's the core team, the community, the person who wrote it, or if it is an abandoned module which is not receiving regular updates. The Ansible team has gone through the list of modules and we have marked about 100 of them as "Core Supported", meaning a member of the Ansible core team should be actively fixing bugs on those modules. The vast majority of the rest will be community supported. This is not really a change from the status quo, this just makes it clearer.
2. Module Status: This field indicates how well supported that module may be. This generally applies to the maturity of the module's parameters, however, not necessarily its bug status.
The documentation pages for modules will be updated to reflect the above information as well, so that users can evaluate the status of a module before committing to using it in playbooks and roles.

8
docs/docsite/rst/dev_guide/shared_snippets/licensing.txt Normal file
View file

@ -0,0 +1,8 @@
.. note::
**LICENSING REQUIREMENTS** Ansible enforces the following licensing requirements:
* Utilities (files in ``lib/ansible/module_utils/``) may have one of two licenses:
* A ``module_util`` used **only** for a specific vendor's hardware, provider, or service may be licensed under GPLv3+.
* All other ``module_utils`` must be licensed under BSD, so GPL-licensed third-party and Galaxy modules can use them.
* If there's doubt about the appropriate license for a ``module_util``, the Ansible Core Team will decide during an Ansible Core Community Meeting.
* All other files shipped with Ansible, including all modules, must be licensed under the GPL license (GPLv3 or later).

11
docs/docsite/rst/dev_guide/testing.rst
View file

@ -1,8 +1,11 @@
.. _developing_testing:
***************
Testing Ansible
***************
.. contents:: Topics
:local:
Introduction
============
@ -19,17 +22,17 @@ Types of tests
At a high level we have the following classifications of tests:
:compile:
* :doc:`testing_compile`
* :ref:`testing_compile`
* Test python code against a variety of Python versions.
:sanity:
* :doc:`testing_sanity`
* :ref:`testing_sanity`
* Sanity tests are made up of scripts and tools used to perform static code analysis.
* The primary purpose of these tests is to enforce Ansible coding standards and requirements.
:integration:
* :doc:`testing_integration`
* :ref:`testing_integration`
* Functional tests of modules and Ansible core functionality.
:units:
* :doc:`testing_units`
* :ref:`testing_units`
* Tests directly against individual parts of the code base.

2
docs/docsite/rst/dev_guide/testing/sanity/ansible-var-precedence-check.rst
View file

@ -1,3 +1,5 @@
.. orphan:
Sanity Tests » ansible-var-precedence-check
===========================================

2
docs/docsite/rst/dev_guide/testing/sanity/docs-build.rst
View file

@ -1,3 +1,5 @@
.. orphan:
Sanity Tests » docs-build
=========================

4
docs/docsite/rst/dev_guide/testing/sanity/no-get-exception.rst
View file

@ -2,8 +2,8 @@ Sanity Tests » no-get-exception
===============================
We created a function, ``ansible.module_utils.pycompat24.get_exception`` to
help retrieve exceptions in a manner compatible with Python-2.4 through
Python-3.6. We no longer support Python-2.4 and Python-2.5 so this is
help retrieve exceptions in a manner compatible with Python 2.4 through
Python 3.6. We no longer support Python 2.4 and Python 2.5 so this is
extraneous and we want to deprecate the function. Porting code should look
something like this:

2
docs/docsite/rst/dev_guide/testing/sanity/no-wildcard-import.rst
View file

@ -1,3 +1,5 @@
.. orphan:
Sanity Tests » no-wildcard-import
=================================

2
docs/docsite/rst/dev_guide/testing/sanity/pylint-ansible-test.rst
View file

@ -1,3 +1,5 @@
.. orphan:
Sanity Tests » pylint-ansible-test
==================================

2
docs/docsite/rst/dev_guide/testing_compile.rst
View file

@ -1,3 +1,5 @@
.. _testing_compile:
*************
Compile Tests
*************

24
docs/docsite/rst/dev_guide/testing_documentation.rst Normal file
View file

@ -0,0 +1,24 @@
.. _testing_documentation:
*********************
Testing documentation
*********************
Before you submit a module for inclusion in the main Ansible repo, you must test your documentation for correct HTML rendering and to ensure that the argspec matches the documentation.
To check the HTML output of your module documentation:
#. Save your completed module file into the correct directory: ``lib/ansible/modules/$CATEGORY/my_code.py``.
#. Move to the docsite directory: ``cd /path/to/ansible/docs/docsite/``.
#. Run the command to build the docs for your module: ``MODULES=my_code make webdocs``.
#. View the HTML page at ``file:///path/to/ansible/docs/docsite/_build/html/my_code_module.html``.
To build the HTML documentation for multiple modules, use a comma-separated list of module names: ``MODULES=my_code,my_other_code make webdocs``.
To ensure that your documentation matches your ``argument_spec``, run the ``validate-modules`` test. Note that this option isn't currently enabled in Shippable due to the time it takes to run.
.. code-block:: bash
# If you don't already, ensure you are using your local checkout
source hacking/env-setup
./test/sanity/validate-modules/validate-modules --arg-spec --warnings lib/ansible/modules/$CATEGORY/my_code.py

2
docs/docsite/rst/dev_guide/testing_integration.rst
View file

@ -1,3 +1,5 @@
.. _testing_integration:
*****************
Integration tests
*****************

105
docs/docsite/rst/dev_guide/testing_integration_legacy.rst
View file

@ -1,105 +0,0 @@
*******************************************
Testing using the Legacy Integration system
*******************************************
.. contents:: Topics
This page details how to run the integration tests that haven't been ported to the new ``ansible-test`` framework.
The following areas are still tested using the legacy ``make tests`` command:
* amazon (some)
* azure
* cloudflare
* cloudscale
* cloudstack
* consul
* exoscale
* gce
* jenkins
* rackspace
Over time the above list will be reduced as tests are ported to the ``ansible-test`` framework.
Running Cloud Tests
====================
Cloud tests exercise capabilities of cloud modules (e.g. ec2_key). These are
not 'tests run in the cloud' so much as tests that leverage the cloud modules
and are organized by cloud provider.
Some AWS tests may use environment variables. It is recommended to either unset any AWS environment variables( such as ``AWS_DEFAULT_PROFILE``, ``AWS_SECRET_ACCESS_KEY``, etc) or be sure that the environment variables match the credentials provided in ``credentials.yml`` to ensure the tests run with consistency to their full capability on the expected account. See `AWS CLI docs <https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html>`_ for information on creating a profile.
Subsets of tests may be run by ``#commenting`` out unnecessary roles in the appropriate playbook, such as ``test/integration/amazon.yml``.
In order to run cloud tests, you must provide access credentials in a file
named ``credentials.yml``. A sample credentials file named
``credentials.template`` is available for syntax help.
Provide cloud credentials::
cp credentials.template credentials.yml
${EDITOR:-vi} credentials.yml
Other configuration
===================
In order to run some tests, you must provide access credentials in a file named
``credentials.yml``. A sample credentials file named ``credentials.template`` is available
for syntax help.
IAM policies for AWS
====================
In order to run the tests in an AWS account ansible needs fairly wide ranging powers which
can be provided to a dedicated user or temporary credentials using a specific policy
configured in the AWS account.
testing-iam-policy.json.j2
--------------------------
The testing-iam-policy.json.j2 file contains a policy which can be given to the user
running the tests to give close to minimum rights required to run the tests. Please note
that this does not fully restrict the user; The user has wide privileges for viewing
account definitions and is also able to manage some resources that are not related to
testing (e.g. AWS lambdas with different names) primarily due to the limitations of the
Amazon ARN notation. At the very least the policy limits the user to one region, however
tests should not be run in a primary production account in any case.
Other Definitions required
--------------------------
Apart from installing the policy and giving it to the user identity running
the tests, a lambda role `ansible_integration_tests` has to be created which
has lambda basic execution privileges.
Running Tests
=============
The tests are invoked via a ``Makefile``.
If you haven't already got Ansible available use the local checkout by doing::
source hacking/env-setup
Run the tests by doing::
cd test/integration/
# TARGET is the name of the test from the list at the top of this page
#make TARGET
# e.g.
make amazon
# To run all cloud tests you can do:
make cloud
.. warning:: Possible cost of running cloud tests
Running cloud integration tests will create and destroy cloud
resources. Running these tests may result in additional fees associated with
your cloud account. Care is taken to ensure that created resources are
removed. However, it is advisable to inspect your AWS console to ensure no
unexpected resources are running.

2
docs/docsite/rst/dev_guide/testing_pep8.rst
View file

@ -1,3 +1,5 @@
.. _testing_pep8:
*****
PEP 8
*****

2
docs/docsite/rst/dev_guide/testing_sanity.rst
View file

@ -1,3 +1,5 @@
.. _testing_sanity:
************
Sanity Tests
************

2
docs/docsite/rst/dev_guide/testing_units.rst
View file

@ -1,3 +1,5 @@
.. _testing_units:
**********
Unit Tests
**********

6
docs/docsite/rst/dev_guide/testing_units_modules.rst
View file

@ -531,10 +531,10 @@ A helpful development approach to this should be to ensure that all of the tests
run under Python 2.6 and that each assertion in the test cases has been checked to work by breaking
the code in Ansible to trigger that failure.
.. warning:: Maintain Python-2.6 compatibility
.. warning:: Maintain Python 2.6 compatibility
Please remember that modules need to maintain compatibility with Python-2.6 so the unittests for
modules should also be compatible with Python-2.6.
Please remember that modules need to maintain compatibility with Python 2.6 so the unittests for
modules should also be compatible with Python 2.6.
.. seealso::

69
docs/docsite/rst/reference_appendices/module_utils.rst Normal file
View file

@ -0,0 +1,69 @@
.. _ansible.module_utils:
.. _module_utils:
***************************************************************
Ansible Reference: Module Utilities
***************************************************************
This page documents the available utilities called with ``ansible.module_utils.util_name``.
Generic
--------
.. glossary::
.. _ansible.module_utils.debug
debug
.. _ansible.module_utils.url
url
.. _ansible.module_utils.log
log
.. _ansible.module_utils.no_log
no_log
.. _Ansible.get_bin_path
Ansible.get_bin_path
.. _AnsibleModule:
.. _ansible.module_utils.basic.AnsibleModule:
AnsibleModule
--------------
.. glossary::
ansible.module_utils.basic.AnsibleModule
Utilities in module_utils.basic.AnsibleModule apply to all module types
.. _AnsibleModule.debug:
AnsibleModule.debug
.. _AnsibleModule._debug:
AnsibleModule._debug
.. _AnsibleModule._diff:
AnsibleModule._diff
.. _AnsibleModule.log:
AnsibleModule.log
.. _AnsibleModule.no_log:
AnsibleModule.no_log
.. _AnsibleModule.params:
AnsibleModule.params
.. _AnsibleModule.run_command:
AnsibleModule.run_command
.. _ansible.module_utils.basic.AnsibleModule._selinux_special_fs:
ansible.module_utils.basic.AnsibleModule._selinux_special_fs
(formerly ansible.module_utils.basic.SELINUX_SPECIAL_FS)
.. _ansible.module_utils.basic:
Basic
------
.. glossary::
ansible.module_utils.basic
Utilities in module_utils.basic apply to all module types.
.. _ansible.module_utils.basic.SELINUX_SPECIAL_FS:
ansible.module_utils.basic.SELINUX_SPECIAL_FS
*deprecated* replaced by :term:`ansible.module_utils.basic.AnsibleModule._selinux_special_fs`
.. _ansible.module_utils.basic._load_params:
load_params