Update the Ansiballz developer docs

docsite/rst/developing_program_flow_modules.rst

@@ -24,10 +24,10 @@ Action Plugins
 --------------
 
 Action Plugins look like modules to end users who are writing :term:`playbooks` but
-they're distinct entities for the purposes of this paper.  Action Plugins
+they're distinct entities for the purposes of this document.  Action Plugins
 always execute on the controller and are sometimes able to do all work there
-(for instance, the debug Action Plugin which prints some text for the user to
+(for instance, the ``debug`` Action Plugin which prints some text for the user to
-see or the assert Action Plugin which can test whether several values in
+see or the ``assert`` Action Plugin which can test whether several values in
 a playbook satisfy certain criteria.)
 
 More often, Action Plugins set up some values on the controller, then invoke an
@@ -57,7 +57,7 @@ connections instead of only one.
 Python
 ^^^^^^
 
-New-style Python modules use the :ref:`ziploader` framework for constructing
+New-style Python modules use the :ref:`Ansiballz` framework for constructing
 modules.  All official modules (shipped with Ansible) use either this or the
 :ref:`powershell module framework <flow_powershell_modules>`.
 
@@ -66,7 +66,7 @@ boilerplate module code, such as argument parsing, formatting of return
 values as :term:`JSON`, and various file operations.
 
 .. note:: In Ansible, up to version 2.0.x, the official Python modules used the
-    :ref:`module_replacer` framework.  For module authors, :ref:`ziploader` is
+    :ref:`module_replacer` framework.  For module authors, :ref:`Ansiballz` is
     largely a superset of :ref:`module_replacer` functionality, so you usually
     do not need to know about one versus the other.
 
@@ -127,6 +127,25 @@ only modifies them to change a shebang line if present.
 .. seealso:: Examples of Non-native modules written in ruby are in the `Ansible
     for Rubyists <https://github.com/ansible/ansible-for-rubyists>`_ repository.
 
+.. _flow_binary_modules:
+
+Binary Modules
+--------------
+
+From Ansible 2.2 onwards, modules may also be small binary programs.  Ansible
+doesn't perform any magic to make these portable to different systems so they
+may be specific to the system on which they were compiled or require other
+binary runtime dependencies.  Despite these drawbacks, a site may sometimes
+have no choice but to compile a custom module against a specific binary
+library if that's the only way they have to get access to certain resources.
+
+Binary modules take their arguments and will return data to Ansible in the same
+way as :ref:`want JSON modules <flow_want_json_modules>`.
+
+.. seealso:: One example of a `binary module
+    <https://github.com/ansible/ansible/blob/devel/test/integration/library/helloworld.go>`_
+    written in go.
+
 .. _flow_old_style_modules:
 
 Old-style Modules
@@ -174,22 +193,23 @@ the primary coordinator of much of the work to actually execute the module on
 the managed machine.
 
 * It takes care of creating a connection to the managed machine by
-  instantiating a Connection class according to the inventory configuration for
+  instantiating a ``Connection`` class according to the inventory
-  that host.
+  configuration for that host.
 * It adds any internal Ansible variables to the module's parameters (for
   instance, the ones that pass along ``no_log`` to the module).
 * It takes care of creating any temporary files on the remote machine and
   cleans up afterwards.
 * It does the actual work of pushing the module and module parameters to the
   remote host, although the :ref:`module_common <flow_executor_module_common>`
-  code described next does the work of deciding which format those will take.
+  code described in the next section does the work of deciding which format
+  those will take.
 * It handles any special cases regarding modules (for instance, various
   complications around Windows modules that must have the same names as Python
   modules, so that internal calling of modules from other Action Plugins work.)
 
 Much of this functionality comes from the :class:`BaseAction` class,
-which lives in :file:`plugins/action/__init__.py`.  It makes use of Connection
+which lives in :file:`plugins/action/__init__.py`.  It makes use of
-and Shell objects to do its work.
+``Connection`` and ``Shell`` objects to do its work.
 
 .. note::
     When :term:`tasks <tasks>` are run with the ``async:`` parameter, Ansible
@@ -207,15 +227,16 @@ to be shipped to the managed node.  The module is first read in, then examined
 to determine its type.  :ref:`PowerShell <flow_powershell_modules>` and
 :ref:`JSON-args modules <flow_jsonargs_modules>` are passed through
 :ref:`Module Replacer <module_replacer>`.  New-style
-:ref:`Python modules <flow_python_modules>` are assembled by :ref:`ziploader`.
+:ref:`Python modules <flow_python_modules>` are assembled by :ref:`Ansiballz`.
-:ref:`Non-native-want-JSON <flow_want_json_modules>` and
+:ref:`Non-native-want-JSON <flow_want_json_modules>`,
+:ref:`Binary modules <flow_binary_modules>`, and
 :ref:`Old-Style modules <flow_old_style_modules>` aren't touched by either of
 these and pass through unchanged.  After the assembling step, one final
 modification is made to all modules that have a shebang line.  Ansible checks
 whether the interpreter in the shebang line has a specific path configured via
 an ``ansible_$X_interpreter`` inventory variable.  If it does, Ansible
 substitutes that path for the interpreter path given in the module.  After
-this Ansible returns the complete module data and the module type to the
+this, Ansible returns the complete module data and the module type to the
 :ref:`Normal Action <_flow_normal_action_plugin>` which continues execution of
 the module.
 
@@ -226,8 +247,9 @@ Next we'll go into some details of the two assembler frameworks.
 Module Replacer
 ^^^^^^^^^^^^^^^
 
-The Module Replacer is essentially a preprocessor (like the C Preprocessor for
+The Module Replacer framework is the original framework implementing new-style
-those familiar with that language).  It does straight substitutions of
+modules.  It is essentially a preprocessor (like the C Preprocessor for those
+familiar with that programming language).  It does straight substitutions of
 specific substring patterns in the module file.  There are two types of
 substitutions:
 
@@ -250,47 +272,56 @@ substitutions:
   replacements, but shouldn't be used directly by modules.
 
   - :code:`"<<ANSIBLE_VERSION>>"` is substituted with the Ansible version.  In
-    a new-style Python module, it's better to use ``from ansible import
+    :ref:`new-style Python modules <flow_python_modules>` under the
-    __version__`` and then use ``__version__`` instead.
+    :ref:`Ansiballz` frameworkthe proper way is to instead instantiate an
+    :class:`AnsibleModule` and then access the version from
+    :attr:``AnsibleModule.ansible_version``.
   - :code:`"<<INCLUDE_ANSIBLE_MODULE_COMPLEX_ARGS>>"` is substituted with
     a string which is the Python ``repr`` of the :term:`JSON` encoded module
     parameters.  Using ``repr`` on the JSON string makes it safe to embed in
-    a Python file.  In :ref:`new-style Python modules <flow_python_modules>`
+    a Python file.  In new-style Python modules under the Ansiballz framework
-    under :ref:`ziploader` this is passed in via an environment variable
+    this is better accessed by instantiating an :class:`AnsibleModule` and
-    instead.
+    then using :attr:`AnsibleModule.params`.
   - :code:`<<SELINUX_SPECIAL_FILESYSTEMS>>` substitutes a string which is
     a comma separated list of file systems which have a file system dependent
-    security context in SELinux.  In new-style Python modules, this is found
+    security context in SELinux.  In new-style Python modules, if you really
-    by looking up ``SELINUX_SPECIAL_FS`` from the
+    need this you should instantiate an :class:`AnsibleModule` and then use
-    :envvar:`ANSIBLE_MODULE_CONSTANTS` environment variable.  See the
+    :attr:`AnsibleModule._selinux_special_fs`.  The variable has also changed
-    :ref:`ziploader` documentation for details.
+    from a comma separated string of file system names to an actual python
+    list of filesystem names.
   - :code:`<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>` substitutes the module
     parameters as a JSON string.  Care must be taken to properly quote the
     string as JSON data may contain quotes.  This pattern is not substituted
-    in new-style Python modules as they can get the module parameters via the
+    in new-style Python modules as they can get the module parameters another
-    environment variable.
+    way.
-  - the string :code:`syslog.LOG_USER` is replaced wherever it occurs with the
+  - The string :code:`syslog.LOG_USER` is replaced wherever it occurs with the
-    value of ``syslog_facility`` from the :file:`ansible.cfg` or any
+    ``syslog_facility`` which was named in :file:`ansible.cfg` or any
     ``ansible_syslog_facility`` inventory variable that applies to this host.  In
-    new-style Python modules, you can get the value of the ``syslog_facility``
+    new-style Python modules this has changed slightly.  If you really need to
-    by looking up ``SYSLOG_FACILITY`` in the :envvar:`ANSIBLE_MODULE_CONSTANTS`
+    access it, you should instantiate an :class:`AnsibleModule` and then use
-    environment variable.  See the :ref:`ziploader` documentation for details.
+    :attr:`AnsibleModule._syslog_facility` to access it.  It is no longer the
+    actual syslog facility and is now the name of the syslog facility.  See
+    the :ref:`documentation on internal arguments <flow_internal_arguments>`
+    for details.
 
-.. _ziploader:
+.. _Ansiballz:
 
-ziploader
+Ansiballz
 ^^^^^^^^^
 
-Ziploader differs from :ref:`module_replacer` in that it uses real Python
+Ansible 2.1 switched from the :ref:`module_replacer` framework to the
-imports of things in module_utils instead of merely preprocessing the module.
+Ansiballz framework for assembling modules.  The Ansiballz framework differs
-It does this by constructing a zipfile--which includes the module file, files
+from module replacer in that it uses real Python imports of things in
+:file:`ansible/module_utils` instead of merely preprocessing the module.  It
+does this by constructing a zipfile -- which includes the module file, files
 in :file:`ansible/module_utils` that are imported by the module, and some
-boilerplate to pass in the constants.  The zipfile is then Base64 encoded and
+boilerplate to pass in the module's parameters.  The zipfile is then Base64
-wrapped in a small Python script which decodes the Base64 encoding and places
+encoded and wrapped in a small Python script which decodes the Base64 encoding
-the zipfile into a temp direcrtory on the managed node.  It then extracts just
+and places the zipfile into a temp directory on the managed node.  It then
-the ansible module script from the zip file and places that in the temporary
+extracts just the ansible module script from the zip file and places that in
-directory as well.  Then it sets the PYTHONPATH to find python modules inside
+the temporary directory as well.  Then it sets the PYTHONPATH to find python
-of the zip file and invokes :command:`python` on the extracted ansible module.
+modules inside of the zip file and invokes :command:`python` on the extracted
+ansible module.
 
 .. note::
     Ansible wraps the zipfile in the Python script for two reasons:
@@ -301,19 +332,20 @@ of the zip file and invokes :command:`python` on the extracted ansible module.
       Python module into the Python interpreter on the remote node.  Python
       understands scripts on stdin but does not understand zip files.
 
-In ziploader, any imports of Python modules from the ``ansible.module_utils``
+In Ansiballz, any imports of Python modules from the
-package trigger inclusion of that Python file into the zipfile.  Instances of
+:py:mod:`ansible.module_utils` package trigger inclusion of that Python file
-:code:`#<<INCLUDE_ANSIBLE_MODULE_COMMON>>` in the module are turned into
+into the zipfile.  Instances of :code:`#<<INCLUDE_ANSIBLE_MODULE_COMMON>>` in
-:code:`from ansible.module_utils.basic import *` and
+the module are turned into :code:`from ansible.module_utils.basic import *`
-:file:`ansible/module-utils/basic.py` is then included in the zipfile.  Files
+and :file:`ansible/module-utils/basic.py` is then included in the zipfile.
-that are included from module_utils are themselves scanned for imports of other
+Files that are included from :file:`module_utils` are themselves scanned for
-Python modules from module_utils to be included in the zipfile as well.
+imports of other Python modules from :file:`module_utils` to be included in
+the zipfile as well.
 
 .. warning::
-    At present, Ziploader cannot determine whether an import should be
+    At present, the Ansiballz Framework cannot determine whether an import
-    included if it is a relative import.  Always use an absolute import that
+    should be included if it is a relative import.  Always use an absolute
-    has ``ansible.module_utils`` in it to allow ziploader to determine that
+    import that has :py:mod:`ansible.module_utils` in it to allow Ansiballz to
-    the file should be included.
+    determine that the file should be included.
 
 .. _flow_passing_module_args:
 
@@ -321,60 +353,133 @@ Passing args
 ~~~~~~~~~~~~
 
 In :ref:`module_replacer`, module arguments are turned into a JSON-ified
-string and substituted into the combined module file.  In :ref:`ziploader`,
+string and substituted into the combined module file.  In :ref:`Ansiballz`,
 the JSON-ified string is passed into the module via stdin.  When
 a  :class:`ansible.module_utils.basic.AnsibleModule` is instantiated,
 it parses this string and places the args into
 :attr:`AnsibleModule.params` where it can be accessed by the module's
 other code.
 
-.. _flow_passing_module_constants:
+.. note::
+    Internally, the :class:`AnsibleModule` uses the helper function,
+    :py:func:`ansible.module_utils.basic._load_params`, to load the parameters
+    from stdin and save them into an internal global variable.  Very dynamic
+    custom modules which need to parse the parameters prior to instantiating
+    an ``AnsibleModule`` may use ``_load_params`` to retrieve the
+    parameters.  Be aware that ``_load_params`` is an internal function and
+    may change in breaking ways if necessary to support changes in the code.
+    However, we'll do our best not to break it gratuitously, which is not
+    something that can be said for either the way parameters are passed or
+    the internal global variable.
 
-Passing constants
+.. _flow_internal_arguments:
-~~~~~~~~~~~~~~~~~
 
-Currently, there are three constants passed from the controller to the modules:
+Internal arguments
-``ANSIBLE_VERSION``, ``SELINUX_SPECIAL_FS``, and ``SYSLOG_FACILITY``.  In
+^^^^^^^^^^^^^^^^^^
-:ref:`module_replacer`, ``ANSIBLE_VERSION`` and ``SELINUX_SPECIAL_FS`` were
-substituted into the global variables
-:code:`ansible.module_utils.basic.ANSIBLE_VERSION` and
-:code:`ansible.module_utils.basic.SELINUX_SPECIAL_FS`.  ``SYSLOG_FACILITY`` didn't
-get placed into a variable.  Instead, any occurrences of the string
-``syslog.LOG_USER`` in the combined module file were replaced with ``syslog.``
-followed by the string contained in ``SYSLOG_FACILITY``.  All of these have
-changed in :ref:`ziploader`.
 
-The Ansible verison can now be used by a module by importing ``__version__``
+Both :ref:`module replacer` and :ref:`Ansiballz` send additional arguments to
-from ansible::
+the module beyond those which the user specified in the playbook.  These
+additional arguments are internal parameters that help implement global
+Ansible features.  Modules often do not need to know about these explicitly as
+the features are implemented in :py:mod:`ansible.module_utils.basic` but certain
+features need support from the module so it's good to know about them.
 
-    from ansible import __version__
+_ansible_no_log
-    module.exit_json({'msg': 'module invoked by ansible %s' % __version__})
+~~~~~~~~~~~~~~~
 
-For now, :code:`ANSIBLE_VERSION` is also available at its old location inside of
+This is a boolean.  If it's True then the playbook specified ``no_log`` (in
-``ansible.module_utils.basic``, but that will eventually be removed.
+a task's parameters or as a play parameter).  This automatically affects calls
+to :py:meth:`AnsibleModule.log`.  If a module implements its own logging then
+it needs to check this value.  The best way to look at this is for the module
+to instantiate an :class:`AnsibleModule` and then check the value of
+:attr:`AnsibleModule.no_log`.
 
-``SELINUX_SPECIAL_FS`` and  ``SYSLOG_FACILITY`` have changed much more.
+.. note::
-:ref:`ziploader` passes these as part of the JSON-ified argument string via stdin.
+    ``no_log`` specified in a module's argument_spec are handled by a different mechanism.
-When
-:class:`ansible.module_utils.basic.AnsibleModule` is instantiated, it parses this
-string and places the constants into :attr:`AnsibleModule.constants`
-where other code can access it.
 
-Unlike the ``ANSIBLE_VERSION``, where some efforts were made to keep the old
+_ansible_debug
-backwards compatible globals available, these two constants are not available
+~~~~~~~~~~~~~~
-at their old names.  This is a combination of the degree to which these are
-internal to the needs of ``module_utils.basic`` and, in the case of
-``SYSLOG_FACILITY``, how hacky and unsafe the previous implementation was.
 
-Porting code from the :ref:`module_replacer` method of getting
+This is a boolean that turns on more verbose logging.  If a module uses
-``SYSLOG_FACILITY`` to the new one is a little more tricky than the other
+:py:meth:`AnsibleModule.debug` rather than :py:meth:`AnsibleModule.log` then
-constants and args, due to just how hacky the old way was.  Here's an example
+the messages are only logged if this is True.  This also turns on logging of
-of using it in the new way::
+external commands that the module executes.  This can be changed via
+the``debug`` setting in :file:`ansible.cfg` or the environment variable
+:envvar:`ANSIBLE_DEBUG`.  If, for some reason, a module must access this, it
+should do so by instantiating an :class:`AnsibleModule` and accessing
+:attr:`AnsibleModule._debug`.
 
+_ansible_diff
+~~~~~~~~~~~~~
+
+This boolean is turned on via the ``--diff`` command line option.  If a module
+supports it, it will tell the module to show a unified diff of changes to be
+made to templated files.  The proper way for a module to access this is by
+instantiating an :class:`AnsibleModule` and accessing
+:attr:`AnsibleModule._diff`.
+
+_ansible_verbosity
+~~~~~~~~~~~~~~~~~~
+
+This value could be used for finer grained control over logging. However, it
+is currently unused.
+
+_ansible_selinux_special_fs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This is a list of names of filesystems which should have a special selinux
+context.  They are used by the :class:`AnsibleModule` methods which operate on
+files (changing attributes, moving, and copying).  The list of names is set
+via a comma separated string of filesystem names from :file:`ansible.cfg`::
+
+  # ansible.cfg
+  [selinux]
+  special_context_filesystems=nfs,vboxsf,fuse,ramfs
+
+If a module cannot use the builtin ``AnsibleModule`` methods to manipulate
+files and needs to know about these special context filesystems, it should
+instantiate an ``AnsibleModule`` and then examine the list in
+:attr:`AnsibleModule._selinux_special_fs`.
+
+This replaces :attr:`ansible.module_utils.basic.SELINUX_SPECIAL_FS` from
+:ref:`module_replacer`.  In module replacer it was a comma separated string of
+filesystem names.  Under Ansiballz it's an actual list.
+
+.. versionadded:: 2.1
+
+_ansible_syslog_facility
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+This parameter controls which syslog facility ansible module logs to.  It may
+be set by changing the ``syslog_facility`` value in :file:`ansible.cfg`.  Most
+modules should just use :meth:`AnsibleModule.log` which will then make use of
+this.  If a module has to use this on its own, it should instantiate an
+:class:`AnsibleModule` and then retrieve the name of the syslog facility from
+:attr:`AnsibleModule._syslog_facility`.  The code will look slightly different
+than it did under :ref:`module_replacer` due to how hacky the old way was::
+
+        # Old way
         import syslog
-        facility_name = module.constants.get('SYSLOG_FACILITY')
+        syslog.openlog(NAME, 0, syslog.LOG_USER)
-        facility = getattr(syslog, facility_name)
+
-        syslog.openlog(str(module), 0, facility)
+        # New way
+        import syslog
+        facility_name = module._syslog_facility
+        facility = getattr(syslog, facility_name, syslog.LOG_USER)
+        syslog.openlog(NAME, 0, facility)
+
+.. versionadded:: 2.1
+
+_ansible_version
+~~~~~~~~~~~~~~~~
+
+This parameter passes the version of ansible that runs the module.  To access
+it, a module should instantiate an :class:`AnsibleModule` and then retrieve it
+from :attr:`AnsibleModule.ansible_version`.  This replaces
+:attr:`ansible.module_utils.basic.ANSIBLE_VERSION` from
+:ref:`module_replacer`.
+
+.. versionadded:: 2.1
 
 .. _flow_special_considerations:
 
@@ -398,3 +503,21 @@ Pipelining only works with modules written in Python at this time because
 Ansible only knows that Python supports this mode of operation.  Supporting
 pipelining means that whatever format the module payload takes before being
 sent over the wire must be executable by Python via stdin.
+
+.. _flow_args_over_stdin:
+
+Why pass args over stdin?
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Passing arguments via stdin was chosen for the following reasons:
+
+* When combined with :ref:`pipelining`, this keeps the module's arguments from
+  temporarily being saved onto disk on the remote machine.  This makes it
+  harder (but not impossible) for a malicious user on the remote machine to
+  steal any sensitive information that may be present in the arguments.
+* Command line arguments would be insecure as most systems allow unprivileged
+  users to read the full commandline of a process.
+* Environment variables are usually more secure than the commandline but some
+  systems limit the total size of the environment.  This could lead to
+  truncation of the parameters if we hit that limit.
+