CmdRunner guide (#8592)

* initial commit

* wip

* WIP

* progressing

* remove copied part

* progressing

* document as_func()

* complete docs about arg formatting

* add PythonRunner documentation

* fix markup

* improve markup

* multiple changes

- revamp the cmd_runner_fmt doc
- add CmdRunner instance and context parameters
- changed formatting of other blocks

* fix typo + reduce tables sizes

* improve text

* fixes and adjustments

* fix sanity

* fix version added

.github/BOTMETA.yml

@@ -1468,14 +1468,16 @@ files:
     maintainers: ericzolf
   docs/docsite/rst/guide_alicloud.rst:
     maintainers: xiaozhu36
+  docs/docsite/rst/guide_cmdrunner.rst:
+    maintainers: russoz
+  docs/docsite/rst/guide_deps.rst:
+    maintainers: russoz
   docs/docsite/rst/guide_online.rst:
     maintainers: remyleone
   docs/docsite/rst/guide_packet.rst:
     maintainers: baldwinSPC nurfet-becirevic t0mk teebes
   docs/docsite/rst/guide_scaleway.rst:
     maintainers: $team_scaleway
-  docs/docsite/rst/guide_deps.rst:
-    maintainers: russoz
   docs/docsite/rst/guide_vardict.rst:
     maintainers: russoz
   docs/docsite/rst/test_guide.rst:

docs/docsite/extra-docs.yml

@@ -18,3 +18,4 @@ sections:
     toctree:
       - guide_deps
       - guide_vardict
+      - guide_cmdrunner

docs/docsite/rst/guide_cmdrunner.rst

@@ -0,0 +1,463 @@
+..
+  Copyright (c) Ansible Project
+  GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+  SPDX-License-Identifier: GPL-3.0-or-later
+
+.. _ansible_collections.community.general.docsite.guide_cmdrunner:
+
+
+Command Runner guide
+====================
+
+
+Introduction
+^^^^^^^^^^^^
+
+The ``ansible_collections.community.general.plugins.module_utils.cmd_runner`` module util provides the
+``CmdRunner`` class to help execute external commands. The class is a wrapper around
+the standard ``AnsibleModule.run_command()`` method, handling command arguments, localization setting,
+output processing output, check mode, and other features.
+
+It is even more useful when one command is used in multiple modules, so that you can define all options
+in a module util file, and each module uses the same runner with different arguments.
+
+For the sake of clarity, throughout this guide, unless otherwise specified, we use the term *option* when referring to
+Ansible module options, and the term *argument* when referring to the command line arguments for the external command.
+
+
+Quickstart
+""""""""""
+
+``CmdRunner`` defines a command and a set of coded instructions on how to format
+the command-line arguments, in which specific order, for a particular execution.
+It relies on ``ansible.module_utils.basic.AnsibleModule.run_command()`` to actually execute the command.
+There are other features, see more details throughout this document.
+
+To use ``CmdRunner`` you must start by creating an object. The example below is a simplified
+version of the actual code in :ansplugin:`community.general.ansible_galaxy_install#module`:
+
+.. code-block:: python
+
+    from ansible_collections.community.general.plugins.module_utils.cmd_runner import CmdRunner, cmd_runner_fmt
+
+    runner = CmdRunner(
+        module,
+        command="ansible-galaxy",
+        arg_formats=dict(
+            type=cmd_runner_fmt.as_func(lambda v: [] if v == 'both' else [v]),
+            galaxy_cmd=cmd_runner_fmt.as_list(),
+            upgrade=cmd_runner_fmt.as_bool("--upgrade"),
+            requirements_file=cmd_runner_fmt.as_opt_val('-r'),
+            dest=cmd_runner_fmt.as_opt_val('-p'),
+            force=cmd_runner_fmt.as_bool("--force"),
+            no_deps=cmd_runner_fmt.as_bool("--no-deps"),
+            version=cmd_runner_fmt.as_fixed("--version"),
+            name=cmd_runner_fmt.as_list(),
+        )
+    )
+
+This is meant to be done once, then every time you need to execute the command you create a context and pass values as needed:
+
+.. code-block:: python
+
+        # Run the command with these arguments, when values exist for them
+        with runner("type galaxy_cmd upgrade force no_deps dest requirements_file name", output_process=process) as ctx:
+            ctx.run(galaxy_cmd="install", upgrade=upgrade)
+
+        # version is fixed, requires no value
+        with runner("version") as ctx:
+            dummy, stdout, dummy = ctx.run()
+
+        # Another way of expressing it
+        dummy, stdout, dummy = runner("version").run()
+
+Note that you can pass values for the arguments when calling ``run()``,
+otherwise ``CmdRunner`` uses the module options with the exact same names to
+provide values for the runner arguments. If no value is passed and no module option
+is found for the name specified, then an exception is raised, unless the
+argument is using ``cmd_runner_fmt.as_fixed`` as format function like the
+``version`` in the example above. See more about it below.
+
+In the first example, values of ``type``, ``force``, ``no_deps`` and others
+are taken straight from the module, whilst ``galaxy_cmd`` and ``upgrade`` are
+passed explicitly.
+
+That generates a resulting command line similar to (example taken from the
+output of an integration test):
+
+.. code-block:: python
+
+        [
+            "<venv>/bin/ansible-galaxy",
+            "collection",
+            "install",
+            "--upgrade",
+            "-p",
+            "<collection-install-path>",
+            "netbox.netbox",
+        ]
+
+
+Argument formats
+^^^^^^^^^^^^^^^^
+
+As seen in the example, ``CmdRunner`` expects a parameter named ``arg_formats``
+defining how to format each CLI named argument.
+An "argument format" is nothing but a function to transform the value of a variable
+into something formatted for the command line.
+
+
+Argument format function
+""""""""""""""""""""""""
+
+An ``arg_format`` function should be of the form:
+
+.. code-block:: python
+
+    def func(value):
+        return ["--some-param-name", value]
+
+The parameter ``value`` can be of any type - although there are convenience
+mechanisms to help handling sequence and mapping objects.
+
+The result is expected to be of the type ``Sequence[str]`` type (most commonly
+``list[str]`` or ``tuple[str]``), otherwise it is considered to be a ``str``,
+and it is coerced into ``list[str]``.
+This resulting sequence of strings is added to the command line when that
+argument is actually used.
+
+For example, if ``func`` returns:
+
+- ``["nee", 2, "shruberries"]``, the command line adds arguments ``"nee" "2" "shruberries"``.
+- ``2 == 2``, the command line adds argument ``True``.
+- ``None``, the command line adds argument ``None``.
+- ``[]``, the command line adds no command line argument for that particular argument.
+
+
+Convenience format methods
+""""""""""""""""""""""""""
+
+In the same module as ``CmdRunner`` there is a class ``cmd_runner_fmt`` which
+provides a set of convenience methods that return format functions for common cases.
+In the first block of code in the `Quickstart`_ section you can see the importing of
+that class:
+
+.. code-block:: python
+
+    from ansible_collections.community.general.plugins.module_utils.cmd_runner import CmdRunner, cmd_runner_fmt
+
+The same example shows how to make use of some of them in the instantiation of the ``CmdRunner`` object.
+A description of each one of the convenience methods available and examples of how to use them is found below.
+In these descriptions ``value`` refers to the single parameter passed to the formatting function.
+
+- ``cmd_runner_fmt.as_list()``
+    This method does not receive any parameter, function returns ``value`` as-is.
+
+    - Creation:
+        ``cmd_runner_fmt.as_list()``
+    - Example:
+        +----------------------+---------------------+
+        | Value                | Outcome             |
+        +======================+=====================+
+        | ``["foo", "bar"]``   | ``["foo", "bar"]``  |
+        +----------------------+---------------------+
+        | ``"foobar"``         | ``["foobar"]``      |
+        +----------------------+---------------------+
+
+- ``cmd_runner_fmt.as_bool()``
+    This method receives two different parameters: ``args_true`` and ``args_false``, latter being optional.
+    If the boolean evaluation of ``value`` is ``True``, the format function returns ``args_true``.
+    If the boolean evaluation is ``False``, then the function returns ``args_false``
+    if it was provided, or ``[]`` otherwise.
+
+    - Creation:
+        ``cmd_runner_fmt.as_bool("--force")``
+    - Example:
+        +------------+--------------------+
+        | Value      | Outcome            |
+        +============+====================+
+        | ``True``   | ``["--force"]``    |
+        +------------+--------------------+
+        | ``False``  | ``[]``             |
+        +------------+--------------------+
+
+- ``cmd_runner_fmt.as_bool_not()``
+    This method receives one parameter, which is returned by the function when the boolean evaluation
+    of ``value`` is ``False``.
+
+    - Creation:
+        ``cmd_runner_fmt.as_bool_not("--no-deps")``
+    - Example:
+        +-------------+---------------------+
+        | Value       | Outcome             |
+        +=============+=====================+
+        | ``True``    | ``[]``              |
+        +-------------+---------------------+
+        | ``False``   | ``["--no-deps"]``   |
+        +-------------+---------------------+
+
+- ``cmd_runner_fmt.as_optval()``
+    This method receives one parameter ``arg``, the function returns the string concatenation
+    of ``arg`` and ``value``.
+
+    - Creation:
+        ``cmd_runner_fmt.as_optval("-i")``
+    - Example:
+        +---------------+---------------------+
+        | Value         | Outcome             |
+        +===============+=====================+
+        | ``3``         | ``["-i3"]``         |
+        +---------------+---------------------+
+        | ``foobar``    | ``["-ifoobar"]``    |
+        +---------------+---------------------+
+
+- ``cmd_runner_fmt.as_opt_val()``
+    This method receives one parameter ``arg``, the function returns ``[arg, value]``.
+
+    - Creation:
+        ``cmd_runner_fmt.as_opt_val("--name")``
+    - Example:
+        +--------------+--------------------------+
+        | Value        | Outcome                  |
+        +==============+==========================+
+        | ``abc``      | ``["--name", "abc"]``    |
+        +--------------+--------------------------+
+
+- ``cmd_runner_fmt.as_opt_eq_val()``
+    This method receives one parameter ``arg``, the function returns the string of the form
+    ``{arg}={value}``.
+
+    - Creation:
+        ``cmd_runner_fmt.as_opt_eq_val("--num-cpus")``
+    - Example:
+        +------------+-------------------------+
+        | Value      | Outcome                 |
+        +============+=========================+
+        | ``10``     | ``["--num-cpus=10"]``   |
+        +------------+-------------------------+
+
+- ``cmd_runner_fmt.as_fixed()``
+    This method receives one parameter ``arg``, the function expects no ``value`` - if one
+    is provided then it is ignored.
+    The function returns ``arg`` as-is.
+
+    - Creation:
+        ``cmd_runner_fmt.as_fixed("--version")``
+    - Example:
+        +---------+-----------------------+
+        | Value   | Outcome               |
+        +=========+=======================+
+        |         | ``["--version"]``     |
+        +---------+-----------------------+
+        | 57      | ``["--version"]``     |
+        +---------+-----------------------+
+
+    - Note:
+        This is the only special case in which a value can be missing for the formatting function.
+        The example also comes from the code in `Quickstart`_.
+        In that case, the module has code to determine the command's version so that it can assert compatibility.
+        There is no *value* to be passed for that CLI argument.
+
+- ``cmd_runner_fmt.as_map()``
+    This method receives one parameter ``arg`` which must be a dictionary, and an optional parameter ``default``.
+    The function returns the evaluation of ``arg[value]``.
+    If ``value not in arg``, then it returns ``default`` if defined, otherwise ``[]``.
+
+    - Creation:
+        ``cmd_runner_fmt.as_map(dict(a=1, b=2, c=3), default=42)``
+    - Example:
+        +---------------------+---------------+
+        | Value               | Outcome       |
+        +=====================+===============+
+        | ``"b"``             | ``["2"]``     |
+        +---------------------+---------------+
+        | ``"yabadabadoo"``   | ``["42"]``    |
+        +---------------------+---------------+
+
+    - Note:
+        If ``default`` is not specified, invalid values return an empty list, meaning they are silently ignored.
+
+- ``cmd_runner_fmt.as_func()``
+    This method receives one parameter ``arg`` which is itself is a format function and it must abide by the rules described above.
+
+    - Creation:
+        ``cmd_runner_fmt.as_func(lambda v: [] if v == 'stable' else ['--channel', '{0}'.format(v)])``
+    - Note:
+        The outcome for that depends entirely on the function provided by the developer.
+
+
+Other features for argument formatting
+""""""""""""""""""""""""""""""""""""""
+
+Some additional features are available as decorators:
+
+- ``cmd_runner_fmt.unpack args()``
+    This decorator unpacks the incoming ``value`` as a list of elements.
+
+    For example, in ``ansible_collections.community.general.plugins.module_utils.puppet``, it is used as:
+
+    .. code-block:: python
+
+          @cmd_runner_fmt.unpack_args
+          def execute_func(execute, manifest):
+              if execute:
+                  return ["--execute", execute]
+              else:
+                  return [manifest]
+
+          runner = CmdRunner(
+              module,
+              command=_prepare_base_cmd(),
+              path_prefix=_PUPPET_PATH_PREFIX,
+              arg_formats=dict(
+                  # ...
+                  _execute=cmd_runner_fmt.as_func(execute_func),
+                  # ...
+              ),
+          )
+
+    Then, in :ansplugin:`community.general.puppet#module` it is put to use with:
+
+    .. code-block:: python
+
+          with runner(args_order) as ctx:
+              rc, stdout, stderr = ctx.run(_execute=[p['execute'], p['manifest']])
+
+- ``cmd_runner_fmt.unpack_kwargs()``
+    Conversely, this decorator unpacks the incoming ``value`` as a ``dict``-like object.
+
+- ``cmd_runner_fmt.stack()``
+    This decorator assumes ``value`` is a sequence and concatenates the output
+    of the wrapped function applied to each element of the sequence.
+
+    For example, in :ansplugin:`community.general.django_check#module`, the argument format for ``database``
+    is defined as:
+
+    .. code-block:: python
+
+          arg_formats = dict(
+              # ...
+              database=cmd_runner_fmt.stack(cmd_runner_fmt.as_opt_val)("--database"),
+              # ...
+          )
+
+    When receiving a list ``["abc", "def"]``, the output is:
+
+    .. code-block:: python
+
+          ["--database", "abc", "--database", "def"]
+
+
+Command Runner
+^^^^^^^^^^^^^^
+
+Settings that can be passed to the ``CmdRunner`` constructor are:
+
+- ``module: AnsibleModule``
+    Module instance. Mandatory parameter.
+- ``command: str | list[str]``
+    Command to be executed. It can be a single string, the executable name, or a list
+    of strings containing the executable name as the first element and, optionally, fixed parameters.
+    Those parameters are used in all executions of the runner.
+- ``arg_formats: dict``
+    Mapping of argument names to formatting functions.
+- ``default_args_order: str``
+    As the name suggests, a default ordering for the arguments. When
+    this is passed, the context can be created without specifying ``args_order``. Defaults to ``()``.
+- ``check_rc: bool``
+    When ``True``, if the return code from the command is not zero, the module exits
+    with an error. Defaults to ``False``.
+- ``path_prefix: list[str]``
+    If the command being executed is installed in a non-standard directory path,
+    additional paths might be provided to search for the executable. Defaults to ``None``.
+- ``environ_update: dict``
+    Pass additional environment variables to be set during the command execution.
+    Defaults to ``None``.
+- ``force_lang: str``
+    It is usually important to force the locale to one specific value, so that responses are consistent and, therefore, parseable.
+    Please note that using this option (which is enabled by default) overwrites the environment variables ``LANGUAGE`` and ``LC_ALL``.
+    To disable this mechanism, set this parameter to ``None``.
+    In community.general 9.1.0 a special value ``auto`` was introduced for this parameter, with the effect
+    that ``CmdRunner`` then tries to determine the best parseable locale for the runtime.
+    It should become the default value in the future, but for the time being the default value is ``C``.
+
+When creating a context, the additional settings that can be passed to the call are:
+
+- ``args_order: str``
+    Establishes the order in which the arguments are rendered in the command line.
+    This parameter is mandatory unless ``default_args_order`` was provided to the runner instance.
+- ``output_process: func``
+    Function to transform the output of the executable into different values or formats.
+    See examples in section below.
+- ``check_mode_skip: bool``
+    Whether to skip the actual execution of the command when the module is in check mode.
+    Defaults to ``False``.
+- ``check_mode_return: any``
+    If ``check_mode_skip=True``, then return this value instead.
+
+Additionally, any other valid parameters for ``AnsibleModule.run_command()`` may be passed, but unexpected behavior
+might occur if redefining options already present in the runner or its context creation. Use with caution.
+
+
+Processing results
+^^^^^^^^^^^^^^^^^^
+
+As mentioned, ``CmdRunner`` uses ``AnsibleModule.run_command()`` to execute the external command,
+and it passes the return value from that method back to caller. That means that,
+by default, the result is going to be a tuple ``(rc, stdout, stderr)``.
+
+If you need to transform or process that output, you can pass a function to the context,
+as the ``output_process`` parameter. It must be a function like:
+
+.. code-block:: python
+
+    def process(rc, stdout, stderr):
+        # do some magic
+        return processed_value    # whatever that is
+
+In that case, the return of ``run()`` is the ``processed_value`` returned by the function.
+
+
+PythonRunner
+^^^^^^^^^^^^
+
+The ``PythonRunner`` class is a specialized version of ``CmdRunner``, geared towards the execution of
+Python scripts. It features two extra and  mutually exclusive parameters ``python`` and ``venv`` in its constructor:
+
+.. code-block:: python
+
+    from ansible_collections.community.general.plugins.module_utils.python_runner import PythonRunner
+    from ansible_collections.community.general.plugins.module_utils.cmd_runner import cmd_runner_fmt
+
+    runner = PythonRunner(
+        module,
+        command=["-m", "django"],
+        arg_formats=dict(...),
+        python="python",
+        venv="/path/to/some/venv",
+    )
+
+The default value for ``python`` is the string ``python``, and the for ``venv`` it is ``None``.
+
+The command line produced by such a command with ``python="python3.12"`` is something like:
+
+.. code-block:: shell
+
+    /usr/bin/python3.12 -m django <arg1> <arg2> ...
+
+And the command line for ``venv="/work/venv"`` is like:
+
+.. code-block:: shell
+
+    /work/venv/bin/python -m django <arg1> <arg2> ...
+
+You may provide the value of the ``command`` argument as a string (in that case the string is used as a script name)
+or as a list, in which case the elements of the list must be valid arguments for the Python interpreter, as in the example above.
+See `Command line and environment <https://docs.python.org/3/using/cmdline.html>`_ for more details.
+
+If the parameter ``python`` is an absolute path, or contains directory separators, such as ``/``, then it is used
+as-is, otherwise the runtime ``PATH`` is searched for that command name.
+
+Other than that, everything else works as in ``CmdRunner``.
+
+.. versionadded:: 4.8.0