pkgutil: add update all, check-mode, squashing and examples (#799) (#1009)

* pkgutil: add update all, check-mode, squashing and examples

Taken from https://github.com/ansible/ansible/pull/51651 by dagwieers, which was taken from https://github.com/ansible/ansible/pull/27866 by scathatheworm.  Let’s have one last attempt to get this merged.

> ##### SUMMARY
>
> Original PR #27866 from scathatheworm
>
> When working with Solaris pkgutil CSW packages, I came across this module being very basic in functionality, in particular, that I could not use it to update all CSW packages.
>
> When going into details into the code I also found it did not incorporate a possibility of doing dry-run from the underlying utility, or supported to specify multiple packages for operations.
>
> This module probably sees very little use, but it seemed like nice functionality to add and make it behave a little more like other package modules.
> ##### ISSUE TYPE
>
>     * Feature Pull Request
>
>
> ##### COMPONENT NAME
>
> pkgutil module
> ##### ANSIBLE VERSION
>
> ```
> ansible 2.3.1.0
>   config file = /etc/ansible/ansible.cfg
>   configured module search path = Default w/o overrides
>   python version = 2.7.5 (default, Aug  2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
> ```
>
> ##### ADDITIONAL INFORMATION
>
>     * Added ability to upgrade all packages:
>
>
> ```yaml
> - pkgutil:
>     name: '*'
>     state: latest
> ```
>
>     * Added ability to modify state of a list of packages:
>
>
> ```yaml
> - pkgutil:
>     name:
>     - CSWtop
>     - CSWwget
>     - CSWlsof
>     state: present
> ```
>
>     * Added ability to have underlying tool perform a dry-run when using check mode, pkgutil -n
>
>     * Added ability to configure force option to force packages to state determined by repository (downgrade for example)
>
>
> ```yaml
> - pkgutil:
>     name: CSWtop
>     state: latest
>     force: yes
> ```
>
>     * Added more examples and documentation to show the new functionality

* Add changelog fragment.

* Observe changelog style guide

https://docs.ansible.com/ansible/devel/community/development_process.html#changelogs

Co-authored-by: Felix Fontein <felix@fontein.de>

* Since module split, version_added no-longer refers to core Ansbile

Co-authored-by: Felix Fontein <felix@fontein.de>

* Tweak documentation

* Apply the new `elements` feature for specifying list types

Co-authored-by: Felix Fontein <felix@fontein.de>

* Set version_added

Co-authored-by: Felix Fontein <felix@fontein.de>

* Document `pkg` alias for `name`

* Be explicit about the purpose of states `installed` and `removed`.

* Force the user to specify their desired state.

* Review documentation for pkgutil module.

* Fully qualify svr4pkg module name

Co-authored-by: Felix Fontein <felix@fontein.de>

Co-authored-by: Felix Fontein <felix@fontein.de>
(cherry picked from commit dd9e999c9f)

Co-authored-by: Peter Oliver <git@mavit.org.uk>

changelogs/fragments/pkgutil-check-mode-etc.yaml

@@ -0,0 +1,4 @@
+minor_changes:
+  - pkgutil - module now supports check mode (https://github.com/ansible-collections/community.general/pull/799).
+  - pkgutil - module can now accept a list of packages (https://github.com/ansible-collections/community.general/pull/799).
+  - pkgutil - module has a new option, ``force``, equivalent to the ``-f`` option to the `pkgutil <http://pkgutil.net/>`_ command (https://github.com/ansible-collections/community.general/pull/799).

plugins/modules/packaging/os/pkgutil.py

@@ -1,7 +1,7 @@
 #!/usr/bin/python
 # -*- coding: utf-8 -*-
 
-# (c) 2013, Alexander Winkler <mail () winkler-alexander.de>
+# Copyright: (c) 2013, Alexander Winkler <mail () winkler-alexander.de>
 # based on svr4pkg by
 #  Boyd Adamson <boyd () boydadamson.com> (2012)
 #
@@ -11,46 +11,55 @@ from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
 
-DOCUMENTATION = '''
+DOCUMENTATION = r'''
 ---
 module: pkgutil
-short_description: Manage CSW-Packages on Solaris
+short_description: OpenCSW package management on Solaris
 description:
-    - Manages CSW packages (SVR4 format) on Solaris 10 and 11.
+- This module installs, updates and removes packages from the OpenCSW project for Solaris.
-    - These were the native packages on Solaris <= 10 and are available
+- Unlike the M(community.general.svr4pkg) module, it will resolve and download dependencies.
-      as a legacy feature in Solaris 11.
+- See U(https://www.opencsw.org/) for more information about the project.
-    - Pkgutil is an advanced packaging system, which resolves dependency on installation.
+author:
-      It is designed for CSW packages.
+- Alexander Winkler (@dermute)
-author: "Alexander Winkler (@dermute)"
+- David Ponessa (@scathatheworm)
 options:
   name:
     description:
-      - Package name, e.g. (C(CSWnrpe))
+    - The name of the package.
+    - When using C(state=latest), this can be C('*'), which updates all installed packages managed by pkgutil.
+    type: list
     required: true
-    type: str
+    elements: str
+    aliases: [ pkg ]
   site:
     description:
-      - Specifies the repository path to install the package from.
+    - The repository path to install the package from.
-      - Its global definition is done in C(/etc/opt/csw/pkgutil.conf).
+    - Its global definition is in C(/etc/opt/csw/pkgutil.conf).
     required: false
     type: str
   state:
     description:
-      - Whether to install (C(present)), or remove (C(absent)) a package.
+    - Whether to install (C(present)/C(installed)), or remove (C(absent)/C(removed)) packages.
-      - The upgrade (C(latest)) operation will update/install the package to the latest version available.
+    - The upgrade (C(latest)) operation will update/install the packages to the latest version available.
-      - "Note: The module has a limitation that (C(latest)) only works for one package, not lists of them."
-    required: true
-    choices: ["present", "absent", "latest"]
     type: str
+    required: true
+    choices: [ absent, installed, latest, present, removed ]
   update_catalog:
     description:
-      - If you want to refresh your catalog from the mirror, set this to (C(yes)).
+    - If you always want to refresh your catalog from the mirror, even when it's not stale, set this to C(yes).
-    required: false
-    default: no
     type: bool
+    default: no
+  force:
+    description:
+    - To allow the update process to downgrade packages to match what is present in the repository, set this to C(yes).
+    - This is useful for rolling back to stable from testing, or similar operations.
+    type: bool
+    version_added: 1.2.0
+notes:
+- In order to check the availability of packages, the catalog cache under C(/var/opt/csw/pkgutil) may be refreshed even in check mode.
 '''
 
-EXAMPLES = '''
+EXAMPLES = r'''
 - name: Install a package
   community.general.pkgutil:
     name: CSWcommon
@@ -59,35 +68,80 @@ EXAMPLES = '''
 - name: Install a package from a specific repository
   community.general.pkgutil:
     name: CSWnrpe
-    site: 'ftp://myinternal.repo/opencsw/kiel'
+    site: ftp://myinternal.repo/opencsw/kiel
     state: latest
+
+- name: Remove a package
+  community.general.pkgutil:
+    name: CSWtop
+    state: absent
+
+- name: Install several packages
+  community.general.pkgutil:
+    name:
+    - CSWsudo
+    - CSWtop
+    state: present
+
+- name: Update all packages
+  community.general.pkgutil:
+    name: '*'
+    state: latest
+
+- name: Update all packages and force versions to match latest in catalog
+  community.general.pkgutil:
+    name: '*'
+    state: latest
+    force: yes
 '''
 
+RETURN = r''' # '''
+
 from ansible.module_utils.basic import AnsibleModule
 
 
-def package_installed(module, name):
+def packages_not_installed(module, names):
-    cmd = ['pkginfo']
+    ''' Check if each package is installed and return list of the ones absent '''
-    cmd.append('-q')
+    pkgs = []
-    cmd.append(name)
+    for pkg in names:
-    rc, out, err = run_command(module, cmd)
+        rc, out, err = run_command(module, ['pkginfo', '-q', pkg])
-    if rc == 0:
+        if rc != 0:
-        return True
+            pkgs.append(pkg)
-    else:
+    return pkgs
-        return False
 
 
-def package_latest(module, name, site):
+def packages_installed(module, names):
-    # Only supports one package
+    ''' Check if each package is installed and return list of the ones present '''
-    cmd = ['pkgutil', '-U', '--single', '-c']
+    pkgs = []
+    for pkg in names:
+        if not pkg.startswith('CSW'):
+            continue
+        rc, out, err = run_command(module, ['pkginfo', '-q', pkg])
+        if rc == 0:
+            pkgs.append(pkg)
+    return pkgs
+
+
+def packages_not_latest(module, names, site, update_catalog):
+    ''' Check status of each package and return list of the ones with an upgrade available '''
+    cmd = ['pkgutil']
+    if update_catalog:
+        cmd.append('-U')
+    cmd.append('-c')
     if site is not None:
-        cmd += ['-t', site]
+        cmd.extend('-t', site)
-    cmd.append(name)
+    if names != ['*']:
+        cmd.extend(names)
     rc, out, err = run_command(module, cmd)
-    # replace | tail -1 |grep -v SAME
+
-    # use -2, because splitting on \n create a empty line
+    # Find packages in the catalog which are not up to date
-    # at the end of the list
+    packages = []
-    return 'SAME' in out.split('\n')[-2]
+    for line in out.split('\n')[1:-1]:
+        if 'catalog' not in line and 'SAME' not in line:
+            packages.append(line.split(' ')[0])
+
+    # Remove duplicates
+    return list(set(packages))
 
 
 def run_command(module, cmd, **kwargs):
@@ -96,117 +150,129 @@ def run_command(module, cmd, **kwargs):
     return module.run_command(cmd, **kwargs)
 
 
-def package_install(module, state, name, site, update_catalog):
+def package_install(module, state, pkgs, site, update_catalog, force):
-    cmd = ['pkgutil', '-iy']
+    cmd = ['pkgutil']
+    if module.check_mode:
+        cmd.append('-n')
+    cmd.append('-iy')
     if update_catalog:
-        cmd += ['-U']
+        cmd.append('-U')
     if site is not None:
-        cmd += ['-t', site]
+        cmd.extend('-t', site)
-    if state == 'latest':
+    if force:
-        cmd += ['-f']
+        cmd.append('-f')
-    cmd.append(name)
+    cmd.extend(pkgs)
-    (rc, out, err) = run_command(module, cmd)
+    return run_command(module, cmd)
-    return (rc, out, err)
 
 
-def package_upgrade(module, name, site, update_catalog):
+def package_upgrade(module, pkgs, site, update_catalog, force):
-    cmd = ['pkgutil', '-ufy']
+    cmd = ['pkgutil']
+    if module.check_mode:
+        cmd.append('-n')
+    cmd.append('-uy')
     if update_catalog:
-        cmd += ['-U']
+        cmd.append('-U')
     if site is not None:
-        cmd += ['-t', site]
+        cmd.extend('-t', site)
-    cmd.append(name)
+    if force:
-    (rc, out, err) = run_command(module, cmd)
+        cmd.append('-f')
-    return (rc, out, err)
+    cmd += pkgs
+    return run_command(module, cmd)
 
 
-def package_uninstall(module, name):
+def package_uninstall(module, pkgs):
-    cmd = ['pkgutil', '-ry', name]
+    cmd = ['pkgutil']
-    (rc, out, err) = run_command(module, cmd)
+    if module.check_mode:
-    return (rc, out, err)
+        cmd.append('-n')
+    cmd.append('-ry')
+    cmd.extend(pkgs)
+    return run_command(module, cmd)
 
 
 def main():
     module = AnsibleModule(
         argument_spec=dict(
-            name=dict(required=True),
+            name=dict(type='list', elements='str', required=True, aliases=['pkg']),
-            state=dict(required=True, choices=['present', 'absent', 'latest']),
+            state=dict(type='str', required=True, choices=['absent', 'installed', 'latest', 'present', 'removed']),
-            site=dict(default=None),
+            site=dict(type='str'),
-            update_catalog=dict(required=False, default=False, type='bool'),
+            update_catalog=dict(type='bool', default=False),
+            force=dict(type='bool', default=False),
         ),
-        supports_check_mode=True
+        supports_check_mode=True,
     )
     name = module.params['name']
     state = module.params['state']
     site = module.params['site']
     update_catalog = module.params['update_catalog']
+    force = module.params['force']
+
     rc = None
     out = ''
     err = ''
-    result = {}
+    result = dict(
-    result['name'] = name
+        name=name,
-    result['state'] = state
+        state=state,
+    )
 
-    if state == 'present':
+    if state in ['installed', 'present']:
-        if not package_installed(module, name):
+        # Fail with an explicit error when trying to "install" '*'
-            if module.check_mode:
+        if name == ['*']:
-                module.exit_json(changed=True)
+            module.fail_json(msg="Can not use 'state: present' with name: '*'")
-            (rc, out, err) = package_install(module, state, name, site, update_catalog)
+
-            # Stdout is normally empty but for some packages can be
+        # Build list of packages that are actually not installed from the ones requested
-            # very long and is not often useful
+        pkgs = packages_not_installed(module, name)
-            if len(out) > 75:
+
-                out = out[:75] + '...'
+        # If the package list is empty then all packages are already present
+        if pkgs == []:
+            module.exit_json(changed=False)
+
+        (rc, out, err) = package_install(module, state, pkgs, site, update_catalog, force)
+        if rc != 0:
+            module.fail_json(msg=(err or out))
+
+    elif state in ['latest']:
+        # When using latest for *
+        if name == ['*']:
+            # Check for packages that are actually outdated
+            pkgs = packages_not_latest(module, name, site, update_catalog)
+
+            # If the package list comes up empty, everything is already up to date
+            if pkgs == []:
+                module.exit_json(changed=False)
+
+            # If there are packages to update, just empty the list and run the command without it
+            # pkgutil logic is to update all when run without packages names
+            pkgs = []
+            (rc, out, err) = package_upgrade(module, pkgs, site, update_catalog, force)
             if rc != 0:
-                if err:
+                module.fail_json(msg=(err or out))
-                    msg = err
-                else:
-                    msg = out
-                module.fail_json(msg=msg)
-
-    elif state == 'latest':
-        if not package_installed(module, name):
-            if module.check_mode:
-                module.exit_json(changed=True)
-            (rc, out, err) = package_install(module, state, name, site, update_catalog)
-            if len(out) > 75:
-                out = out[:75] + '...'
-            if rc != 0:
-                if err:
-                    msg = err
-                else:
-                    msg = out
-                module.fail_json(msg=msg)
-
         else:
-            if not package_latest(module, name, site):
+            # Build list of packages that are either outdated or not installed
-                if module.check_mode:
+            pkgs = packages_not_installed(module, name)
-                    module.exit_json(changed=True)
+            pkgs += packages_not_latest(module, name, site, update_catalog)
-                (rc, out, err) = package_upgrade(module, name, site, update_catalog)
-                if len(out) > 75:
-                    out = out[:75] + '...'
-                if rc != 0:
-                    if err:
-                        msg = err
-                    else:
-                        msg = out
-                    module.fail_json(msg=msg)
 
-    elif state == 'absent':
+            # If the package list is empty that means all packages are installed and up to date
-        if package_installed(module, name):
+            if pkgs == []:
-            if module.check_mode:
+                module.exit_json(changed=False)
-                module.exit_json(changed=True)
+
-            (rc, out, err) = package_uninstall(module, name)
+            (rc, out, err) = package_upgrade(module, pkgs, site, update_catalog, force)
-            if len(out) > 75:
-                out = out[:75] + '...'
             if rc != 0:
-                if err:
+                module.fail_json(msg=(err or out))
-                    msg = err
+
-                else:
+    elif state in ['absent', 'removed']:
-                    msg = out
+        # Build list of packages requested for removal that are actually present
-                module.fail_json(msg=msg)
+        pkgs = packages_installed(module, name)
+
+        # If the list is empty, no packages need to be removed
+        if pkgs == []:
+            module.exit_json(changed=False)
+
+        (rc, out, err) = package_uninstall(module, pkgs)
+        if rc != 0:
+            module.fail_json(msg=(err or out))
 
     if rc is None:
-        # pkgutil was not executed because the package was already present/absent
+        # pkgutil was not executed because the package was already present/absent/up to date
         result['changed'] = False
     elif rc == 0:
         result['changed'] = True

tests/integration/targets/pkgutil/aliases

@@ -0,0 +1,2 @@
+destructive
+unsupported

tests/integration/targets/pkgutil/tasks/main.yml

@@ -0,0 +1,116 @@
+# Test code for the pkgutil module
+
+# Copyright: (c) 2019, Dag Wieers (@dagwieers) <dag@wieers.com>
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
+
+
+# CLEAN ENVIRONMENT
+- name: Remove CSWtop
+  pkgutil:
+    name: CSWtop
+    state: absent
+  register: originally_installed
+
+
+# ADD PACKAGE
+- name: Add package (check_mode)
+  pkgutil:
+    name: CSWtop
+    state: present
+  check_mode: yes
+  register: cm_add_package
+
+- name: Verify cm_add_package
+  assert:
+    that:
+    - cm_add_package is changed
+
+- name: Add package (normal mode)
+  pkgutil:
+    name: CSWtop
+    state: present
+  register: nm_add_package
+
+- name: Verify nm_add_package
+  assert:
+    that:
+    - nm_add_package is changed
+
+- name: Add package again (check_mode)
+  pkgutil:
+    name: CSWtop
+    state: present
+  check_mode: yes
+  register: cm_add_package_again
+
+- name: Verify cm_add_package_again
+  assert:
+    that:
+    - cm_add_package_again is not changed
+
+- name: Add package again (normal mode)
+  pkgutil:
+    name: CSWtop
+    state: present
+  register: nm_add_package_again
+
+- name: Verify nm_add_package_again
+  assert:
+    that:
+    - nm_add_package_again is not changed
+
+
+# REMOVE PACKAGE
+- name: Remove package (check_mode)
+  pkgutil:
+    name: CSWtop
+    state: absent
+  check_mode: yes
+  register: cm_remove_package
+
+- name: Verify cm_remove_package
+  assert:
+    that:
+    - cm_remove_package is changed
+
+- name: Remove package (normal mode)
+  pkgutil:
+    name: CSWtop
+    state: absent
+  register: nm_remove_package
+
+- name: Verify nm_remove_package
+  assert:
+    that:
+    - nm_remove_package is changed
+
+- name: Remove package again (check_mode)
+  pkgutil:
+    name: CSWtop
+    state: absent
+  check_mode: yes
+  register: cm_remove_package_again
+
+- name: Verify cm_remove_package_again
+  assert:
+    that:
+    - cm_remove_package_again is not changed
+
+- name: Remove package again (normal mode)
+  pkgutil:
+    name: CSWtop
+    state: absent
+  register: nm_remove_package_again
+
+- name: Verify nm_remove_package_again
+  assert:
+    that:
+    - nm_remove_package_again is not changed
+
+
+# RESTORE ENVIRONMENT
+- name: Reinstall CSWtop
+  pkgutil:
+    name: CSWtop
+    state: present
+  when: originally_installed is changed