Pkgng many packages one command (#3393)

* pkgng: join package list into one command

Change the pkgng module so all packages being
installed (or upgraded) are acted on in one
command (per action). This will make installs
and upgrades a bit faster, because pkg will be
invoked fewer times per module run. More important,
module actions will be more atomic, making it less
likely that some packages are acted on because they
appear earlier in the argument list.

This change also improves the status reporting of
packages acted on, specifying the number of packages
for each action (install or upgrade).

* pkgng: make upgrade check lazily evaluated

Make upgrade_available an inner function so that the
if statement that checks whether installed packages
are up-to-date only runs the upgrade check on packages
that are already installed. This gets lazily evaluated
because of boolean operator short-circuiting:
https://docs.python.org/3.8/library/stdtypes.html#boolean-operations-and-or-not

Previously, the module would always check for upgrades,
even for not-installed packages, when running with
`state=latest`.

* pkgng: add changelog fragment

* pkgng: Apply changelog suggestions from code review

Co-authored-by: Felix Fontein <felix@fontein.de>

* pkgng: resolve pep8 style issue

Remove inline function. It's purpose would be confusing for
future maintainers, and someone refactoring it to a variable,
with good intentions, would introduce a performance regression.

Including the `query_update()` call in the if expression makes
the intent more legible and still ensures lazy evaluation of the
function call if the first `and` is `False`.

* pkgng: Fix changelog fragment syntax issue

Need to escape quotes so YAML doesn't eat them

Co-authored-by: Felix Fontein <felix@fontein.de>

* pkgng: Improve output message English grammar

Make word "package" plural only if reporting on more than one package

Co-authored-by: Felix Fontein <felix@fontein.de>

Co-authored-by: Felix Fontein <felix@fontein.de>

changelogs/fragments/3393-pkgng-many_packages_one_command.yml

@@ -0,0 +1,5 @@
+minor_changes:
+  - pkgng - packages being installed (or upgraded) are acted on in one command (per action)
+    (https://github.com/ansible-collections/community.general/issues/2265).
+  - pkgng - status message specifies number of packages installed and/or upgraded separately.
+    Previously, all changes were reported as one count of packages "added" (https://github.com/ansible-collections/community.general/pull/3393).

plugins/modules/packaging/os/pkgng.py

@@ -134,6 +134,7 @@ EXAMPLES = '''
 '''
 
 
+from collections import defaultdict
 import re
 from ansible.module_utils.basic import AnsibleModule
 
@@ -226,7 +227,8 @@ def remove_packages(module, pkgng_path, packages, dir_arg):
 
 
 def install_packages(module, pkgng_path, packages, cached, pkgsite, dir_arg, state, ignoreosver):
-    install_c = 0
+    action_queue = defaultdict(list)
+    action_count = defaultdict(int)
     stdout = ""
     stderr = ""
 
@@ -263,29 +265,48 @@ def install_packages(module, pkgng_path, packages, cached, pkgsite, dir_arg, sta
         if already_installed and state == "present":
             continue
 
-        update_available = query_update(module, pkgng_path, package, dir_arg, old_pkgng, pkgsite)
+        if (
-        if not update_available and already_installed and state == "latest":
+            already_installed and state == "latest"
+            and not query_update(module, pkgng_path, package, dir_arg, old_pkgng, pkgsite)
+        ):
             continue
 
-        if not module.check_mode:
+        if already_installed:
-            if already_installed:
+            action_queue["upgrade"].append(package)
-                action = "upgrade"
+        else:
-            else:
+            action_queue["install"].append(package)
-                action = "install"
+
+    if not module.check_mode:
+        # install/upgrade all named packages with one pkg command
+        for (action, package_list) in action_queue.items():
+            packages = ' '.join(package_list)
             if old_pkgng:
-                rc, out, err = module.run_command("%s %s %s %s -g -U -y %s" % (batch_var, pkgsite, pkgng_path, action, package))
+                rc, out, err = module.run_command("%s %s %s %s -g -U -y %s" % (batch_var, pkgsite, pkgng_path, action, packages))
             else:
-                rc, out, err = module.run_command("%s %s %s %s %s -g -U -y %s" % (batch_var, pkgng_path, dir_arg, action, pkgsite, package))
+                rc, out, err = module.run_command("%s %s %s %s %s -g -U -y %s" % (batch_var, pkgng_path, dir_arg, action, pkgsite, packages))
             stdout += out
             stderr += err
 
-        if not module.check_mode and not query_package(module, pkgng_path, package, dir_arg):
+            # individually verify packages are in requested state
-            module.fail_json(msg="failed to %s %s: %s" % (action, package, out), stdout=stdout, stderr=stderr)
+            for package in package_list:
+                verified = False
+                if action == 'install':
+                    verified = query_package(module, pkgng_path, package, dir_arg)
+                elif action == 'upgrade':
+                    verified = not query_update(module, pkgng_path, package, dir_arg, old_pkgng, pkgsite)
 
-        install_c += 1
+                if verified:
+                    action_count[action] += 1
+                else:
+                    module.fail_json(msg="failed to %s %s" % (action, package), stdout=stdout, stderr=stderr)
 
-    if install_c > 0:
+    if sum(action_count.values()) > 0:
-        return (True, "added %s package(s)" % (install_c), stdout, stderr)
+        past_tense = {'install': 'installed', 'upgrade': 'upgraded'}
+        messages = []
+        for (action, count) in action_count.items():
+            messages.append("%s %s package%s" % (past_tense.get(action, action), count, "s" if count != 1 else ""))
+
+        return (True, '; '.join(messages), stdout, stderr)
 
     return (False, "package(s) already %s" % (state), stdout, stderr)