pkgng: add basic integration tests (#3526)

* pkgng: package reinstallations count as changed

`upgrade_packages()` only looked for the string
"Number of packages to be upgraded", but the
`pkg upgrade` command also reports "Number of packages to be
reinstalled". Reinstallation occurs when package metadata other
than version changes (e.g. build options, single architecture to `*`
architecture). In any other respect, though, a required
reinstallation is the same as an upgrade.

* pkgng: check_mode should count queued actions

Writing tests caught a bug in PR #3393, which enabled
installing more than one package per `pkg` execution.

In converting the module's install/upgrade code to a
queue structure, check_mode got broken because the count
of actions performed was only updated in the `if not check_mode`
block that invokes `pkg`. This two-line change counts
the number of actions in the queue when check mode is
enabled.

* pkgng: add basic integration tests

Test installing a package
Test installing a package into a jail

* pkgng: test with zsh not bash package

Had been using bash package, because it's not
likely to disappear from the package repository
any time soon. Turns out that bash is already
installed on the Ansible community.general
FreeBSD CI VM image, which makes the test fail.

Zsh probably isn't going away any time soon,
either, and likely won't be installed on the
CI image.

* pkgng: remove redundant test for docker container

These tests should be skip/docker, but the test
playbook also redundantly checked whether it was
running in a docker container.

Checking whether `ansible_facts.distribution` is
`FreeBSD` is really sufficient to be sure whether
the test is running in an environment that supports
the `pkgng` module.

* pkgng: add state=absent test

Also renumber `jail=...` test to make diffs less
noisy when adding more non-jailed tests.

* pkgng: add state=latest idemptency test

Add test between state=present and
state=absent that ensures state=latest runs
successfully on an already up-to-date package
with changed=false.

* pkgng: add state=latest test

Including small filesize out-of-date package from
FreeBSD 11 package repository, because FreeBSD 11 is
currently EOL. This test might fail at some point in the
future if the pkg utility in a FreeBSD version > 14
makes breaking changes that prevents it from installing
older package formats.

If that occurs, the fix is to replace
`files/freebsd-release-manifests-20210413__FreeBSD_11.pkg`
with the version of freebsd-release-manifests from the oldest
non-breaking release of FreeBSD, and update the
references to it in tasks/main.yml accordingly.

* pkgng: use hand-generated test package

Instead of relying on a broken-ish installation
of an older package from FreeBSD 11, hand-generate
a package named `zsh` with no contents and a version
number of `0`. It can be installed on any architecture
and any revision of FreeBSD sucessfully, and it will
always be eligible for upgrade.

* pkgng: CI seems to dislike plus-signs in filenames

* pkgng: refactor renaming test package manifests

refactored to make intent of adding '+' sign clearer

* pkgng: refactor upgraded pattern matching

Implement russoz's suggestion to put all
variants in the pattern.

* pkgng: add cached=yes test

Changed idempotency test, because this covers both.

* pkgng: test pkg works on any FreeBSD version

Removing `ansible_distribution_version`, because the
test out-of-date zsh package should install on any
FreeBSD version.

* pkgng: move FreeBSD tasks to imported file

Refactoring tests for more reuse and easier
readability.

* pkgng: refactor tests for task reuse

Several tests need to install an out-of-date package
to test whether upgrades occur successfully, so this
commit refactors out the generation of the test package.

Also, factor out things like the name and path of the
test package so that is more maintainable should the
target test package (currently `zsh`) ever need to
change.

* pkgng: test install multiple packages

Multiple packages should install with a single
invocation of the `pkg` utility.

* pkgng: handle space- and comma-separated lists

The module expects a list of strings in the `name` parameter,
but long-standing documentation showed space- and comma-delimited
lists as a valid way of telling the module to act on multiple
packages. Passing these lists through to the `pkg` command can
have unexpected side-effects of upgrading packages when
`state=present` rather than `state=latest` and could result
in the module reporting the wrong number of packages for each
action performed.

* pkgng: test state=latest for mixed install/upgrade

Test that a list of packages given, one not installed
and one installed but out-of-date are handled correctly.

* pkgng: use YAML lists for multiple packages

I had been following the documentation examples, and
specifiying multiple packages on one line. The right way
to do it is with YAML list syntax.

* pkgng: add test for autoremove=yes

Install package with known dependencies, then
remove it and ask autoremove to run.

* pkgng: test autoremove=yes only on FreeBSD > 12

The CI test runner on FreeBSD 12.0 times out when
downloading large packages.

* pkgng: test jail creation logging less verbose

`ezjail-admin` had been spewing every filename copied
into the jail onto stderr, making the `ansible-test -v`
logs hard to scroll through. Changed it so that ezjail
output only shows up in the ansible task logs if
`ansible_verbosity > 1` (`-vv...`). Full `ezjail-admin`
output is always logged to `/tmp/ezjail.log`.

* pkgng: pass tests when package dependencies change

Tests that install packages with dependencies were failing
when those dependencies were not already installed, because
the count of installed packages was greater than the count
of requested to be installed packages. This change checks
for a count of installed packages that is greater than or
equal to the count of requested to be installed packages.

Example:
  - Test installs packages `zsh` and `fish`
  - `fish` has a dependency on `pcre2`
  - `pkg` reports `Number of packages to be installed: 3`

* pkgng: test annotation for single package

Add/modify/remove annotation for a single package

* pkgng: fix annotation operations

Annotation has been broken at least since the migration to
collections. There are some breaking typos and function argument
omissions in the code with nothing in `git blame` but "Initial commit".

New integration tests uncovered this breakage. Works now.

* pkgng: test multiple annotations

Test multiple annotations specified both in new (YAML list)
syntax and old (comma-delimited) syntax.

Rename some annotation test tags from the single-annotation
tests to make sure that single and multiple annotation tests
don't collide.

* pkgng: test invalid annotate strings

Ensure that some likely to occur, invalid strings
given to the annotate module parameter 1. fail,
2. do not cause an exception, and 3. do not create
any actual annotations on packages.

* pkgng: fix check_mode for annotate

Actions specified in the `annotate` parameter would
always be performed, even if `check_mode=yes`.

This commit fixes `check_mode` for the annotation
functions and adds integration tests to ensure that
check mode is honored in the future.

* pkgng: call module.run_command with list

Instead of calling `run_command` with a formatted
string, call it with a list to maintain safer argument
separation.

Also, introduce a wrapper for `run_command`, `run_pkgng`,
which manages the process environment and some common
command line switches, based upon the module parameters.

Introduced in this commit, also pass annotation values
to `pkg annotate` via stdin, which is safer with long
values than putting them in argv.

* pkgng: update documentation to match annotate type

Missed updating the documentation to match the change
of the annotate parameter from comma-separated string
to list of strings.

* pkgng: fix syntax for Python 2

Seems *args, **kwargs have to be the last
two arguments in Python 2 syntax. CI sanity
tests were failing.

* pkgng: oops, pkgsite should be from closure

I put `pkgsite` as a keyword argument to `run_pkgng()',
which wasn't correct at all and resulted it in capturing
the second positional argument given by any caller.

`pkgsite` should have been `p["pkgsite"]`, coming from the
closure environment of `main()`.

* pkgng: Fix changelog fragment quote formatting

Co-authored-by: Felix Fontein <felix@fontein.de>

* pkgng: add test prepare task

Make sure test environment does not contain the
test package.

* pkgng: make integration test package more flexible

Make integration test package building template-based,
so more flexible if in the future the name of the test package,
currently `zsh`, must change.

* pkgng: convert generator to list for 2.9

Ansible 2.9 can't count items from a generator,
so convert `select` filter output via `list`
before passing to `count`.

* pkgng: Ansible 2.9 has no "false" Jinja2 test

Apparently Ansible 2.9 doesn't have the "false" test
in Jinja2 contexts. Switching to use `rejectattr(...)`
instead of `selectattr(..., "false")`.

* pkgng: test pkgsite parameter

* pkgng: add tests for chroot and rootdir params

Refactor install tasks into an include file so they
can be reused and tested with various parameters.

* pkgng: clean up test package creation

Make pkg json manifest more readable. Create package using
FreeBSD's `pkg create` instead of manually using tar. This
change also simplifies the manifest to a single file for
the integration test role.

* pkgng: fix #3428; install from local file

Fixes a bug in which the module reported failure
when installing from a local (to the target host)
file path.

Fixes #3428

* pkgng: changelog fragment formatting issue

* pkgng: check for all test package extensions

pkg recently changed file extensions, so for the
tests to work on multiple recent versions of FreeBSD,
we must check for `pkg create` output with varying
filename.

* Revert "pkgng: fix #3428; install from local file"

As the module has invoked pkg with the `-g` flag for
at least 7 years, I'm not sure when it was possible
to install packages from files on the target host
filesystem, because pkg rejects file paths when
the `--glob` flag is enabled.

I considered doing a rework of the pkg invocation
to enable good support for installing from local
files, but it looks like more of a job than for this PR.

This reverts commit 5f94eac41f.

* pkgng: Add minor_changes fragment for #3526

Co-authored-by: Felix Fontein <felix@fontein.de>

Co-authored-by: Felix Fontein <felix@fontein.de>

changelogs/fragments/3526-pkgng-add-integration-tests.yml

@@ -0,0 +1,6 @@
+bugfixes:
+  - 'pkgng - ``name=* state=latest`` check for upgrades did not count "Number of packages to be reinstalled" as a `changed` action, giving incorrect results in both regular and check mode (https://github.com/ansible-collections/community.general/pull/3526).'
+  - 'pkgng - an `earlier PR <https://github.com/ansible-collections/community.general/pull/3393>`_ broke check mode so that the module always reports `not changed`. This is now fixed so that the module reports number of upgrade or install actions that would be performed (https://github.com/ansible-collections/community.general/pull/3526).'
+  - 'pkgng - the ``annotation`` functionality was broken and is now fixed, and now also works with check mode (https://github.com/ansible-collections/community.general/pull/3526).'
+minor_changes:
+  - 'pkgng - ``annotation`` can now also be a YAML list (https://github.com/ansible-collections/community.general/pull/3526).'

plugins/modules/packaging/os/pkgng.py

@@ -50,13 +50,14 @@ options:
         default: no
     annotation:
         description:
-            - A comma-separated list of keyvalue-pairs of the form
+            - A list of keyvalue-pairs of the form
               C(<+/-/:><key>[=<value>]). A C(+) denotes adding an annotation, a
               C(-) denotes removing an annotation, and C(:) denotes modifying an
               annotation.
               If setting or modifying annotations, a value must be provided.
         required: false
-        type: str
+        type: list
+        elements: str
     pkgsite:
         description:
             - For pkgng versions before 1.1.4, specify packagesite to use
@@ -113,12 +114,16 @@ EXAMPLES = '''
 
 - name: Annotate package foo and bar
   community.general.pkgng:
-    name: foo,bar
+    name:
+      - foo
+      - bar
     annotation: '+test1=baz,-test2,:test3=foobar'
 
 - name: Remove packages foo and bar
   community.general.pkgng:
-    name: foo,bar
+    name:
+      - foo
+      - bar
     state: absent
 
 # "latest" support added in 2.7
@@ -139,9 +144,9 @@ import re
 from ansible.module_utils.basic import AnsibleModule
 
 
-def query_package(module, pkgng_path, name, dir_arg):
+def query_package(module, run_pkgng, name):
 
-    rc, out, err = module.run_command("%s %s info -g -e %s" % (pkgng_path, dir_arg, name))
+    rc, out, err = run_pkgng('info', '-g', '-e', name)
 
     if rc == 0:
         return True
@@ -149,15 +154,12 @@ def query_package(module, pkgng_path, name, dir_arg):
     return False
 
 
-def query_update(module, pkgng_path, name, dir_arg, old_pkgng, pkgsite):
+def query_update(module, run_pkgng, name):
 
     # Check to see if a package upgrade is available.
     # rc = 0, no updates available or package not installed
     # rc = 1, updates available
-    if old_pkgng:
+    rc, out, err = run_pkgng('upgrade', '-g', '-n', name)
-        rc, out, err = module.run_command("%s %s upgrade -g -n %s" % (pkgsite, pkgng_path, name))
-    else:
-        rc, out, err = module.run_command("%s %s upgrade %s -g -n %s" % (pkgng_path, dir_arg, pkgsite, name))
 
     if rc == 1:
         return True
@@ -167,7 +169,7 @@ def query_update(module, pkgng_path, name, dir_arg, old_pkgng, pkgsite):
 
 def pkgng_older_than(module, pkgng_path, compare_version):
 
-    rc, out, err = module.run_command("%s -v" % pkgng_path)
+    rc, out, err = module.run_command([pkgng_path, '-v'])
     version = [int(x) for x in re.split(r'[\._]', out)]
 
     i = 0
@@ -182,40 +184,39 @@ def pkgng_older_than(module, pkgng_path, compare_version):
     return not new_pkgng
 
 
-def upgrade_packages(module, pkgng_path, dir_arg):
+def upgrade_packages(module, run_pkgng):
     # Run a 'pkg upgrade', updating all packages.
     upgraded_c = 0
 
-    cmd = "%s %s upgrade -y" % (pkgng_path, dir_arg)
+    pkgng_args = ['upgrade']
-    if module.check_mode:
+    pkgng_args.append('-n' if module.check_mode else '-y')
-        cmd += " -n"
+    rc, out, err = run_pkgng(*pkgng_args)
-    rc, out, err = module.run_command(cmd)
 
-    match = re.search('^Number of packages to be upgraded: ([0-9]+)', out, re.MULTILINE)
+    matches = re.findall('^Number of packages to be (?:upgraded|reinstalled): ([0-9]+)', out, re.MULTILINE)
-    if match:
+    for match in matches:
-        upgraded_c = int(match.group(1))
+        upgraded_c += int(match)
 
     if upgraded_c > 0:
         return (True, "updated %s package(s)" % upgraded_c, out, err)
     return (False, "no packages need upgrades", out, err)
 
 
-def remove_packages(module, pkgng_path, packages, dir_arg):
+def remove_packages(module, run_pkgng, packages):
     remove_c = 0
     stdout = ""
     stderr = ""
     # Using a for loop in case of error, we can report the package that failed
     for package in packages:
         # Query the package first, to see if we even need to remove
-        if not query_package(module, pkgng_path, package, dir_arg):
+        if not query_package(module, run_pkgng, package):
             continue
 
         if not module.check_mode:
-            rc, out, err = module.run_command("%s %s delete -y %s" % (pkgng_path, dir_arg, package))
+            rc, out, err = run_pkgng('delete', '-y', package)
             stdout += out
             stderr += err
 
-        if not module.check_mode and query_package(module, pkgng_path, package, dir_arg):
+        if not module.check_mode and query_package(module, run_pkgng, package):
             module.fail_json(msg="failed to remove %s: %s" % (package, out), stdout=stdout, stderr=stderr)
 
         remove_c += 1
@@ -226,48 +227,27 @@ def remove_packages(module, pkgng_path, packages, dir_arg):
     return (False, "package(s) already absent", stdout, stderr)
 
 
-def install_packages(module, pkgng_path, packages, cached, pkgsite, dir_arg, state, ignoreosver):
+def install_packages(module, run_pkgng, packages, cached, state):
     action_queue = defaultdict(list)
     action_count = defaultdict(int)
     stdout = ""
     stderr = ""
 
-    # as of pkg-1.1.4, PACKAGESITE is deprecated in favor of repository definitions
-    # in /usr/local/etc/pkg/repos
-    old_pkgng = pkgng_older_than(module, pkgng_path, [1, 1, 4])
-    if pkgsite != "":
-        if old_pkgng:
-            pkgsite = "PACKAGESITE=%s" % (pkgsite)
-        else:
-            pkgsite = "-r %s" % (pkgsite)
-
-    # This environment variable skips mid-install prompts,
-    # setting them to their default values.
-    batch_var = 'env BATCH=yes'
-
-    if ignoreosver:
-        # Ignore FreeBSD OS version check,
-        #   useful on -STABLE and -CURRENT branches.
-        batch_var = batch_var + ' IGNORE_OSVERSION=yes'
-
     if not module.check_mode and not cached:
-        if old_pkgng:
+        rc, out, err = run_pkgng('update')
-            rc, out, err = module.run_command("%s %s update" % (pkgsite, pkgng_path))
-        else:
-            rc, out, err = module.run_command("%s %s %s update" % (batch_var, pkgng_path, dir_arg))
         stdout += out
         stderr += err
         if rc != 0:
             module.fail_json(msg="Could not update catalogue [%d]: %s %s" % (rc, out, err), stdout=stdout, stderr=stderr)
 
     for package in packages:
-        already_installed = query_package(module, pkgng_path, package, dir_arg)
+        already_installed = query_package(module, run_pkgng, package)
         if already_installed and state == "present":
             continue
 
         if (
             already_installed and state == "latest"
-            and not query_update(module, pkgng_path, package, dir_arg, old_pkgng, pkgsite)
+            and not query_update(module, run_pkgng, package)
         ):
             continue
 
@@ -276,29 +256,32 @@ def install_packages(module, pkgng_path, packages, cached, pkgsite, dir_arg, sta
         else:
             action_queue["install"].append(package)
 
-    if not module.check_mode:
+    # install/upgrade all named packages with one pkg command
-        # install/upgrade all named packages with one pkg command
+    for (action, package_list) in action_queue.items():
-        for (action, package_list) in action_queue.items():
+        if module.check_mode:
-            packages = ' '.join(package_list)
+            # Do nothing, but count up how many actions
-            if old_pkgng:
+            # would be performed so that the changed/msg
-                rc, out, err = module.run_command("%s %s %s %s -g -U -y %s" % (batch_var, pkgsite, pkgng_path, action, packages))
+            # is correct.
+            action_count[action] += len(package_list)
+            continue
+
+        pkgng_args = [action, '-g', '-U', '-y'] + package_list
+        rc, out, err = run_pkgng(*pkgng_args)
+        stdout += out
+        stderr += err
+
+        # individually verify packages are in requested state
+        for package in package_list:
+            verified = False
+            if action == 'install':
+                verified = query_package(module, run_pkgng, package)
+            elif action == 'upgrade':
+                verified = not query_update(module, run_pkgng, package)
+
+            if verified:
+                action_count[action] += 1
             else:
-                rc, out, err = module.run_command("%s %s %s %s %s -g -U -y %s" % (batch_var, pkgng_path, dir_arg, action, pkgsite, packages))
+                module.fail_json(msg="failed to %s %s" % (action, package), stdout=stdout, stderr=stderr)
-            stdout += out
-            stderr += err
-
-            # individually verify packages are in requested state
-            for package in package_list:
-                verified = False
-                if action == 'install':
-                    verified = query_package(module, pkgng_path, package, dir_arg)
-                elif action == 'upgrade':
-                    verified = not query_update(module, pkgng_path, package, dir_arg, old_pkgng, pkgsite)
-
-                if verified:
-                    action_count[action] += 1
-                else:
-                    module.fail_json(msg="failed to %s %s" % (action, package), stdout=stdout, stderr=stderr)
 
     if sum(action_count.values()) > 0:
         past_tense = {'install': 'installed', 'upgrade': 'upgraded'}
@@ -311,28 +294,28 @@ def install_packages(module, pkgng_path, packages, cached, pkgsite, dir_arg, sta
     return (False, "package(s) already %s" % (state), stdout, stderr)
 
 
-def annotation_query(module, pkgng_path, package, tag, dir_arg):
+def annotation_query(module, run_pkgng, package, tag):
-    rc, out, err = module.run_command("%s %s info -g -A %s" % (pkgng_path, dir_arg, package))
+    rc, out, err = run_pkgng('info', '-g', '-A', package)
     match = re.search(r'^\s*(?P<tag>%s)\s*:\s*(?P<value>\w+)' % tag, out, flags=re.MULTILINE)
     if match:
         return match.group('value')
     return False
 
 
-def annotation_add(module, pkgng_path, package, tag, value, dir_arg):
+def annotation_add(module, run_pkgng, package, tag, value):
-    _value = annotation_query(module, pkgng_path, package, tag, dir_arg)
+    _value = annotation_query(module, run_pkgng, package, tag)
     if not _value:
         # Annotation does not exist, add it.
-        rc, out, err = module.run_command('%s %s annotate -y -A %s %s "%s"'
+        if not module.check_mode:
-                                          % (pkgng_path, dir_arg, package, tag, value))
+            rc, out, err = run_pkgng('annotate', '-y', '-A', package, tag, data=value, binary_data=True)
-        if rc != 0:
+            if rc != 0:
-            module.fail_json(msg="could not annotate %s: %s"
+                module.fail_json(msg="could not annotate %s: %s"
-                             % (package, out), stderr=err)
+                                 % (package, out), stderr=err)
         return True
     elif _value != value:
         # Annotation exists, but value differs
         module.fail_json(
-            mgs="failed to annotate %s, because %s is already set to %s, but should be set to %s"
+            msg="failed to annotate %s, because %s is already set to %s, but should be set to %s"
             % (package, tag, _value, value))
         return False
     else:
@@ -340,21 +323,21 @@ def annotation_add(module, pkgng_path, package, tag, value, dir_arg):
         return False
 
 
-def annotation_delete(module, pkgng_path, package, tag, value, dir_arg):
+def annotation_delete(module, run_pkgng, package, tag, value):
-    _value = annotation_query(module, pkgng_path, package, tag, dir_arg)
+    _value = annotation_query(module, run_pkgng, package, tag)
     if _value:
-        rc, out, err = module.run_command('%s %s annotate -y -D %s %s'
+        if not module.check_mode:
-                                          % (pkgng_path, dir_arg, package, tag))
+            rc, out, err = run_pkgng('annotate', '-y', '-D', package, tag)
-        if rc != 0:
+            if rc != 0:
-            module.fail_json(msg="could not delete annotation to %s: %s"
+                module.fail_json(msg="could not delete annotation to %s: %s"
-                             % (package, out), stderr=err)
+                                 % (package, out), stderr=err)
         return True
     return False
 
 
-def annotation_modify(module, pkgng_path, package, tag, value, dir_arg):
+def annotation_modify(module, run_pkgng, package, tag, value):
-    _value = annotation_query(module, pkgng_path, package, tag, dir_arg)
+    _value = annotation_query(module, run_pkgng, package, tag)
-    if not value:
+    if not _value:
         # No such tag
         module.fail_json(msg="could not change annotation to %s: tag %s does not exist"
                          % (package, tag))
@@ -362,20 +345,27 @@ def annotation_modify(module, pkgng_path, package, tag, value, dir_arg):
         # No change in value
         return False
     else:
-        rc, out, err = module.run_command('%s %s annotate -y -M %s %s "%s"'
+        if not module.check_mode:
-                                          % (pkgng_path, dir_arg, package, tag, value))
+            rc, out, err = run_pkgng('annotate', '-y', '-M', package, tag, data=value, binary_data=True)
-        if rc != 0:
+
-            module.fail_json(msg="could not change annotation annotation to %s: %s"
+            # pkg sometimes exits with rc == 1, even though the modification succeeded
-                             % (package, out), stderr=err)
+            # Check the output for a success message
+            if (
+                rc != 0
+                and re.search(r'^%s-[^:]+: Modified annotation tagged: %s' % (package, tag), out, flags=re.MULTILINE) is None
+            ):
+                module.fail_json(msg="failed to annotate %s, could not change annotation %s to %s: %s"
+                                 % (package, tag, value, out), stderr=err)
         return True
 
 
-def annotate_packages(module, pkgng_path, packages, annotation, dir_arg):
+def annotate_packages(module, run_pkgng, packages, annotations):
     annotate_c = 0
-    annotations = map(lambda _annotation:
+    if len(annotations) == 1:
-                      re.match(r'(?P<operation>[\+-:])(?P<tag>\w+)(=(?P<value>\w+))?',
+        # Split on commas with optional trailing whitespace,
-                               _annotation).groupdict(),
+        # to support the old style of multiple annotations
-                      re.split(r',', annotation))
+        # on a single line, rather than YAML list syntax
+        annotations = re.split(r'\s*,\s*', annotations[0])
 
     operation = {
         '+': annotation_add,
@@ -384,8 +374,21 @@ def annotate_packages(module, pkgng_path, packages, annotation, dir_arg):
     }
 
     for package in packages:
-        for _annotation in annotations:
+        for annotation_string in annotations:
-            if operation[_annotation['operation']](module, pkgng_path, package, _annotation['tag'], _annotation['value']):
+            # Note to future maintainers: A dash (-) in a regex character class ([-+:] below)
+            # must appear as the first character in the class, or it will be interpreted
+            # as a range of characters.
+            annotation = \
+                re.match(r'(?P<operation>[-+:])(?P<tag>[^=]+)(=(?P<value>.+))?', annotation_string)
+
+            if annotation is None:
+                module.fail_json(
+                    msg="failed to annotate %s, invalid annotate string: %s"
+                    % (package, annotation_string)
+                )
+
+            annotation = annotation.groupdict()
+            if operation[annotation['operation']](module, run_pkgng, package, annotation['tag'], annotation['value']):
                 annotate_c += 1
 
     if annotate_c > 0:
@@ -393,10 +396,10 @@ def annotate_packages(module, pkgng_path, packages, annotation, dir_arg):
     return (False, "changed no annotations")
 
 
-def autoremove_packages(module, pkgng_path, dir_arg):
+def autoremove_packages(module, run_pkgng):
     stdout = ""
     stderr = ""
-    rc, out, err = module.run_command("%s %s autoremove -n" % (pkgng_path, dir_arg))
+    rc, out, err = run_pkgng('autoremove', '-n')
 
     autoremove_c = 0
 
@@ -408,7 +411,7 @@ def autoremove_packages(module, pkgng_path, dir_arg):
         return (False, "no package(s) to autoremove", stdout, stderr)
 
     if not module.check_mode:
-        rc, out, err = module.run_command("%s %s autoremove -y" % (pkgng_path, dir_arg))
+        rc, out, err = run_pkgng('autoremove', '-y')
         stdout += out
         stderr += err
 
@@ -422,11 +425,11 @@ def main():
             name=dict(aliases=["pkg"], required=True, type='list', elements='str'),
             cached=dict(default=False, type='bool'),
             ignore_osver=dict(default=False, required=False, type='bool'),
-            annotation=dict(default="", required=False),
+            annotation=dict(required=False, type='list', elements='str'),
-            pkgsite=dict(default="", required=False),
+            pkgsite=dict(required=False),
-            rootdir=dict(default="", required=False, type='path'),
+            rootdir=dict(required=False, type='path'),
-            chroot=dict(default="", required=False, type='path'),
+            chroot=dict(required=False, type='path'),
-            jail=dict(default="", required=False, type='str'),
+            jail=dict(required=False, type='str'),
             autoremove=dict(default=False, type='bool')),
         supports_check_mode=True,
         mutually_exclusive=[["rootdir", "chroot", "jail"]])
@@ -441,61 +444,90 @@ def main():
     msgs = []
     stdout = ""
     stderr = ""
-    dir_arg = ""
+    dir_arg = None
 
-    if p["rootdir"] != "":
+    if p["rootdir"] is not None:
-        old_pkgng = pkgng_older_than(module, pkgng_path, [1, 5, 0])
+        rootdir_not_supported = pkgng_older_than(module, pkgng_path, [1, 5, 0])
-        if old_pkgng:
+        if rootdir_not_supported:
             module.fail_json(msg="To use option 'rootdir' pkg version must be 1.5 or greater")
         else:
-            dir_arg = "--rootdir %s" % (p["rootdir"])
+            dir_arg = "--rootdir=%s" % (p["rootdir"])
 
     if p["ignore_osver"]:
-        old_pkgng = pkgng_older_than(module, pkgng_path, [1, 11, 0])
+        ignore_osver_not_supported = pkgng_older_than(module, pkgng_path, [1, 11, 0])
-        if old_pkgng:
+        if ignore_osver_not_supported:
             module.fail_json(msg="To use option 'ignore_osver' pkg version must be 1.11 or greater")
 
-    if p["chroot"] != "":
+    if p["chroot"] is not None:
-        dir_arg = '--chroot %s' % (p["chroot"])
+        dir_arg = '--chroot=%s' % (p["chroot"])
 
-    if p["jail"] != "":
+    if p["jail"] is not None:
-        dir_arg = '--jail %s' % (p["jail"])
+        dir_arg = '--jail=%s' % (p["jail"])
+
+    # as of pkg-1.1.4, PACKAGESITE is deprecated in favor of repository definitions
+    # in /usr/local/etc/pkg/repos
+    repo_flag_not_supported = pkgng_older_than(module, pkgng_path, [1, 1, 4])
+
+    def run_pkgng(action, *args, **kwargs):
+        cmd = [pkgng_path, dir_arg, action]
+
+        pkgng_env = {'BATCH': 'yes'}
+
+        if p["ignore_osver"]:
+            pkgng_env['IGNORE_OSVERSION'] = 'yes'
+
+        if p['pkgsite'] is not None and action in ('update', 'install', 'upgrade',):
+            if repo_flag_not_supported:
+                pkgng_env['PACKAGESITE'] = p['pkgsite']
+            else:
+                cmd.append('--repository=%s' % (p['pkgsite'],))
+
+        # If environ_update is specified to be "passed through"
+        # to module.run_command, then merge its values into pkgng_env
+        pkgng_env.update(kwargs.pop('environ_update', dict()))
+
+        return module.run_command(cmd + list(args), environ_update=pkgng_env, **kwargs)
 
     if pkgs == ['*'] and p["state"] == 'latest':
         # Operate on all installed packages. Only state: latest makes sense here.
-        _changed, _msg, _stdout, _stderr = upgrade_packages(module, pkgng_path, dir_arg)
+        _changed, _msg, _stdout, _stderr = upgrade_packages(module, run_pkgng)
         changed = changed or _changed
         stdout += _stdout
         stderr += _stderr
         msgs.append(_msg)
 
     # Operate on named packages
+    if len(pkgs) == 1:
+        # The documentation used to show multiple packages specified in one line
+        # with comma or space delimiters. That doesn't result in a YAML list, and
+        # wrong actions (install vs upgrade) can be reported if those
+        # comma- or space-delimited strings make it to the pkg command line.
+        pkgs = re.split(r'[,\s]', pkgs[0])
     named_packages = [pkg for pkg in pkgs if pkg != '*']
     if p["state"] in ("present", "latest") and named_packages:
-        _changed, _msg, _out, _err = install_packages(module, pkgng_path, named_packages,
+        _changed, _msg, _out, _err = install_packages(module, run_pkgng, named_packages,
-                                                      p["cached"], p["pkgsite"], dir_arg,
+                                                      p["cached"], p["state"])
-                                                      p["state"], p["ignore_osver"])
         stdout += _out
         stderr += _err
         changed = changed or _changed
         msgs.append(_msg)
 
     elif p["state"] == "absent" and named_packages:
-        _changed, _msg, _out, _err = remove_packages(module, pkgng_path, named_packages, dir_arg)
+        _changed, _msg, _out, _err = remove_packages(module, run_pkgng, named_packages)
         stdout += _out
         stderr += _err
         changed = changed or _changed
         msgs.append(_msg)
 
     if p["autoremove"]:
-        _changed, _msg, _stdout, _stderr = autoremove_packages(module, pkgng_path, dir_arg)
+        _changed, _msg, _stdout, _stderr = autoremove_packages(module, run_pkgng)
         changed = changed or _changed
         stdout += _stdout
         stderr += _stderr
         msgs.append(_msg)
 
-    if p["annotation"]:
+    if p["annotation"] is not None:
-        _changed, _msg = annotate_packages(module, pkgng_path, pkgs, p["annotation"], dir_arg)
+        _changed, _msg = annotate_packages(module, run_pkgng, pkgs, p["annotation"])
         changed = changed or _changed
         msgs.append(_msg)
 

tests/integration/targets/pkgng/aliases

@@ -0,0 +1,5 @@
+shippable/posix/group1
+needs/root
+skip/docker
+skip/osx
+skip/rhel

tests/integration/targets/pkgng/tasks/create-outofdate-pkg.yml

@@ -0,0 +1,49 @@
+---
+- name: Create temporary directory for package creation
+  tempfile:
+    state: directory
+  register: pkgng_test_outofdate_pkg_tempdir
+
+- name: Copy intentionally out-of-date package manifest to testhost
+  template:
+    src: MANIFEST.json.j2
+    # Plus-sign must be added at the destination
+    # CI doesn't like files with '+' in them in the repository
+    dest: '{{ pkgng_test_outofdate_pkg_tempdir.path }}/MANIFEST'
+
+- name: Create out-of-date test package file
+  command:
+    argv:
+      - pkg
+      - create
+      - '--verbose'
+      - '--out-dir'
+      - '{{ pkgng_test_outofdate_pkg_tempdir.path }}'
+      - '--manifest'
+      - '{{ pkgng_test_outofdate_pkg_tempdir.path }}/MANIFEST'
+    warn: no
+
+# pkg switched from .txz to .pkg in version 1.17.0
+# Might as well look for all valid pkg extensions.
+- name: Find created package file
+  find:
+    path: '{{ pkgng_test_outofdate_pkg_tempdir.path }}'
+    use_regex: yes
+    pattern: '.*\.(pkg|tzst|t[xbg]z|tar)'
+  register: pkgng_test_outofdate_pkg_tempfile
+
+- name: There should be only one package
+  assert:
+    that:
+      - pkgng_test_outofdate_pkg_tempfile.files | count == 1
+
+- name: Copy the created package file to the expected location
+  copy:
+    remote_src: yes
+    src: '{{ pkgng_test_outofdate_pkg_tempfile.files[0].path }}'
+    dest: '{{ pkgng_test_outofdate_pkg_path }}'
+
+- name: Remove temporary directory
+  file:
+    state: absent
+    path: '{{ pkgng_test_outofdate_pkg_tempdir.path }}'

tests/integration/targets/pkgng/tasks/freebsd.yml

@@ -0,0 +1,493 @@
+---
+##
+## pkgng - prepare test environment
+##
+- name: Remove test package
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    state: absent
+
+##
+## pkgng - example - state=present for single package
+##
+- name: 'state=present for single package'
+  include_tasks: install_single_package.yml
+
+##
+## pkgng - example - state=latest for already up-to-date package
+##
+- name: Upgrade package (idempotent)
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    state: latest
+  register: pkgng_example2
+
+- name: Ensure pkgng does not upgrade up-to-date package
+  assert:
+    that:
+      - not pkgng_example2.changed
+
+##
+## pkgng - example - state=absent for single package
+##
+- name: Verify package sentinel file is present
+  stat:
+    path: '{{ pkgng_test_pkg_sentinelfile_path }}'
+    get_attributes: no
+    get_checksum: no
+    get_mime: no
+  register: pkgng_example3_stat_before
+
+- name: Install package (checkmode)
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+  check_mode: yes
+  register: pkgng_example3_checkmode
+
+- name: Remove package
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    state: absent
+  register: pkgng_example3
+
+- name: Remove package (idempotent)
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    state: absent
+  register: pkgng_example3_idempotent
+
+- name: Verify package sentinel file is not present
+  stat:
+    path: '{{ pkgng_test_pkg_sentinelfile_path }}'
+    get_attributes: no
+    get_checksum: no
+    get_mime: no
+  register: pkgng_example3_stat_after
+
+- name: Ensure pkgng installs package correctly
+  assert:
+    that:
+      - pkgng_example3_stat_before.stat.exists
+      - pkgng_example3_stat_before.stat.executable
+      - not pkgng_example3_checkmode.changed
+      - pkgng_example3.changed
+      - not pkgng_example3_idempotent.changed
+      - not pkgng_example3_stat_after.stat.exists
+
+##
+## pkgng - example - state=latest for out-of-date package
+##
+- name: Install intentionally out-of-date package and upgrade it
+  #
+  # NOTE: The out-of-date package provided is a minimal,
+  #       no-contents test package that declares {{ pkgng_test_pkg_name }} with
+  #       a version of 0, so it should always be upgraded.
+  #
+  #       This test might fail at some point in the
+  #       future if the FreeBSD package format receives
+  #       breaking changes that prevent pkg from installing
+  #       older package formats.
+  #
+  block:
+    - name: Create out-of-date test package
+      import_tasks: create-outofdate-pkg.yml
+
+    - name: Install out-of-date test package
+      command: 'pkg add {{ pkgng_test_outofdate_pkg_path }}'
+      register: pkgng_example4_prepare
+
+    - name: Check for any available package upgrades (checkmode)
+      pkgng:
+        name: '*'
+        state: latest
+      check_mode: yes
+      register: pkgng_example4_wildcard_checkmode
+
+    - name: Check for available package upgrade (checkmode)
+      pkgng:
+        name: '{{ pkgng_test_pkg_name }}'
+        state: latest
+      check_mode: yes
+      register: pkgng_example4_checkmode
+
+    - name: Upgrade out-of-date package
+      pkgng:
+        name: '{{ pkgng_test_pkg_name }}'
+        state: latest
+      register: pkgng_example4
+
+    - name: Upgrade out-of-date package (idempotent)
+      pkgng:
+        name: '{{ pkgng_test_pkg_name }}'
+        state: latest
+      register: pkgng_example4_idempotent
+
+    - name: Remove test out-of-date package
+      pkgng:
+        name: '{{ pkgng_test_pkg_name }}'
+        state: absent
+
+    - name: Ensure pkgng upgrades package correctly
+      assert:
+        that:
+          - not pkgng_example4_prepare.failed
+          - pkgng_example4_wildcard_checkmode.changed
+          - pkgng_example4_checkmode.changed
+          - pkgng_example4.changed
+          - not pkgng_example4_idempotent.changed
+
+##
+## pkgng - example - Install multiple packages in one command
+##
+- name: Remove test package (checkmode)
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    state: absent
+  check_mode: yes
+  register: pkgng_example5_prepare
+
+- name: Install three packages
+  pkgng:
+    name:
+      - '{{ pkgng_test_pkg_name }}'
+      - fish
+      - busybox
+  register: pkgng_example5
+
+- name: Remove three packages
+  pkgng:
+    name:
+      - '{{ pkgng_test_pkg_name }}'
+      - fish
+      - busybox
+    state: absent
+  register: pkgng_example5_cleanup
+
+- name: Ensure pkgng installs multiple packages with one command
+  assert:
+    that:
+      - not pkgng_example5_prepare.changed
+      - pkgng_example5.changed
+      - '(pkgng_example5.stdout | regex_search("^Number of packages to be installed: (\d+)", "\\1", multiline=True) | first | int) >= 3'
+      - '(pkgng_example5.stdout | regex_findall("^Number of packages to be", multiline=True) | count) == 1'
+      - pkgng_example5_cleanup.changed
+
+##
+## pkgng - example - state=latest multiple packages, some already installed
+##
+- name: Remove test package (checkmode)
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    state: absent
+  check_mode: yes
+  register: pkgng_example6_check
+
+- name: Create out-of-date test package
+  import_tasks: create-outofdate-pkg.yml
+
+- name: Install out-of-date test package
+  command: 'pkg add {{ pkgng_test_outofdate_pkg_path }}'
+  register: pkgng_example6_prepare
+
+- name: Upgrade and/or install two packages
+  pkgng:
+    name:
+      - '{{ pkgng_test_pkg_name }}'
+      - fish
+    state: latest
+  register: pkgng_example6
+
+- name: Remove two packages
+  pkgng:
+    name:
+      - '{{ pkgng_test_pkg_name }}'
+      - fish
+    state: absent
+  register: pkgng_example6_cleanup
+
+- name: Ensure pkgng installs multiple packages with one command
+  assert:
+    that:
+      - not pkgng_example6_check.changed
+      - not pkgng_example6_prepare.failed
+      - pkgng_example6.changed
+      - '(pkgng_example6.stdout | regex_search("^Number of packages to be installed: (\d+)", "\\1", multiline=True) | first | int) >= 1'
+      - '(pkgng_example6.stdout | regex_search("^Number of packages to be upgraded: (\d+)", "\\1", multiline=True) | first | int) >= 1'
+      # Checking that "will be affected" occurs twice in the output ensures
+      # that the module runs two separate commands for install and upgrade,
+      # as the pkg command only outputs the string once per invocation.
+      - '(pkgng_example6.stdout | regex_findall("will be affected", multiline=True) | count) == 2'
+      - pkgng_example6_cleanup.changed
+
+##
+## pkgng - example - autoremove=yes
+##
+- name: "Test autoremove=yes"
+  #
+  # NOTE: FreeBSD 12.0 test runner receives a "connection reset by peer" after ~20% downloaded so we are
+  #       only running this on 12.1 or higher
+  #
+  when: ansible_distribution_version is version('12.01', '>=')
+  block:
+    - name: Install GNU autotools
+      pkgng:
+        name: autotools
+        state: latest
+      register: pkgng_example7_prepare_install
+
+    - name: Remove GNU autotools and run pkg autoremove
+      pkgng:
+        name: autotools
+        state: absent
+        autoremove: yes
+      register: pkgng_example7
+
+    - name: Check if autoremove uninstalled known autotools dependencies
+      pkgng:
+        name:
+          - autoconf
+          - automake
+          - libtool
+          - m4
+        state: absent
+      check_mode: yes
+      register: pkgng_example7_cleanup
+
+    - name: Ensure pkgng autoremove works correctly
+      assert:
+        that:
+          - pkgng_example7_prepare_install.changed
+          - "'autoremoved' is in(pkgng_example7.msg)"
+          - not pkgng_example7_cleanup.changed
+
+##
+## pkgng - example - single annotations
+##
+- name: Install and annotate single package
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation: '+ansibletest_example8=added'
+  register: pkgng_example8_add_annotation
+
+- name: Should fail to add duplicate annotation
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation: '+ansibletest_example8=duplicate'
+  ignore_errors: yes
+  register: pkgng_example8_add_annotation_failure
+
+- name: Verify annotation is actually there
+  command: 'pkg annotate -q -S {{ pkgng_test_pkg_name }} ansibletest_example8'
+  register: pkgng_example8_add_annotation_verify
+
+- name: Install and annotate single package (checkmode, not changed)
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation: '+ansibletest_example8=added'
+  check_mode: yes
+  register: pkgng_example8_add_annotation_checkmode_nochange
+
+- name: Install and annotate single package (checkmode, changed)
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation: '+ansibletest_example8_checkmode=added'
+  check_mode: yes
+  register: pkgng_example8_add_annotation_checkmode_change
+
+- name: Verify check_mode did not add an annotation
+  command: 'pkg annotate -q -S {{ pkgng_test_pkg_name }} ansibletest_example8_checkmode'
+  register: pkgng_example8_add_annotation_checkmode_change_verify
+
+- name: Modify annotation on single package
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation: ':ansibletest_example8=modified'
+  register: pkgng_example8_modify_annotation
+
+- name: Should fail to modify missing annotation
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation: ':ansiblemissing=modified'
+  ignore_errors: yes
+  register: pkgng_example8_modify_annotation_failure
+
+- name: Verify annotation has been modified
+  command: 'pkg annotate -q -S {{ pkgng_test_pkg_name }} ansibletest_example8'
+  register: pkgng_example8_modify_annotation_verify
+
+- name: Remove annotation on single package
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation: '-ansibletest_example8'
+  register: pkgng_example8_remove_annotation
+
+- name: Verify annotation has been removed
+  command: 'pkg annotate -q -S {{ pkgng_test_pkg_name }} ansibletest_example8'
+  register: pkgng_example8_remove_annotation_verify
+
+- name: Ensure pkgng annotations on single packages work correctly
+  assert:
+    that:
+      - pkgng_example8_add_annotation.changed
+      - pkgng_example8_add_annotation_failure.failed
+      - pkgng_example8_add_annotation_checkmode_nochange is not changed
+      - pkgng_example8_add_annotation_checkmode_change is changed
+      - 'pkgng_example8_add_annotation_checkmode_change_verify.stdout_lines | count == 0'
+      - 'pkgng_example8_add_annotation_verify.stdout_lines | first == "added"'
+      - pkgng_example8_modify_annotation.changed
+      - pkgng_example8_modify_annotation_failure.failed
+      - 'pkgng_example8_modify_annotation_verify.stdout_lines | first == "modified"'
+      - pkgng_example8_remove_annotation.changed
+      - 'pkgng_example8_remove_annotation_verify.stdout_lines | count == 0'
+
+##
+## pkgng - example - multiple annotations
+##
+- name: Annotate single package with multiple annotations
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation:
+      - '+ansibletest_example9_1=added'
+      - '+ansibletest_example9_2=added'
+  register: pkgng_example9_add_annotation
+
+- name: Verify annotation is actually there
+  command: 'pkg info -q -A {{ pkgng_test_pkg_name }}'
+  register: pkgng_example9_add_annotation_verify
+  # Assert, below, tests that stdout includes:
+  # ```
+  # ansibletest_example9_1   : added
+  # ansibletest_example9_2   : added
+  # ```
+
+- name: Multiple annotation operations on single package
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation:
+      - ':ansibletest_example9_1=modified'
+      - '+ansibletest_example9_3=added'
+  register: pkgng_example9_multiple_annotation
+
+- name: Verify multiple operations succeeded
+  command: 'pkg info -q -A {{ pkgng_test_pkg_name }}'
+  register: pkgng_example9_multiple_annotation_verify
+  # Assert, below, tests that stdout includes:
+  # ```
+  # ansibletest_example9_1   : modified
+  # ansibletest_example9_2   : added
+  # ansibletest_example9_3   : added
+  # ```
+
+- name: Add multiple annotations with old syntax
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation: '+ansibletest_example9_4=added,+ansibletest_example9_5=added'
+  register: pkgng_example9_add_annotation_old
+
+- name: Verify annotation is actually there
+  command: 'pkg info -q -A {{ pkgng_test_pkg_name }}'
+  register: pkgng_example9_add_annotation_old_verify
+  # Assert, below, tests that stdout includes:
+  # ```
+  # ansibletest_example9_4   : added
+  # ansibletest_example9_5   : added
+  # ```
+
+- name: Ensure multiple annotations work correctly
+  assert:
+    that:
+      - pkgng_example9_add_annotation.changed
+      - '(pkgng_example9_add_annotation_verify.stdout_lines | select("match",  "ansibletest_example9_[12]\s*:\s*added") | list | count) == 2'
+      - pkgng_example9_multiple_annotation.changed
+      - '(pkgng_example9_multiple_annotation_verify.stdout_lines | select("match",  "ansibletest_example9_1\s*:\s*modified") | list | count) == 1'
+      - '(pkgng_example9_multiple_annotation_verify.stdout_lines | select("match",  "ansibletest_example9_[23]\s*:\s*added") | list | count) == 2'
+      - pkgng_example9_add_annotation_old.changed
+      - '(pkgng_example9_add_annotation_old_verify.stdout_lines | select("match",  "ansibletest_example9_[45]\s*:\s*added") | list | count) == 2'
+
+##
+## pkgng - example - invalid annotation strings
+##
+- name: Should fail on invalid annotate strings
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    annotation: '{{ item }}'
+  ignore_errors: yes
+  register: pkgng_example8_invalid_annotation_failure
+  loop:
+    - 'naked_string'
+    - '/invalid_operation'
+    - ',empty_first_tag=validsecond'
+    - '=notag'
+
+- name: Verify invalid annotate strings did not add annotations
+  command: 'pkg info -q -A {{ pkgng_test_pkg_name }}'
+  register: pkgng_example8_invalid_annotation_verify
+
+- name: Ensure invalid annotate strings fail safely
+  assert:
+    that:
+      # Invalid strings should not change anything
+      - '(pkgng_example8_invalid_annotation_failure.results | selectattr("changed") | list | count) == 0'
+      # Invalid strings should always fail
+      - '(pkgng_example8_invalid_annotation_failure.results | rejectattr("failed") | list | count) == 0'
+      # Invalid strings should not cause an exception
+      - '(pkgng_example8_invalid_annotation_failure.results | selectattr("exception", "defined") | list | count) == 0'
+      # Verify annotations are unaffected
+      - '(pkgng_example8_invalid_annotation_verify.stdout_lines | select("search",  "(naked_string|invalid_operation|empty_first_tag|validsecond|notag)") | list | count) == 0'
+
+##
+## pkgng - example - pkgsite=...
+##
+# NOTE: testing for failure here to not have to set up our own
+#       or depend on a third-party, alternate package repo
+- name: Should fail with invalid pkgsite
+  pkgng:
+    name: '{{ pkgng_test_pkg_name }}'
+    pkgsite: DoesNotExist
+  ignore_errors: yes
+  register: pkgng_example10_invalid_pkgsite_failure
+
+- name: Ensure invalid pkgsite fails as expected
+  assert:
+    that:
+      - pkgng_example10_invalid_pkgsite_failure.failed
+      - 'pkgng_example10_invalid_pkgsite_failure.stdout is search("^No repositories are enabled.", multiline=True)'
+
+##
+## pkgng - example - Install single package in jail
+##
+- name: Test within jail
+  #
+  # NOTE: FreeBSD 12.0 test runner receives a "connection reset by peer" after ~20% downloaded so we are
+  #       only running this on 12.1 or higher
+  #
+  when: ansible_distribution_version is version('12.01', '>=')
+  block:
+    - name: Setup testjail
+      include: setup-testjail.yml
+
+    - name: Install package in jail as rootdir
+      include_tasks: install_single_package.yml
+      vars:
+        pkgng_test_rootdir: /usr/jails/testjail
+        pkgng_test_install_prefix: /usr/jails/testjail
+        pkgng_test_install_cleanup: yes
+
+    - name: Install package in jail
+      include_tasks: install_single_package.yml
+      vars:
+        pkgng_test_jail: testjail
+        pkgng_test_install_prefix: /usr/jails/testjail
+        pkgng_test_install_cleanup: yes
+
+    - name: Install package in jail as chroot
+      include_tasks: install_single_package.yml
+      vars:
+        pkgng_test_chroot: /usr/jails/testjail
+        pkgng_test_install_prefix: /usr/jails/testjail
+        pkgng_test_install_cleanup: yes
+  always:
+    - name: Stop and remove testjail
+      failed_when: false
+      changed_when: false
+      command: "ezjail-admin delete -wf testjail"

tests/integration/targets/pkgng/tasks/install_single_package.yml

@@ -0,0 +1,54 @@
+---
+- name: Verify package sentinel file is not present
+  stat:
+    path: '{{ pkgng_test_install_prefix | default("") }}{{ pkgng_test_pkg_sentinelfile_path }}'
+    get_attributes: no
+    get_checksum: no
+    get_mime: no
+  register: pkgng_install_stat_before
+
+- name: Install package
+  pkgng: &pkgng_install_params
+    name: '{{ pkgng_test_pkg_name }}'
+    jail: '{{ pkgng_test_jail | default(omit) }}'
+    chroot: '{{ pkgng_test_chroot | default(omit) }}'
+    rootdir: '{{ pkgng_test_rootdir | default(omit) }}'
+  register: pkgng_install
+
+- name: Remove package (checkmode)
+  pkgng:
+    <<: *pkgng_install_params
+    state: absent
+  check_mode: yes
+  register: pkgng_install_checkmode
+
+- name: Install package (idempotent, cached)
+  pkgng:
+    <<: *pkgng_install_params
+    cached: yes
+  register: pkgng_install_idempotent_cached
+
+- name: Verify package sentinel file is present
+  stat:
+    path: '{{ pkgng_test_install_prefix | default("") }}{{ pkgng_test_pkg_sentinelfile_path }}'
+    get_attributes: no
+    get_checksum: no
+    get_mime: no
+  register: pkgng_install_stat_after
+
+- name: Remove test package (if requested)
+  pkgng:
+    <<: *pkgng_install_params
+    state: absent
+  when: 'pkgng_test_install_cleanup | default(False)'
+
+- name: Ensure pkgng installs package correctly
+  assert:
+    that:
+      - not pkgng_install_stat_before.stat.exists
+      - pkgng_install.changed
+      - pkgng_install_checkmode.changed
+      - not pkgng_install_idempotent_cached.changed
+      - not pkgng_install_idempotent_cached.stdout is match("Updating \w+ repository catalogue\.\.\.")
+      - pkgng_install_stat_after.stat.exists
+      - pkgng_install_stat_after.stat.executable

tests/integration/targets/pkgng/tasks/main.yml

@@ -0,0 +1,4 @@
+---
+- import_tasks: freebsd.yml
+  when:
+    - ansible_facts.distribution == 'FreeBSD'

tests/integration/targets/pkgng/tasks/setup-testjail.yml

@@ -0,0 +1,96 @@
+---
+#
+# Instructions for setting up a jail
+# https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/jails-ezjail.html
+#
+- name: Setup cloned interfaces
+  lineinfile:
+    dest: /etc/rc.conf
+    regexp: ^cloned_interfaces=lo1
+    line: cloned_interfaces=lo1
+
+- name: Activate cloned interfaces
+  command: "service netif cloneup"
+  changed_when: false
+
+- name: Add nat rule for cloned interfaces
+  copy:
+    dest: /etc/pf.conf
+    content: |
+      nat on {{ ansible_default_ipv4.interface }} from 127.0.1.0/24 -> {{ ansible_default_ipv4.interface }}:0
+    validate: "pfctl -nf %s"
+
+- name: Start pf firewall
+  service:
+    name: pf
+    state: started
+    enabled: yes
+
+- name: Install ezjail
+  pkgng:
+    name: ezjail
+
+- name: Configure ezjail to use http
+  when: ansible_distribution_version is version('11.01', '>')
+  lineinfile:
+    dest: /usr/local/etc/ezjail.conf
+    regexp: ^ezjail_ftphost
+    line: ezjail_ftphost=http://ftp.freebsd.org
+
+- name: Configure ezjail to use archive for old freebsd releases
+  when: ansible_distribution_version is version('11.01', '<=')
+  lineinfile:
+    dest: /usr/local/etc/ezjail.conf
+    regexp: ^ezjail_ftphost
+    line: ezjail_ftphost=http://ftp-archive.freebsd.org
+
+- name: Start ezjail
+  ignore_errors: yes
+  service:
+    name: ezjail
+    state: started
+    enabled: yes
+
+- name: Redirect logs depending on verbosity
+  set_fact:
+    pkgng_jail_log_redirect: "2>&1 | tee -a /tmp/ezjail.log {{ '> /dev/null' if ansible_verbosity < 2 else '' }}"
+
+- name: Has ezjail
+  register: ezjail_base_jail
+  stat:
+    path: /usr/jails/basejail
+
+- name: Setup ezjail base
+  when: not ezjail_base_jail.stat.exists
+  shell: "ezjail-admin install {{ pkgng_jail_log_redirect }}"
+  changed_when: false
+
+- name: Has testjail
+  register: ezjail_test_jail
+  stat:
+    path: /usr/jails/testjail
+
+- name: Create testjail
+  when: not ezjail_test_jail.stat.exists
+  shell: "ezjail-admin create testjail 'lo1|127.0.1.1' {{ pkgng_jail_log_redirect }}"
+  changed_when: false
+
+- name: Configure testjail to use Cloudflare DNS
+  lineinfile:
+    dest: /usr/jails/testjail/etc/resolv.conf
+    regexp: "^nameserver[[:blank:]]+{{ item }}$"
+    line: "nameserver {{ item }}"
+    create: yes
+  loop:
+    - "1.1.1.1"
+    - "1.0.0.1"
+
+- name: Is testjail running
+  shell: "jls | grep testjail"
+  changed_when: false
+  failed_when: false
+  register: is_testjail_up
+
+- name: Start testjail
+  when: is_testjail_up.rc == 1
+  command: "ezjail-admin start testjail"

tests/integration/targets/pkgng/templates/MANIFEST.json.j2

@@ -0,0 +1,16 @@
+{
+  "name": "{{ pkgng_test_pkg_name }}",
+  "origin": "{{ pkgng_test_pkg_category }}/{{ pkgng_test_pkg_name }}",
+  "version": "{{ pkgng_test_pkg_version | default('0') }}",
+  "comment": "{{ pkgng_test_pkg_name }} (Ansible Integration Test Package)",
+  "maintainer": "ansible-devel@googlegroups.com",
+  "www": "https://github.com/ansible-collections/community.general",
+  "abi": "FreeBSD:*:*",
+  "arch": "freebsd:*:*",
+  "prefix": "/usr/local",
+  "flatsize":0,
+  "licenselogic": "single",
+  "licenses":["GPLv3"],
+  "desc": "This package is only installed temporarily for integration testing of the community.general.pkgng Ansible module.\nIts version number is 0 so that ANY version of the real package, with the same name, will be considered an upgrade.\nIts architecture and abi are FreeBSD:*:* so that it will install on any version or architecture of FreeBSD,\nthus future-proof as long as the package MANIFEST format does not change\nand a wildcard in the version portion of the abi or arch field is not prohibited.",
+  "categories":["{{ pkgng_test_pkg_category }}"]
+}

tests/integration/targets/pkgng/vars/main.yml

@@ -0,0 +1,5 @@
+---
+pkgng_test_outofdate_pkg_path: "/tmp/ansible_pkgng_test_package.pkg"
+pkgng_test_pkg_name: zsh
+pkgng_test_pkg_category: shells
+pkgng_test_pkg_sentinelfile_path: /usr/local/bin/zsh