Module listen ports facts extend output (#4953)

* Initial Rework of netstat and ss to include additional information.
State, foreign address, process.

* Fixed sanity tests. Python 2 compatible code. pylint errors resolved.

* Sanity tests. ss_parse fix minor error I created before.

* Rename variable for clarity

* Python2 rsplit takes no keyword argument. -> remove keyword argument

* Generic improvments for split_pid_name. Added changelog

* Sanity Test (no type hints for python2.7)

* add include_non_listening param. Add param to test. Add documentation. Only return state and foreign_address when include_non_listening

* Update changelogs/fragments/4953-listen-ports-facts-extend-output.yaml

Co-authored-by: Felix Fontein <felix@fontein.de>

* Add info to changelog fragment. Clarify documentation.

* The case where we have multiple entries in pids for udp eg: users:(("rpcbind",pid=733,fd=5),("systemd",pid=1,fd=30)) is not in the tests. So roll back to previous approach where this is covered. Fix wrong if condition for include_non_listening.

* Rewrite documentation and formatting.

* Last small documentation adjustments.

* Update parameters to match description.

* added test cases to check if include_non_listening is set to no by default. And test if ports and foreign_address exists if set to yes

* undo rename from address to local_address -> breaking change

* Replace choice with bool, as it is the correct fit here

* nestat distinguishes between tcp6 and tcp output should always be tcp

* Minor adjustments in the docs (no -> false, is set to yes -> true)

Co-authored-by: Paul-Kehnel <paul.kehnel@ocean.ibm.com>
Co-authored-by: Felix Fontein <felix@fontein.de>

changelogs/fragments/4953-listen-ports-facts-extend-output.yaml

@@ -0,0 +1,2 @@
+minor_changes:
+  - listen_ports_facts - add new ``include_non_listening`` option which adds ``-a`` option to ``netstat`` and ``ss``. This shows both listening and non-listening (for TCP this means established connections) sockets, and returns ``state`` and ``foreign_address`` (https://github.com/ansible-collections/community.general/issues/4762, https://github.com/ansible-collections/community.general/pull/4953).

plugins/modules/system/listen_ports_facts.py

@@ -32,6 +32,13 @@ options:
       - netstat
       - ss
     version_added: 4.1.0
+  include_non_listening:
+    description:
+        - Show both listening and non-listening sockets (for TCP this means established connections).
+        - Adds the return values C(state) and C(foreign_address) to the returned facts.
+    type: bool
+    default: false
+    version_added: 5.4.0
 '''
 
 EXAMPLES = r'''
@@ -59,6 +66,11 @@ EXAMPLES = r'''
 - name: List all ports
   ansible.builtin.debug:
     msg: "{{ (ansible_facts.tcp_listen + ansible_facts.udp_listen) | map(attribute='port') | unique | sort | list }}"
+
+- name: Gather facts on all ports and override which command to use
+  community.general.listen_ports_facts:
+    command: 'netstat'
+    include_non_listening: 'yes'
 '''
 
 RETURN = r'''
@@ -77,6 +89,18 @@ ansible_facts:
           returned: always
           type: str
           sample: "0.0.0.0"
+        foreign_address:
+          description: The address of the remote end of the socket.
+          returned: if I(include_non_listening=true)
+          type: str
+          sample: "10.80.0.1"
+          version_added: 5.4.0
+        state:
+          description: The state of the socket.
+          returned: if I(include_non_listening=true)
+          type: str
+          sample: "ESTABLISHED"
+          version_added: 5.4.0
         name:
           description: The name of the listening process.
           returned: if user permissions allow
@@ -117,6 +141,18 @@ ansible_facts:
           returned: always
           type: str
           sample: "0.0.0.0"
+        foreign_address:
+          description: The address of the remote end of the socket.
+          returned: if I(include_non_listening=true)
+          type: str
+          sample: "10.80.0.1"
+          version_added: 5.4.0
+        state:
+          description: The state of the socket. UDP is a connectionless protocol. Shows UCONN or ESTAB.
+          returned: if I(include_non_listening=true)
+          type: str
+          sample: "UCONN"
+          version_added: 5.4.0
         name:
           description: The name of the listening process.
           returned: if user permissions allow
@@ -155,38 +191,68 @@ from ansible.module_utils.common.text.converters import to_native
 from ansible.module_utils.basic import AnsibleModule
 
 
+def split_pid_name(pid_name):
+    """
+    Split the entry PID/Program name into the PID (int) and the name (str)
+    :param pid_name:  PID/Program String seperated with a dash. E.g 51/sshd: returns pid = 51 and name = sshd
+    :return: PID (int) and the program name (str)
+    """
+    try:
+        pid, name = pid_name.split("/", 1)
+    except ValueError:
+        # likely unprivileged user, so add empty name & pid
+        return 0, ""
+    else:
+        name = name.rstrip(":")
+        return int(pid), name
+
+
 def netStatParse(raw):
+    """
+    The netstat result can be either split in 6,7 or 8 elements depending on the values of state, process and name.
+    For UDP the state is always empty. For UDP and TCP the process can be empty.
+    So these cases have to be checked.
+    :param raw: Netstat raw output String. First line explains the format, each following line contains a connection.
+    :return: List of dicts, each dict contains protocol, state, local address, foreign address, port, name, pid for one
+     connection.
+    """
     results = list()
     for line in raw.splitlines():
-        listening_search = re.search('[^ ]+:[0-9]+', line)
+        if line.startswith(("tcp", "udp")):
-        if listening_search:
+            # set variables to default state, in case they are not specified
-            splitted = line.split()
+            state = ""
-            conns = re.search('([^ ]+):([0-9]+)', splitted[3])
+            pid_and_name = ""
-            pidstr = ''
+            process = ""
-            if 'tcp' in splitted[0]:
+            formatted_line = line.split()
-                protocol = 'tcp'
+            protocol, recv_q, send_q, address, foreign_address, rest = \
-                pidstr = splitted[6]
+                formatted_line[0], formatted_line[1], formatted_line[2], formatted_line[3], formatted_line[4], formatted_line[5:]
-            elif 'udp' in splitted[0]:
+            address, port = address.rsplit(":", 1)
-                protocol = 'udp'
+
-                pidstr = splitted[5]
+            if protocol.startswith("tcp"):
-            pids = re.search(r'(([0-9]+)/(.*)|-)', pidstr)
+                # nestat distinguishes between tcp6 and tcp
-            if conns and pids:
+                protocol = "tcp"
-                address = conns.group(1)
+                if len(rest) == 3:
-                port = conns.group(2)
+                    state, pid_and_name, process = rest
-                if (pids.group(2)):
+                if len(rest) == 2:
-                    pid = pids.group(2)
+                    state, pid_and_name = rest
-                else:
+
-                    pid = 0
+            if protocol.startswith("udp"):
-                if (pids.group(3)):
+                # safety measure, similar to tcp6
-                    name = pids.group(3)
+                protocol = "udp"
-                else:
+                if len(rest) == 2:
-                    name = ''
+                    pid_and_name, process = rest
+                if len(rest) == 1:
+                    pid_and_name = rest[0]
+
+            pid, name = split_pid_name(pid_name=pid_and_name)
             result = {
-                    'pid': int(pid),
-                    'address': address,
-                    'port': int(port),
                 'protocol': protocol,
+                'state': state,
+                'address': address,
+                'foreign_address': foreign_address,
+                'port': int(port),
                 'name': name,
+                'pid': int(pid),
             }
             if result not in results:
                 results.append(result)
@@ -196,6 +262,13 @@ def netStatParse(raw):
 
 
 def ss_parse(raw):
+    """
+    The ss_parse result can be either split in 6 or 7 elements depending on the process column,
+    e.g. due to unprivileged user.
+    :param raw: ss raw output String. First line explains the format, each following line contains a connection.
+    :return: List of dicts, each dict contains protocol, state, local address, foreign address, port, name, pid for one
+     connection.
+    """
     results = list()
     regex_conns = re.compile(pattern=r'\[?(.+?)\]?:([0-9]+)$')
     regex_pid = re.compile(pattern=r'"(.*?)",pid=(\d+)')
@@ -221,8 +294,8 @@ def ss_parse(raw):
         except ValueError:
             # unexpected stdout from ss
             raise EnvironmentError(
-                'Expected `ss` table layout "Netid, State, Recv-Q, Send-Q, Local Address:Port, Peer Address:Port" and optionally "Process", \
+                'Expected `ss` table layout "Netid, State, Recv-Q, Send-Q, Local Address:Port, Peer Address:Port" and \
-                    but got something else: {0}'.format(line)
+                 optionally "Process", but got something else: {0}'.format(line)
             )
 
         conns = regex_conns.search(local_addr_port)
@@ -239,46 +312,44 @@ def ss_parse(raw):
         port = conns.group(2)
         for name, pid in pids:
             result = {
-                'pid': int(pid),
-                'address': address,
-                'port': int(port),
                 'protocol': protocol,
-                'name': name
+                'state': state,
+                'address': address,
+                'foreign_address': peer_addr_port,
+                'port': int(port),
+                'name': name,
+                'pid': int(pid),
             }
             results.append(result)
     return results
 
 
 def main():
+    command_args = ['-p', '-l', '-u', '-n', '-t']
     commands_map = {
         'netstat': {
-            'args': [
+            'args': [],
-                '-p',
-                '-l',
-                '-u',
-                '-n',
-                '-t',
-            ],
             'parse_func': netStatParse
         },
         'ss': {
-            'args': [
+            'args': [],
-                '-p',
-                '-l',
-                '-u',
-                '-n',
-                '-t',
-            ],
             'parse_func': ss_parse
         },
     }
     module = AnsibleModule(
         argument_spec=dict(
-            command=dict(type='str', choices=list(sorted(commands_map)))
+            command=dict(type='str', choices=list(sorted(commands_map))),
+            include_non_listening=dict(default=False, type='bool'),
         ),
         supports_check_mode=True,
     )
 
+    if module.params['include_non_listening']:
+        command_args = ['-p', '-u', '-n', '-t', '-a']
+
+    commands_map['netstat']['args'] = command_args
+    commands_map['ss']['args'] = command_args
+
     if platform.system() != 'Linux':
         module.fail_json(msg='This module requires Linux.')
 
@@ -333,13 +404,17 @@ def main():
             parse_func = commands_map[command]['parse_func']
             results = parse_func(stdout)
 
-            for p in results:
+            for connection in results:
-                p['stime'] = getPidSTime(p['pid'])
+                # only display state and foreign_address for include_non_listening.
-                p['user'] = getPidUser(p['pid'])
+                if not module.params['include_non_listening']:
-                if p['protocol'].startswith('tcp'):
+                    connection.pop('state', None)
-                    result['ansible_facts']['tcp_listen'].append(p)
+                    connection.pop('foreign_address', None)
-                elif p['protocol'].startswith('udp'):
+                connection['stime'] = getPidSTime(connection['pid'])
-                    result['ansible_facts']['udp_listen'].append(p)
+                connection['user'] = getPidUser(connection['pid'])
+                if connection['protocol'].startswith('tcp'):
+                    result['ansible_facts']['tcp_listen'].append(connection)
+                elif connection['protocol'].startswith('udp'):
+                    result['ansible_facts']['udp_listen'].append(connection)
     except (KeyError, EnvironmentError) as e:
         module.fail_json(msg=to_native(e))
 

tests/integration/targets/listen_ports_facts/tasks/main.yml

@@ -58,14 +58,23 @@
   listen_ports_facts:
   when: ansible_os_family == "RedHat" or ansible_os_family == "Debian"
 
-- name: Gather listening ports facts explicitly via netstat
+- name: check that the include_non_listening parameters ('state' and 'foreign_address') are not active in default setting
+  assert:
+    that:
+      - ansible_facts.tcp_listen | selectattr('state', 'defined') | list | length == 0
+      - ansible_facts.tcp_listen | selectattr('foreign_address', 'defined') | list | length == 0
+  when: ansible_os_family == "RedHat" or ansible_os_family == "Debian"
+
+- name: Gather listening ports facts explicitly via netstat and include_non_listening
   listen_ports_facts:
     command: 'netstat'
+    include_non_listening: 'yes'
   when: (ansible_os_family == "RedHat" and ansible_distribution_major_version|int < 7) or ansible_os_family == "Debian"
 
-- name: Gather listening ports facts explicitly via ss
+- name: Gather listening ports facts explicitly via ss and include_non_listening
   listen_ports_facts:
     command: 'ss'
+    include_non_listening: 'yes'
   when: ansible_os_family == "RedHat" and ansible_distribution_major_version|int >= 7
 
 - name: check for ansible_facts.udp_listen exists
@@ -78,6 +87,13 @@
     that: ansible_facts.tcp_listen is defined
   when: ansible_os_family == "RedHat" or ansible_os_family == "Debian"
 
+- name: check that the include_non_listening parameter 'state' and 'foreign_address' exists
+  assert:
+    that:
+      - ansible_facts.tcp_listen | selectattr('state', 'defined') | list | length > 0
+      - ansible_facts.tcp_listen | selectattr('foreign_address', 'defined') | list | length > 0
+  when: ansible_os_family == "RedHat" or ansible_os_family == "Debian"
+
 - name: check TCP 5556 is in listening ports
   assert:
     that: 5556 in ansible_facts.tcp_listen | map(attribute='port') | sort | list