Allow terraform module to specify complex variable structures (#4797)

* Adding capability to specify complex variables type to terraform * Terrform variable types are mapped to ansible veriable types * Currently handles Dict, List, Str, Int, Bool types * Updated the documentation accordingly * Updated with an example. * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Wonder how that missed the PEP8 checks :). Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Adding the changelog fragment * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Adding ``integer_types`` from ``module_utils`` Simplified the ``integer_types``, ``str`` and ``float`` value population through ``json.dumps()``. Now the strings can have special characters which can break the module execution. * Update changelogs/fragments/4797-terraform-complex-variables.yml Co-authored-by: Felix Fontein <felix@fontein.de> * * Changed to approach to make the code more readble and simple to understand. * Maintaining the original for loop for the top_level variables. Therefore the rocess_conplex_args() now only handle second level variables when the type() is either Dict or List. * Json dumps are used only for the low level variables. Terraform CLI had issues interpreting escape sequecences from json.dumps() * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * adding boolean explicitly, although boolean is a subclass of integer, adding this for self documentation pupose and the clarity of the code. * fixing the doc strings * Update terraform.py Fixing docstrings * * Introducing format_args funtion to simplify formatting each argument type for top_level and lower level. * Terraform Lists of strings, numbers, objects and lists are supported. * Adding COMMAND: to the fail_json msg, for plan failures to help troubleshoot command line arguments. * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * * Adding full terraform command to fail_json() when the terrafor plan fails * Fixing a spelling mistake. * plan_command if a list, stringifying the list * * Fixing the new line for the change fragments * Removed CR (\r) from the output messages. Now output lines carry only LF (\n), not CRLF (\r\n). * Added integration testing for complex variables. * Restructured integration testing code to be more expandable. * Update changelogs/fragments/4797-terraform-complex-variables.yml Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com> * double-quotes are not properly escaped in shell, and python string escaping are nullified the way terraform handle second tier string variables (within terraform). * changing all the task actions to FQCN format. * integration testing now includes: 1. Top level strings containing, special shell characters, spaces, double-quotes. 2. Second level strings containing, special shell characters, spaces, double-quotes repeating double-quotes to ensure proper regex substitution. * Adding colon ':' to string test casses. * Added complex_vars to switch between the old and the new variable interpretations. Updated the documentations to reflect the changes. Updated the examples. Handling '\' as well with the escape sequence. * Added tests for the new escape sequences. Added multilines tests. * Restructuring the documente strings to a shorter string. Argument_spec changed to 'bool' * Update changelogs/fragments/4797-terraform-complex-variables.yml Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein <felix@fontein.de> Co-authored-by: Felix Fontein <felix@fontein.de> Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com>
2024-09-14 20:13:21 +02:00 · 2022-10-04 09:10:16 +13:00 · 2022-10-04 09:10:16 +13:00 · beef93f687
commit beef93f687
parent 6fe2a84e87
7 changed files with 306 additions and 26 deletions
--- a/changelogs/fragments/4797-terraform-complex-variables.yml
+++ b/changelogs/fragments/4797-terraform-complex-variables.yml
@ -0,0 +1,3 @@
+minor_changes:
+  - terraform - adds capability to handle complex variable structures for ``variables`` parameter in the module.
+    This must be enabled with the new ``complex_vars`` parameter (https://github.com/ansible-collections/community.general/pull/4797).
--- a/plugins/modules/cloud/misc/terraform.py
+++ b/plugins/modules/cloud/misc/terraform.py
@ -80,9 +80,25 @@ options:
    aliases: [ 'variables_file' ]
  variables:
    description:
-      - A group of key-values to override template variables or those in
-        variables files.
+      - A group of key-values pairs to override template variables or those in variables files.
+        By default, only string and number values are allowed, which are passed on unquoted.
+      - Support complex variable structures (lists, dictionaries, numbers, and booleans) to reflect terraform variable syntax when I(complex_vars=true).
+      - Ansible integers or floats are mapped to terraform numbers.
+      - Ansible strings are mapped to terraform strings.
+      - Ansible dictionaries are mapped to terraform objects.
+      - Ansible lists are mapped to terraform lists.
+      - Ansible booleans are mapped to terraform booleans.
+      - "B(Note) passwords passed as variables will be visible in the log output. Make sure to use I(no_log=true) in production!"
    type: dict
+  complex_vars:
+    description:
+      - Enable/disable capability to handle complex variable structures for C(terraform).
+      - If C(true) the I(variables) also accepts dictionaries, lists, and booleans to be passed to C(terraform).
+        Strings that are passed are correctly quoted.
+      - When disabled, supports only simple variables (strings, integers, and floats), and passes them on unquoted.
+    type: bool
+    default: false
+    version_added: 5.7.0
  targets:
    description:
      - A list of specific resources to target in this plan/application. The
@ -188,6 +204,26 @@ EXAMPLES = """
      - /path/to/plugins_dir_1
      - /path/to/plugins_dir_2

+- name: Complex variables example
+  community.general.terraform:
+    project_path: '{{ project_dir }}'
+    state: present
+    camplex_vars: true
+    variables:
+      vm_name: "{{ inventory_hostname }}"
+      vm_vcpus: 2
+      vm_mem: 2048
+      vm_additional_disks:
+        - label: "Third Disk"
+          size: 40
+          thin_provisioned: true
+          unit_number: 2
+        - label: "Fourth Disk"
+          size: 22
+          thin_provisioned: true
+          unit_number: 3
+    force_init: true
+
 ### Example directory structure for plugin_paths example
 # $ tree /path/to/plugins_dir_1
 # /path/to/plugins_dir_1/
@ -237,6 +273,7 @@ import os
 import json
 import tempfile
 from ansible.module_utils.six.moves import shlex_quote
+from ansible.module_utils.six import integer_types

 from ansible.module_utils.basic import AnsibleModule

@ -298,7 +335,7 @@ def get_workspace_context(bin_path, project_path):
    command = [bin_path, 'workspace', 'list', '-no-color']
    rc, out, err = module.run_command(command, cwd=project_path)
    if rc != 0:
-        module.warn("Failed to list Terraform workspaces:\r\n{0}".format(err))
+        module.warn("Failed to list Terraform workspaces:\n{0}".format(err))
    for item in out.split('\n'):
        stripped_item = item.strip()
        if not stripped_item:
@ -360,12 +397,25 @@ def build_plan(command, project_path, variables_args, state_file, targets, state
        return plan_path, False, out, err, plan_command if state == 'planned' else command
    elif rc == 1:
        # failure to plan
-        module.fail_json(msg='Terraform plan could not be created\r\nSTDOUT: {0}\r\n\r\nSTDERR: {1}'.format(out, err))
+        module.fail_json(
+            msg='Terraform plan could not be created\nSTDOUT: {out}\nSTDERR: {err}\nCOMMAND: {cmd} {args}'.format(
+                out=out,
+                err=err,
+                cmd=' '.join(plan_command),
+                args=' '.join([shlex_quote(arg) for arg in variables_args])
+            )
+        )
    elif rc == 2:
        # changes, but successful
        return plan_path, True, out, err, plan_command if state == 'planned' else command

-    module.fail_json(msg='Terraform plan failed with unexpected exit code {0}. \r\nSTDOUT: {1}\r\n\r\nSTDERR: {2}'.format(rc, out, err))
+    module.fail_json(msg='Terraform plan failed with unexpected exit code {rc}.\nSTDOUT: {out}\nSTDERR: {err}\nCOMMAND: {cmd} {args}'.format(
+        rc=rc,
+        out=out,
+        err=err,
+        cmd=' '.join(plan_command),
+        args=' '.join([shlex_quote(arg) for arg in variables_args])
+    ))


 def main():
@ -379,6 +429,7 @@ def main():
            purge_workspace=dict(type='bool', default=False),
            state=dict(default='present', choices=['present', 'absent', 'planned']),
            variables=dict(type='dict'),
+            complex_vars=dict(type='bool', default=False),
            variables_files=dict(aliases=['variables_file'], type='list', elements='path'),
            plan_file=dict(type='path'),
            state_file=dict(type='path'),
@ -405,6 +456,7 @@ def main():
    purge_workspace = module.params.get('purge_workspace')
    state = module.params.get('state')
    variables = module.params.get('variables') or {}
+    complex_vars = module.params.get('complex_vars')
    variables_files = module.params.get('variables_files')
    plan_file = module.params.get('plan_file')
    state_file = module.params.get('state_file')
@ -449,12 +501,77 @@ def main():
    if state == 'present' and module.params.get('parallelism') is not None:
        command.append('-parallelism=%d' % module.params.get('parallelism'))

+    def format_args(vars):
+        if isinstance(vars, str):
+            return '"{string}"'.format(string=vars.replace('\\', '\\\\').replace('"', '\\"'))
+        elif isinstance(vars, bool):
+            if vars:
+                return 'true'
+            else:
+                return 'false'
+        return str(vars)
+
+    def process_complex_args(vars):
+        ret_out = []
+        if isinstance(vars, dict):
+            for k, v in vars.items():
+                if isinstance(v, dict):
+                    ret_out.append('{0}={{{1}}}'.format(k, process_complex_args(v)))
+                elif isinstance(v, list):
+                    ret_out.append("{0}={1}".format(k, process_complex_args(v)))
+                elif isinstance(v, (integer_types, float, str, bool)):
+                    ret_out.append('{0}={1}'.format(k, format_args(v)))
+                else:
+                    # only to handle anything unforeseen
+                    module.fail_json(msg="Supported types are, dictionaries, lists, strings, integer_types, boolean and float.")
+        if isinstance(vars, list):
+            l_out = []
+            for item in vars:
+                if isinstance(item, dict):
+                    l_out.append("{{{0}}}".format(process_complex_args(item)))
+                elif isinstance(item, list):
+                    l_out.append("{0}".format(process_complex_args(item)))
+                elif isinstance(item, (str, integer_types, float, bool)):
+                    l_out.append(format_args(item))
+                else:
+                    # only to handle anything unforeseen
+                    module.fail_json(msg="Supported types are, dictionaries, lists, strings, integer_types, boolean and float.")
+
+            ret_out.append("[{0}]".format(",".join(l_out)))
+        return ",".join(ret_out)
+
    variables_args = []
-    for k, v in variables.items():
-        variables_args.extend([
-            '-var',
-            '{0}={1}'.format(k, v)
-        ])
+    if complex_vars:
+        for k, v in variables.items():
+            if isinstance(v, dict):
+                variables_args.extend([
+                    '-var',
+                    '{0}={{{1}}}'.format(k, process_complex_args(v))
+                ])
+            elif isinstance(v, list):
+                variables_args.extend([
+                    '-var',
+                    '{0}={1}'.format(k, process_complex_args(v))
+                ])
+            # on the top-level we need to pass just the python string with necessary
+            # terraform string escape sequences
+            elif isinstance(v, str):
+                variables_args.extend([
+                    '-var',
+                    "{0}={1}".format(k, v)
+                ])
+            else:
+                variables_args.extend([
+                    '-var',
+                    '{0}={1}'.format(k, format_args(v))
+                ])
+    else:
+        for k, v in variables.items():
+            variables_args.extend([
+                '-var',
+                '{0}={1}'.format(k, v)
+            ])
+
    if variables_files:
        for f in variables_files:
            variables_args.extend(['-var-file', f])
--- a/tests/integration/targets/terraform/files/complex_variables/main.tf
+++ b/tests/integration/targets/terraform/files/complex_variables/main.tf
@ -0,0 +1,35 @@
+# Copyright (c) Ansible Project
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+resource "null_resource" "mynullresource" {
+  triggers = {
+    # plain dictionaries
+    dict_name = var.dictionaries.name
+    dict_age  = var.dictionaries.age
+
+    # list of dicrs
+    join_dic_name = join(",", var.list_of_objects.*.name)
+
+    # list-of-strings
+    join_list = join(",", var.list_of_strings.*)
+
+    # testing boolean
+    name = var.boolean ? var.dictionaries.name : var.list_of_objects[0].name
+
+    # top level string
+    sample_string_1 = var.string_type
+
+    # nested lists
+    num_from_matrix = var.list_of_lists[1][2]
+  }
+
+}
+
+output "string_type" {
+  value = var.string_type
+}
+
+output "multiline_string" {
+  value = var.multiline_string
+}
--- a/tests/integration/targets/terraform/files/complex_variables/variables.tf
+++ b/tests/integration/targets/terraform/files/complex_variables/variables.tf
@ -0,0 +1,62 @@
+# Copyright (c) Ansible Project
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+variable "dictionaries" {
+  type = object({
+    name = string
+    age  = number
+  })
+  description = "Same as ansible Dict"
+  default = {
+    age = 1
+    name = "value"
+  }
+}
+
+variable "list_of_strings" {
+  type        = list(string)
+  description = "list of strings"
+  validation {
+    condition     = (var.list_of_strings[1] == "cli specials\"&$%@#*!(){}[]:\"\" \\\\")
+    error_message = "Strings do not match."
+  }
+}
+
+variable "list_of_objects" {
+  type = list(object({
+    name = string
+    age  = number
+  }))
+  validation {
+    condition     = (var.list_of_objects[1].name == "cli specials\"&$%@#*!(){}[]:\"\" \\\\")
+    error_message = "Strings do not match."
+  }
+}
+
+variable "boolean" {
+  type        = bool
+  description = "boolean"
+
+}
+
+variable "string_type" {
+  type = string
+  validation {
+    condition     = (var.string_type == "cli specials\"&$%@#*!(){}[]:\"\" \\\\")
+    error_message = "Strings do not match."
+  }
+}
+
+variable "multiline_string" {
+  type = string
+  validation {
+    condition     = (var.multiline_string == "one\ntwo\n")
+    error_message = "Strings do not match."
+  }
+}
+
+variable "list_of_lists" {
+  type = list(list(any))
+  default = [ [ 1 ], [1, 2, 3], [3] ]
+}
--- a/tests/integration/targets/terraform/tasks/complex_variables.yml
+++ b/tests/integration/targets/terraform/tasks/complex_variables.yml
@ -0,0 +1,60 @@
+---
+# Copyright (c) Ansible Project
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+- name: Create terraform project directory (complex variables)
+  ansible.builtin.file:
+    path: "{{ terraform_project_dir }}/complex_vars"
+    state: directory
+    mode: 0755
+
+- name: copy terraform files to work space
+  ansible.builtin.copy:
+    src: "complex_variables/{{ item }}"
+    dest: "{{ terraform_project_dir }}/complex_vars/{{ item }}"
+  with_items:
+    - main.tf
+    - variables.tf
+
+# This task would test the various complex variable structures of the with the
+# terraform null_resource
+- name: test complex variables
+  community.general.terraform:
+    project_path: "{{ terraform_project_dir }}/complex_vars"
+    binary_path: "{{ terraform_binary_path }}"
+    force_init: yes
+    complex_vars: true
+    variables:
+      dictionaries:
+        name: "kosala"
+        age: 99
+      list_of_strings:
+        - "kosala"
+        - 'cli specials"&$%@#*!(){}[]:"" \\'
+        - "xxx"
+        - "zzz"
+      list_of_objects:
+        - name: "kosala"
+          age: 99
+        - name: 'cli specials"&$%@#*!(){}[]:"" \\'
+          age: 0.1
+        - name: "zzz"
+          age: 9.789
+        - name: "lll"
+          age: 1000
+      boolean: true
+      string_type: 'cli specials"&$%@#*!(){}[]:"" \\'
+      multiline_string: |
+        one
+        two
+      list_of_lists:
+        - [ 1 ]
+        - [ 11, 12, 13 ]
+        - [ 2 ]
+        - [ 3 ]
+    state: present
+  register: terraform_init_result
+
+- assert:
+    that: terraform_init_result is not failed
--- a/tests/integration/targets/terraform/tasks/main.yml
+++ b/tests/integration/targets/terraform/tasks/main.yml
@ -9,17 +9,17 @@
 - name: Check for existing Terraform in path
  block:
  - name: Check if terraform is present in path
-    command: "command -v terraform"
+    ansible.builtin.command: "command -v terraform"
    register: terraform_binary_path
    ignore_errors: true

  - name: Check Terraform version
-    command: terraform version
+    ansible.builtin.command: terraform version
    register: terraform_version_output
    when: terraform_binary_path.rc == 0

  - name: Set terraform version
-    set_fact:
+    ansible.builtin.set_fact:
      terraform_version_installed: "{{ terraform_version_output.stdout | regex_search('(?!Terraform.*v)([0-9]+\\.[0-9]+\\.[0-9]+)') }}"
    when: terraform_version_output.changed

@ -30,7 +30,7 @@
  block:

  - name: Install Terraform
-    debug:
+    ansible.builtin.debug:
      msg: "Installing terraform {{ terraform_version }}, found: {{ terraform_version_installed | default('no terraform binary found') }}."

  - name: Ensure unzip is present
@ -39,7 +39,7 @@
      state: present

  - name: Install Terraform binary
-    unarchive:
+    ansible.builtin.unarchive:
      src: "{{ terraform_url }}"
      dest: "{{ remote_tmp_dir }}"
      mode: 0755
@ -52,22 +52,16 @@
 # path from the 'Check if terraform is present in path' task, and lastly, the fallback path.

 - name: Set path to terraform binary
-  set_fact:
+  ansible.builtin.set_fact:
    terraform_binary_path: "{{ terraform_binary_path.stdout or remote_tmp_dir ~ '/terraform' }}"

- name: Create terraform project directory
-  file:
-    path: "{{ terraform_project_dir }}/{{ item['name'] }}"
-    state: directory
-    mode: 0755
-  loop: "{{ terraform_provider_versions }}"
-  loop_control:
-    index_var: provider_index
-
 - name: Loop over provider upgrade test tasks
-  include_tasks: test_provider_upgrade.yml
+  ansible.builtin.include_tasks: test_provider_upgrade.yml
  vars:
    tf_provider: "{{ terraform_provider_versions[provider_index] }}"
  loop: "{{ terraform_provider_versions }}"
  loop_control:
    index_var: provider_index
+
+- name: Test Complex Varibles
+  ansible.builtin.include_tasks: complex_variables.yml
--- a/tests/integration/targets/terraform/tasks/test_provider_upgrade.yml
+++ b/tests/integration/targets/terraform/tasks/test_provider_upgrade.yml
@ -3,6 +3,15 @@
 # GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
 # SPDX-License-Identifier: GPL-3.0-or-later

+- name: Create terraform project directory (provider upgrade)
+  file:
+    path: "{{ terraform_project_dir }}/{{ item['name'] }}"
+    state: directory
+    mode: 0755
+  loop: "{{ terraform_provider_versions }}"
+  loop_control:
+    index_var: provider_index
+
 - name: Output terraform provider test project
  ansible.builtin.template:
    src: templates/provider_test/main.tf.j2