From beef93f687b9255ec675f078b9e93e78b938a81e Mon Sep 17 00:00:00 2001 From: Kosala Atapattu Date: Tue, 4 Oct 2022 09:10:16 +1300 Subject: [PATCH] Allow terraform module to specify complex variable structures (#4797) * Adding capability to specify complex variables type to terraform * Terrform variable types are mapped to ansible veriable types * Currently handles Dict, List, Str, Int, Bool types * Updated the documentation accordingly * Updated with an example. * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Wonder how that missed the PEP8 checks :). Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Adding the changelog fragment * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Adding ``integer_types`` from ``module_utils`` Simplified the ``integer_types``, ``str`` and ``float`` value population through ``json.dumps()``. Now the strings can have special characters which can break the module execution. * Update changelogs/fragments/4797-terraform-complex-variables.yml Co-authored-by: Felix Fontein * * Changed to approach to make the code more readble and simple to understand. * Maintaining the original for loop for the top_level variables. Therefore the rocess_conplex_args() now only handle second level variables when the type() is either Dict or List. * Json dumps are used only for the low level variables. Terraform CLI had issues interpreting escape sequecences from json.dumps() * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * adding boolean explicitly, although boolean is a subclass of integer, adding this for self documentation pupose and the clarity of the code. * fixing the doc strings * Update terraform.py Fixing docstrings * * Introducing format_args funtion to simplify formatting each argument type for top_level and lower level. * Terraform Lists of strings, numbers, objects and lists are supported. * Adding COMMAND: to the fail_json msg, for plan failures to help troubleshoot command line arguments. * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * * Adding full terraform command to fail_json() when the terrafor plan fails * Fixing a spelling mistake. * plan_command if a list, stringifying the list * * Fixing the new line for the change fragments * Removed CR (\r) from the output messages. Now output lines carry only LF (\n), not CRLF (\r\n). * Added integration testing for complex variables. * Restructured integration testing code to be more expandable. * Update changelogs/fragments/4797-terraform-complex-variables.yml Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com> * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com> * double-quotes are not properly escaped in shell, and python string escaping are nullified the way terraform handle second tier string variables (within terraform). * changing all the task actions to FQCN format. * integration testing now includes: 1. Top level strings containing, special shell characters, spaces, double-quotes. 2. Second level strings containing, special shell characters, spaces, double-quotes repeating double-quotes to ensure proper regex substitution. * Adding colon ':' to string test casses. * Added complex_vars to switch between the old and the new variable interpretations. Updated the documentations to reflect the changes. Updated the examples. Handling '\' as well with the escape sequence. * Added tests for the new escape sequences. Added multilines tests. * Restructuring the documente strings to a shorter string. Argument_spec changed to 'bool' * Update changelogs/fragments/4797-terraform-complex-variables.yml Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein * Update plugins/modules/cloud/misc/terraform.py Co-authored-by: Felix Fontein Co-authored-by: Felix Fontein Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com> --- .../4797-terraform-complex-variables.yml | 3 + plugins/modules/cloud/misc/terraform.py | 137 ++++++++++++++++-- .../terraform/files/complex_variables/main.tf | 35 +++++ .../files/complex_variables/variables.tf | 62 ++++++++ .../terraform/tasks/complex_variables.yml | 60 ++++++++ .../targets/terraform/tasks/main.yml | 26 ++-- .../terraform/tasks/test_provider_upgrade.yml | 9 ++ 7 files changed, 306 insertions(+), 26 deletions(-) create mode 100644 changelogs/fragments/4797-terraform-complex-variables.yml create mode 100644 tests/integration/targets/terraform/files/complex_variables/main.tf create mode 100644 tests/integration/targets/terraform/files/complex_variables/variables.tf create mode 100644 tests/integration/targets/terraform/tasks/complex_variables.yml diff --git a/changelogs/fragments/4797-terraform-complex-variables.yml b/changelogs/fragments/4797-terraform-complex-variables.yml new file mode 100644 index 0000000000..f872210a4d --- /dev/null +++ b/changelogs/fragments/4797-terraform-complex-variables.yml @@ -0,0 +1,3 @@ +minor_changes: + - terraform - adds capability to handle complex variable structures for ``variables`` parameter in the module. + This must be enabled with the new ``complex_vars`` parameter (https://github.com/ansible-collections/community.general/pull/4797). diff --git a/plugins/modules/cloud/misc/terraform.py b/plugins/modules/cloud/misc/terraform.py index e4da9cbbf0..7b6fbc5126 100644 --- a/plugins/modules/cloud/misc/terraform.py +++ b/plugins/modules/cloud/misc/terraform.py @@ -80,9 +80,25 @@ options: aliases: [ 'variables_file' ] variables: description: - - A group of key-values to override template variables or those in - variables files. + - A group of key-values pairs to override template variables or those in variables files. + By default, only string and number values are allowed, which are passed on unquoted. + - Support complex variable structures (lists, dictionaries, numbers, and booleans) to reflect terraform variable syntax when I(complex_vars=true). + - Ansible integers or floats are mapped to terraform numbers. + - Ansible strings are mapped to terraform strings. + - Ansible dictionaries are mapped to terraform objects. + - Ansible lists are mapped to terraform lists. + - Ansible booleans are mapped to terraform booleans. + - "B(Note) passwords passed as variables will be visible in the log output. Make sure to use I(no_log=true) in production!" type: dict + complex_vars: + description: + - Enable/disable capability to handle complex variable structures for C(terraform). + - If C(true) the I(variables) also accepts dictionaries, lists, and booleans to be passed to C(terraform). + Strings that are passed are correctly quoted. + - When disabled, supports only simple variables (strings, integers, and floats), and passes them on unquoted. + type: bool + default: false + version_added: 5.7.0 targets: description: - A list of specific resources to target in this plan/application. The @@ -188,6 +204,26 @@ EXAMPLES = """ - /path/to/plugins_dir_1 - /path/to/plugins_dir_2 +- name: Complex variables example + community.general.terraform: + project_path: '{{ project_dir }}' + state: present + camplex_vars: true + variables: + vm_name: "{{ inventory_hostname }}" + vm_vcpus: 2 + vm_mem: 2048 + vm_additional_disks: + - label: "Third Disk" + size: 40 + thin_provisioned: true + unit_number: 2 + - label: "Fourth Disk" + size: 22 + thin_provisioned: true + unit_number: 3 + force_init: true + ### Example directory structure for plugin_paths example # $ tree /path/to/plugins_dir_1 # /path/to/plugins_dir_1/ @@ -237,6 +273,7 @@ import os import json import tempfile from ansible.module_utils.six.moves import shlex_quote +from ansible.module_utils.six import integer_types from ansible.module_utils.basic import AnsibleModule @@ -298,7 +335,7 @@ def get_workspace_context(bin_path, project_path): command = [bin_path, 'workspace', 'list', '-no-color'] rc, out, err = module.run_command(command, cwd=project_path) if rc != 0: - module.warn("Failed to list Terraform workspaces:\r\n{0}".format(err)) + module.warn("Failed to list Terraform workspaces:\n{0}".format(err)) for item in out.split('\n'): stripped_item = item.strip() if not stripped_item: @@ -360,12 +397,25 @@ def build_plan(command, project_path, variables_args, state_file, targets, state return plan_path, False, out, err, plan_command if state == 'planned' else command elif rc == 1: # failure to plan - module.fail_json(msg='Terraform plan could not be created\r\nSTDOUT: {0}\r\n\r\nSTDERR: {1}'.format(out, err)) + module.fail_json( + msg='Terraform plan could not be created\nSTDOUT: {out}\nSTDERR: {err}\nCOMMAND: {cmd} {args}'.format( + out=out, + err=err, + cmd=' '.join(plan_command), + args=' '.join([shlex_quote(arg) for arg in variables_args]) + ) + ) elif rc == 2: # changes, but successful return plan_path, True, out, err, plan_command if state == 'planned' else command - module.fail_json(msg='Terraform plan failed with unexpected exit code {0}. \r\nSTDOUT: {1}\r\n\r\nSTDERR: {2}'.format(rc, out, err)) + module.fail_json(msg='Terraform plan failed with unexpected exit code {rc}.\nSTDOUT: {out}\nSTDERR: {err}\nCOMMAND: {cmd} {args}'.format( + rc=rc, + out=out, + err=err, + cmd=' '.join(plan_command), + args=' '.join([shlex_quote(arg) for arg in variables_args]) + )) def main(): @@ -379,6 +429,7 @@ def main(): purge_workspace=dict(type='bool', default=False), state=dict(default='present', choices=['present', 'absent', 'planned']), variables=dict(type='dict'), + complex_vars=dict(type='bool', default=False), variables_files=dict(aliases=['variables_file'], type='list', elements='path'), plan_file=dict(type='path'), state_file=dict(type='path'), @@ -405,6 +456,7 @@ def main(): purge_workspace = module.params.get('purge_workspace') state = module.params.get('state') variables = module.params.get('variables') or {} + complex_vars = module.params.get('complex_vars') variables_files = module.params.get('variables_files') plan_file = module.params.get('plan_file') state_file = module.params.get('state_file') @@ -449,12 +501,77 @@ def main(): if state == 'present' and module.params.get('parallelism') is not None: command.append('-parallelism=%d' % module.params.get('parallelism')) + def format_args(vars): + if isinstance(vars, str): + return '"{string}"'.format(string=vars.replace('\\', '\\\\').replace('"', '\\"')) + elif isinstance(vars, bool): + if vars: + return 'true' + else: + return 'false' + return str(vars) + + def process_complex_args(vars): + ret_out = [] + if isinstance(vars, dict): + for k, v in vars.items(): + if isinstance(v, dict): + ret_out.append('{0}={{{1}}}'.format(k, process_complex_args(v))) + elif isinstance(v, list): + ret_out.append("{0}={1}".format(k, process_complex_args(v))) + elif isinstance(v, (integer_types, float, str, bool)): + ret_out.append('{0}={1}'.format(k, format_args(v))) + else: + # only to handle anything unforeseen + module.fail_json(msg="Supported types are, dictionaries, lists, strings, integer_types, boolean and float.") + if isinstance(vars, list): + l_out = [] + for item in vars: + if isinstance(item, dict): + l_out.append("{{{0}}}".format(process_complex_args(item))) + elif isinstance(item, list): + l_out.append("{0}".format(process_complex_args(item))) + elif isinstance(item, (str, integer_types, float, bool)): + l_out.append(format_args(item)) + else: + # only to handle anything unforeseen + module.fail_json(msg="Supported types are, dictionaries, lists, strings, integer_types, boolean and float.") + + ret_out.append("[{0}]".format(",".join(l_out))) + return ",".join(ret_out) + variables_args = [] - for k, v in variables.items(): - variables_args.extend([ - '-var', - '{0}={1}'.format(k, v) - ]) + if complex_vars: + for k, v in variables.items(): + if isinstance(v, dict): + variables_args.extend([ + '-var', + '{0}={{{1}}}'.format(k, process_complex_args(v)) + ]) + elif isinstance(v, list): + variables_args.extend([ + '-var', + '{0}={1}'.format(k, process_complex_args(v)) + ]) + # on the top-level we need to pass just the python string with necessary + # terraform string escape sequences + elif isinstance(v, str): + variables_args.extend([ + '-var', + "{0}={1}".format(k, v) + ]) + else: + variables_args.extend([ + '-var', + '{0}={1}'.format(k, format_args(v)) + ]) + else: + for k, v in variables.items(): + variables_args.extend([ + '-var', + '{0}={1}'.format(k, v) + ]) + if variables_files: for f in variables_files: variables_args.extend(['-var-file', f]) diff --git a/tests/integration/targets/terraform/files/complex_variables/main.tf b/tests/integration/targets/terraform/files/complex_variables/main.tf new file mode 100644 index 0000000000..8b7956ec0a --- /dev/null +++ b/tests/integration/targets/terraform/files/complex_variables/main.tf @@ -0,0 +1,35 @@ +# Copyright (c) Ansible Project +# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt) +# SPDX-License-Identifier: GPL-3.0-or-later + +resource "null_resource" "mynullresource" { + triggers = { + # plain dictionaries + dict_name = var.dictionaries.name + dict_age = var.dictionaries.age + + # list of dicrs + join_dic_name = join(",", var.list_of_objects.*.name) + + # list-of-strings + join_list = join(",", var.list_of_strings.*) + + # testing boolean + name = var.boolean ? var.dictionaries.name : var.list_of_objects[0].name + + # top level string + sample_string_1 = var.string_type + + # nested lists + num_from_matrix = var.list_of_lists[1][2] + } + +} + +output "string_type" { + value = var.string_type +} + +output "multiline_string" { + value = var.multiline_string +} diff --git a/tests/integration/targets/terraform/files/complex_variables/variables.tf b/tests/integration/targets/terraform/files/complex_variables/variables.tf new file mode 100644 index 0000000000..34b050747b --- /dev/null +++ b/tests/integration/targets/terraform/files/complex_variables/variables.tf @@ -0,0 +1,62 @@ +# Copyright (c) Ansible Project +# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt) +# SPDX-License-Identifier: GPL-3.0-or-later + +variable "dictionaries" { + type = object({ + name = string + age = number + }) + description = "Same as ansible Dict" + default = { + age = 1 + name = "value" + } +} + +variable "list_of_strings" { + type = list(string) + description = "list of strings" + validation { + condition = (var.list_of_strings[1] == "cli specials\"&$%@#*!(){}[]:\"\" \\\\") + error_message = "Strings do not match." + } +} + +variable "list_of_objects" { + type = list(object({ + name = string + age = number + })) + validation { + condition = (var.list_of_objects[1].name == "cli specials\"&$%@#*!(){}[]:\"\" \\\\") + error_message = "Strings do not match." + } +} + +variable "boolean" { + type = bool + description = "boolean" + +} + +variable "string_type" { + type = string + validation { + condition = (var.string_type == "cli specials\"&$%@#*!(){}[]:\"\" \\\\") + error_message = "Strings do not match." + } +} + +variable "multiline_string" { + type = string + validation { + condition = (var.multiline_string == "one\ntwo\n") + error_message = "Strings do not match." + } +} + +variable "list_of_lists" { + type = list(list(any)) + default = [ [ 1 ], [1, 2, 3], [3] ] +} diff --git a/tests/integration/targets/terraform/tasks/complex_variables.yml b/tests/integration/targets/terraform/tasks/complex_variables.yml new file mode 100644 index 0000000000..180a1fb98c --- /dev/null +++ b/tests/integration/targets/terraform/tasks/complex_variables.yml @@ -0,0 +1,60 @@ +--- +# Copyright (c) Ansible Project +# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt) +# SPDX-License-Identifier: GPL-3.0-or-later + +- name: Create terraform project directory (complex variables) + ansible.builtin.file: + path: "{{ terraform_project_dir }}/complex_vars" + state: directory + mode: 0755 + +- name: copy terraform files to work space + ansible.builtin.copy: + src: "complex_variables/{{ item }}" + dest: "{{ terraform_project_dir }}/complex_vars/{{ item }}" + with_items: + - main.tf + - variables.tf + +# This task would test the various complex variable structures of the with the +# terraform null_resource +- name: test complex variables + community.general.terraform: + project_path: "{{ terraform_project_dir }}/complex_vars" + binary_path: "{{ terraform_binary_path }}" + force_init: yes + complex_vars: true + variables: + dictionaries: + name: "kosala" + age: 99 + list_of_strings: + - "kosala" + - 'cli specials"&$%@#*!(){}[]:"" \\' + - "xxx" + - "zzz" + list_of_objects: + - name: "kosala" + age: 99 + - name: 'cli specials"&$%@#*!(){}[]:"" \\' + age: 0.1 + - name: "zzz" + age: 9.789 + - name: "lll" + age: 1000 + boolean: true + string_type: 'cli specials"&$%@#*!(){}[]:"" \\' + multiline_string: | + one + two + list_of_lists: + - [ 1 ] + - [ 11, 12, 13 ] + - [ 2 ] + - [ 3 ] + state: present + register: terraform_init_result + +- assert: + that: terraform_init_result is not failed diff --git a/tests/integration/targets/terraform/tasks/main.yml b/tests/integration/targets/terraform/tasks/main.yml index 75c096bade..db9fc3fc5b 100644 --- a/tests/integration/targets/terraform/tasks/main.yml +++ b/tests/integration/targets/terraform/tasks/main.yml @@ -9,17 +9,17 @@ - name: Check for existing Terraform in path block: - name: Check if terraform is present in path - command: "command -v terraform" + ansible.builtin.command: "command -v terraform" register: terraform_binary_path ignore_errors: true - name: Check Terraform version - command: terraform version + ansible.builtin.command: terraform version register: terraform_version_output when: terraform_binary_path.rc == 0 - name: Set terraform version - set_fact: + ansible.builtin.set_fact: terraform_version_installed: "{{ terraform_version_output.stdout | regex_search('(?!Terraform.*v)([0-9]+\\.[0-9]+\\.[0-9]+)') }}" when: terraform_version_output.changed @@ -30,7 +30,7 @@ block: - name: Install Terraform - debug: + ansible.builtin.debug: msg: "Installing terraform {{ terraform_version }}, found: {{ terraform_version_installed | default('no terraform binary found') }}." - name: Ensure unzip is present @@ -39,7 +39,7 @@ state: present - name: Install Terraform binary - unarchive: + ansible.builtin.unarchive: src: "{{ terraform_url }}" dest: "{{ remote_tmp_dir }}" mode: 0755 @@ -52,22 +52,16 @@ # path from the 'Check if terraform is present in path' task, and lastly, the fallback path. - name: Set path to terraform binary - set_fact: + ansible.builtin.set_fact: terraform_binary_path: "{{ terraform_binary_path.stdout or remote_tmp_dir ~ '/terraform' }}" -- name: Create terraform project directory - file: - path: "{{ terraform_project_dir }}/{{ item['name'] }}" - state: directory - mode: 0755 - loop: "{{ terraform_provider_versions }}" - loop_control: - index_var: provider_index - - name: Loop over provider upgrade test tasks - include_tasks: test_provider_upgrade.yml + ansible.builtin.include_tasks: test_provider_upgrade.yml vars: tf_provider: "{{ terraform_provider_versions[provider_index] }}" loop: "{{ terraform_provider_versions }}" loop_control: index_var: provider_index + +- name: Test Complex Varibles + ansible.builtin.include_tasks: complex_variables.yml diff --git a/tests/integration/targets/terraform/tasks/test_provider_upgrade.yml b/tests/integration/targets/terraform/tasks/test_provider_upgrade.yml index ac76c38837..711dfc1a33 100644 --- a/tests/integration/targets/terraform/tasks/test_provider_upgrade.yml +++ b/tests/integration/targets/terraform/tasks/test_provider_upgrade.yml @@ -3,6 +3,15 @@ # GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt) # SPDX-License-Identifier: GPL-3.0-or-later +- name: Create terraform project directory (provider upgrade) + file: + path: "{{ terraform_project_dir }}/{{ item['name'] }}" + state: directory + mode: 0755 + loop: "{{ terraform_provider_versions }}" + loop_control: + index_var: provider_index + - name: Output terraform provider test project ansible.builtin.template: src: templates/provider_test/main.tf.j2