1
0
Fork 0
mirror of https://github.com/ansible-collections/community.general.git synced 2024-09-14 20:13:21 +02:00

Allow terraform module to specify complex variable structures (#4797)

* Adding capability to specify complex variables type to terraform

* Terrform variable types are mapped to ansible veriable types

* Currently handles Dict, List, Str, Int, Bool types

* Updated the documentation accordingly

* Updated with an example.

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Wonder how that missed the PEP8 checks :).

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Adding the changelog fragment

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Adding ``integer_types`` from ``module_utils``

Simplified the ``integer_types``,  ``str`` and ``float`` value population through ``json.dumps()``.  Now the strings can have special characters which can break the module execution.

* Update changelogs/fragments/4797-terraform-complex-variables.yml

Co-authored-by: Felix Fontein <felix@fontein.de>

* * Changed to approach to make the code more readble and simple to understand.

    * Maintaining the original for loop for the top_level variables. Therefore the rocess_conplex_args() now only handle second level variables when the type() is either Dict or List.

    * Json dumps are used only for the low level variables. Terraform CLI had issues interpreting escape sequecences from json.dumps()

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* adding boolean explicitly, although boolean is a subclass of integer, adding this for self documentation pupose and the clarity of the code.

* fixing the doc strings

* Update terraform.py

Fixing docstrings

* * Introducing format_args funtion to simplify formatting each argument type for top_level and lower level.

* Terraform Lists of strings, numbers, objects and lists are supported.

* Adding COMMAND: to the fail_json msg, for plan failures to help troubleshoot command line arguments.

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* * Adding full terraform command to fail_json() when the terrafor plan fails
    * Fixing a spelling mistake.

* plan_command if a list, stringifying the list

* * Fixing the new line for the change fragments

* Removed CR (\r) from the output messages. Now output lines carry only LF (\n), not CRLF (\r\n).

* Added integration testing for complex variables.

* Restructured integration testing code to be more expandable.

* Update changelogs/fragments/4797-terraform-complex-variables.yml

Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com>

* double-quotes are not properly escaped in shell, and python string
escaping are nullified the way terraform handle second tier string
variables (within terraform).

* changing all the task actions to FQCN format.

* integration testing now includes:

1. Top level strings containing, special shell characters, spaces,
   double-quotes.

2. Second level strings containing, special shell characters, spaces,
   double-quotes repeating double-quotes to ensure proper regex
substitution.

* Adding colon ':' to string test casses.

* Added complex_vars to switch between the old and the new variable
interpretations.
Updated the documentations to reflect the changes.
Updated the examples.
Handling '\' as well with the escape sequence.

* Added tests for the new escape sequences.
Added multilines tests.

* Restructuring the documente strings to a shorter string.
Argument_spec changed to 'bool'

* Update changelogs/fragments/4797-terraform-complex-variables.yml

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update plugins/modules/cloud/misc/terraform.py

Co-authored-by: Felix Fontein <felix@fontein.de>

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Alexei Znamensky <103110+russoz@users.noreply.github.com>
This commit is contained in:
Kosala Atapattu 2022-10-04 09:10:16 +13:00 committed by GitHub
parent 6fe2a84e87
commit beef93f687
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
7 changed files with 306 additions and 26 deletions

View file

@ -0,0 +1,3 @@
minor_changes:
- terraform - adds capability to handle complex variable structures for ``variables`` parameter in the module.
This must be enabled with the new ``complex_vars`` parameter (https://github.com/ansible-collections/community.general/pull/4797).

View file

@ -80,9 +80,25 @@ options:
aliases: [ 'variables_file' ] aliases: [ 'variables_file' ]
variables: variables:
description: description:
- A group of key-values to override template variables or those in - A group of key-values pairs to override template variables or those in variables files.
variables files. By default, only string and number values are allowed, which are passed on unquoted.
- Support complex variable structures (lists, dictionaries, numbers, and booleans) to reflect terraform variable syntax when I(complex_vars=true).
- Ansible integers or floats are mapped to terraform numbers.
- Ansible strings are mapped to terraform strings.
- Ansible dictionaries are mapped to terraform objects.
- Ansible lists are mapped to terraform lists.
- Ansible booleans are mapped to terraform booleans.
- "B(Note) passwords passed as variables will be visible in the log output. Make sure to use I(no_log=true) in production!"
type: dict type: dict
complex_vars:
description:
- Enable/disable capability to handle complex variable structures for C(terraform).
- If C(true) the I(variables) also accepts dictionaries, lists, and booleans to be passed to C(terraform).
Strings that are passed are correctly quoted.
- When disabled, supports only simple variables (strings, integers, and floats), and passes them on unquoted.
type: bool
default: false
version_added: 5.7.0
targets: targets:
description: description:
- A list of specific resources to target in this plan/application. The - A list of specific resources to target in this plan/application. The
@ -188,6 +204,26 @@ EXAMPLES = """
- /path/to/plugins_dir_1 - /path/to/plugins_dir_1
- /path/to/plugins_dir_2 - /path/to/plugins_dir_2
- name: Complex variables example
community.general.terraform:
project_path: '{{ project_dir }}'
state: present
camplex_vars: true
variables:
vm_name: "{{ inventory_hostname }}"
vm_vcpus: 2
vm_mem: 2048
vm_additional_disks:
- label: "Third Disk"
size: 40
thin_provisioned: true
unit_number: 2
- label: "Fourth Disk"
size: 22
thin_provisioned: true
unit_number: 3
force_init: true
### Example directory structure for plugin_paths example ### Example directory structure for plugin_paths example
# $ tree /path/to/plugins_dir_1 # $ tree /path/to/plugins_dir_1
# /path/to/plugins_dir_1/ # /path/to/plugins_dir_1/
@ -237,6 +273,7 @@ import os
import json import json
import tempfile import tempfile
from ansible.module_utils.six.moves import shlex_quote from ansible.module_utils.six.moves import shlex_quote
from ansible.module_utils.six import integer_types
from ansible.module_utils.basic import AnsibleModule from ansible.module_utils.basic import AnsibleModule
@ -298,7 +335,7 @@ def get_workspace_context(bin_path, project_path):
command = [bin_path, 'workspace', 'list', '-no-color'] command = [bin_path, 'workspace', 'list', '-no-color']
rc, out, err = module.run_command(command, cwd=project_path) rc, out, err = module.run_command(command, cwd=project_path)
if rc != 0: if rc != 0:
module.warn("Failed to list Terraform workspaces:\r\n{0}".format(err)) module.warn("Failed to list Terraform workspaces:\n{0}".format(err))
for item in out.split('\n'): for item in out.split('\n'):
stripped_item = item.strip() stripped_item = item.strip()
if not stripped_item: if not stripped_item:
@ -360,12 +397,25 @@ def build_plan(command, project_path, variables_args, state_file, targets, state
return plan_path, False, out, err, plan_command if state == 'planned' else command return plan_path, False, out, err, plan_command if state == 'planned' else command
elif rc == 1: elif rc == 1:
# failure to plan # failure to plan
module.fail_json(msg='Terraform plan could not be created\r\nSTDOUT: {0}\r\n\r\nSTDERR: {1}'.format(out, err)) module.fail_json(
msg='Terraform plan could not be created\nSTDOUT: {out}\nSTDERR: {err}\nCOMMAND: {cmd} {args}'.format(
out=out,
err=err,
cmd=' '.join(plan_command),
args=' '.join([shlex_quote(arg) for arg in variables_args])
)
)
elif rc == 2: elif rc == 2:
# changes, but successful # changes, but successful
return plan_path, True, out, err, plan_command if state == 'planned' else command return plan_path, True, out, err, plan_command if state == 'planned' else command
module.fail_json(msg='Terraform plan failed with unexpected exit code {0}. \r\nSTDOUT: {1}\r\n\r\nSTDERR: {2}'.format(rc, out, err)) module.fail_json(msg='Terraform plan failed with unexpected exit code {rc}.\nSTDOUT: {out}\nSTDERR: {err}\nCOMMAND: {cmd} {args}'.format(
rc=rc,
out=out,
err=err,
cmd=' '.join(plan_command),
args=' '.join([shlex_quote(arg) for arg in variables_args])
))
def main(): def main():
@ -379,6 +429,7 @@ def main():
purge_workspace=dict(type='bool', default=False), purge_workspace=dict(type='bool', default=False),
state=dict(default='present', choices=['present', 'absent', 'planned']), state=dict(default='present', choices=['present', 'absent', 'planned']),
variables=dict(type='dict'), variables=dict(type='dict'),
complex_vars=dict(type='bool', default=False),
variables_files=dict(aliases=['variables_file'], type='list', elements='path'), variables_files=dict(aliases=['variables_file'], type='list', elements='path'),
plan_file=dict(type='path'), plan_file=dict(type='path'),
state_file=dict(type='path'), state_file=dict(type='path'),
@ -405,6 +456,7 @@ def main():
purge_workspace = module.params.get('purge_workspace') purge_workspace = module.params.get('purge_workspace')
state = module.params.get('state') state = module.params.get('state')
variables = module.params.get('variables') or {} variables = module.params.get('variables') or {}
complex_vars = module.params.get('complex_vars')
variables_files = module.params.get('variables_files') variables_files = module.params.get('variables_files')
plan_file = module.params.get('plan_file') plan_file = module.params.get('plan_file')
state_file = module.params.get('state_file') state_file = module.params.get('state_file')
@ -449,12 +501,77 @@ def main():
if state == 'present' and module.params.get('parallelism') is not None: if state == 'present' and module.params.get('parallelism') is not None:
command.append('-parallelism=%d' % module.params.get('parallelism')) command.append('-parallelism=%d' % module.params.get('parallelism'))
def format_args(vars):
if isinstance(vars, str):
return '"{string}"'.format(string=vars.replace('\\', '\\\\').replace('"', '\\"'))
elif isinstance(vars, bool):
if vars:
return 'true'
else:
return 'false'
return str(vars)
def process_complex_args(vars):
ret_out = []
if isinstance(vars, dict):
for k, v in vars.items():
if isinstance(v, dict):
ret_out.append('{0}={{{1}}}'.format(k, process_complex_args(v)))
elif isinstance(v, list):
ret_out.append("{0}={1}".format(k, process_complex_args(v)))
elif isinstance(v, (integer_types, float, str, bool)):
ret_out.append('{0}={1}'.format(k, format_args(v)))
else:
# only to handle anything unforeseen
module.fail_json(msg="Supported types are, dictionaries, lists, strings, integer_types, boolean and float.")
if isinstance(vars, list):
l_out = []
for item in vars:
if isinstance(item, dict):
l_out.append("{{{0}}}".format(process_complex_args(item)))
elif isinstance(item, list):
l_out.append("{0}".format(process_complex_args(item)))
elif isinstance(item, (str, integer_types, float, bool)):
l_out.append(format_args(item))
else:
# only to handle anything unforeseen
module.fail_json(msg="Supported types are, dictionaries, lists, strings, integer_types, boolean and float.")
ret_out.append("[{0}]".format(",".join(l_out)))
return ",".join(ret_out)
variables_args = [] variables_args = []
for k, v in variables.items(): if complex_vars:
variables_args.extend([ for k, v in variables.items():
'-var', if isinstance(v, dict):
'{0}={1}'.format(k, v) variables_args.extend([
]) '-var',
'{0}={{{1}}}'.format(k, process_complex_args(v))
])
elif isinstance(v, list):
variables_args.extend([
'-var',
'{0}={1}'.format(k, process_complex_args(v))
])
# on the top-level we need to pass just the python string with necessary
# terraform string escape sequences
elif isinstance(v, str):
variables_args.extend([
'-var',
"{0}={1}".format(k, v)
])
else:
variables_args.extend([
'-var',
'{0}={1}'.format(k, format_args(v))
])
else:
for k, v in variables.items():
variables_args.extend([
'-var',
'{0}={1}'.format(k, v)
])
if variables_files: if variables_files:
for f in variables_files: for f in variables_files:
variables_args.extend(['-var-file', f]) variables_args.extend(['-var-file', f])

View file

@ -0,0 +1,35 @@
# Copyright (c) Ansible Project
# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
# SPDX-License-Identifier: GPL-3.0-or-later
resource "null_resource" "mynullresource" {
triggers = {
# plain dictionaries
dict_name = var.dictionaries.name
dict_age = var.dictionaries.age
# list of dicrs
join_dic_name = join(",", var.list_of_objects.*.name)
# list-of-strings
join_list = join(",", var.list_of_strings.*)
# testing boolean
name = var.boolean ? var.dictionaries.name : var.list_of_objects[0].name
# top level string
sample_string_1 = var.string_type
# nested lists
num_from_matrix = var.list_of_lists[1][2]
}
}
output "string_type" {
value = var.string_type
}
output "multiline_string" {
value = var.multiline_string
}

View file

@ -0,0 +1,62 @@
# Copyright (c) Ansible Project
# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
# SPDX-License-Identifier: GPL-3.0-or-later
variable "dictionaries" {
type = object({
name = string
age = number
})
description = "Same as ansible Dict"
default = {
age = 1
name = "value"
}
}
variable "list_of_strings" {
type = list(string)
description = "list of strings"
validation {
condition = (var.list_of_strings[1] == "cli specials\"&$%@#*!(){}[]:\"\" \\\\")
error_message = "Strings do not match."
}
}
variable "list_of_objects" {
type = list(object({
name = string
age = number
}))
validation {
condition = (var.list_of_objects[1].name == "cli specials\"&$%@#*!(){}[]:\"\" \\\\")
error_message = "Strings do not match."
}
}
variable "boolean" {
type = bool
description = "boolean"
}
variable "string_type" {
type = string
validation {
condition = (var.string_type == "cli specials\"&$%@#*!(){}[]:\"\" \\\\")
error_message = "Strings do not match."
}
}
variable "multiline_string" {
type = string
validation {
condition = (var.multiline_string == "one\ntwo\n")
error_message = "Strings do not match."
}
}
variable "list_of_lists" {
type = list(list(any))
default = [ [ 1 ], [1, 2, 3], [3] ]
}

View file

@ -0,0 +1,60 @@
---
# Copyright (c) Ansible Project
# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
# SPDX-License-Identifier: GPL-3.0-or-later
- name: Create terraform project directory (complex variables)
ansible.builtin.file:
path: "{{ terraform_project_dir }}/complex_vars"
state: directory
mode: 0755
- name: copy terraform files to work space
ansible.builtin.copy:
src: "complex_variables/{{ item }}"
dest: "{{ terraform_project_dir }}/complex_vars/{{ item }}"
with_items:
- main.tf
- variables.tf
# This task would test the various complex variable structures of the with the
# terraform null_resource
- name: test complex variables
community.general.terraform:
project_path: "{{ terraform_project_dir }}/complex_vars"
binary_path: "{{ terraform_binary_path }}"
force_init: yes
complex_vars: true
variables:
dictionaries:
name: "kosala"
age: 99
list_of_strings:
- "kosala"
- 'cli specials"&$%@#*!(){}[]:"" \\'
- "xxx"
- "zzz"
list_of_objects:
- name: "kosala"
age: 99
- name: 'cli specials"&$%@#*!(){}[]:"" \\'
age: 0.1
- name: "zzz"
age: 9.789
- name: "lll"
age: 1000
boolean: true
string_type: 'cli specials"&$%@#*!(){}[]:"" \\'
multiline_string: |
one
two
list_of_lists:
- [ 1 ]
- [ 11, 12, 13 ]
- [ 2 ]
- [ 3 ]
state: present
register: terraform_init_result
- assert:
that: terraform_init_result is not failed

View file

@ -9,17 +9,17 @@
- name: Check for existing Terraform in path - name: Check for existing Terraform in path
block: block:
- name: Check if terraform is present in path - name: Check if terraform is present in path
command: "command -v terraform" ansible.builtin.command: "command -v terraform"
register: terraform_binary_path register: terraform_binary_path
ignore_errors: true ignore_errors: true
- name: Check Terraform version - name: Check Terraform version
command: terraform version ansible.builtin.command: terraform version
register: terraform_version_output register: terraform_version_output
when: terraform_binary_path.rc == 0 when: terraform_binary_path.rc == 0
- name: Set terraform version - name: Set terraform version
set_fact: ansible.builtin.set_fact:
terraform_version_installed: "{{ terraform_version_output.stdout | regex_search('(?!Terraform.*v)([0-9]+\\.[0-9]+\\.[0-9]+)') }}" terraform_version_installed: "{{ terraform_version_output.stdout | regex_search('(?!Terraform.*v)([0-9]+\\.[0-9]+\\.[0-9]+)') }}"
when: terraform_version_output.changed when: terraform_version_output.changed
@ -30,7 +30,7 @@
block: block:
- name: Install Terraform - name: Install Terraform
debug: ansible.builtin.debug:
msg: "Installing terraform {{ terraform_version }}, found: {{ terraform_version_installed | default('no terraform binary found') }}." msg: "Installing terraform {{ terraform_version }}, found: {{ terraform_version_installed | default('no terraform binary found') }}."
- name: Ensure unzip is present - name: Ensure unzip is present
@ -39,7 +39,7 @@
state: present state: present
- name: Install Terraform binary - name: Install Terraform binary
unarchive: ansible.builtin.unarchive:
src: "{{ terraform_url }}" src: "{{ terraform_url }}"
dest: "{{ remote_tmp_dir }}" dest: "{{ remote_tmp_dir }}"
mode: 0755 mode: 0755
@ -52,22 +52,16 @@
# path from the 'Check if terraform is present in path' task, and lastly, the fallback path. # path from the 'Check if terraform is present in path' task, and lastly, the fallback path.
- name: Set path to terraform binary - name: Set path to terraform binary
set_fact: ansible.builtin.set_fact:
terraform_binary_path: "{{ terraform_binary_path.stdout or remote_tmp_dir ~ '/terraform' }}" terraform_binary_path: "{{ terraform_binary_path.stdout or remote_tmp_dir ~ '/terraform' }}"
- name: Create terraform project directory
file:
path: "{{ terraform_project_dir }}/{{ item['name'] }}"
state: directory
mode: 0755
loop: "{{ terraform_provider_versions }}"
loop_control:
index_var: provider_index
- name: Loop over provider upgrade test tasks - name: Loop over provider upgrade test tasks
include_tasks: test_provider_upgrade.yml ansible.builtin.include_tasks: test_provider_upgrade.yml
vars: vars:
tf_provider: "{{ terraform_provider_versions[provider_index] }}" tf_provider: "{{ terraform_provider_versions[provider_index] }}"
loop: "{{ terraform_provider_versions }}" loop: "{{ terraform_provider_versions }}"
loop_control: loop_control:
index_var: provider_index index_var: provider_index
- name: Test Complex Varibles
ansible.builtin.include_tasks: complex_variables.yml

View file

@ -3,6 +3,15 @@
# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt) # GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
# SPDX-License-Identifier: GPL-3.0-or-later # SPDX-License-Identifier: GPL-3.0-or-later
- name: Create terraform project directory (provider upgrade)
file:
path: "{{ terraform_project_dir }}/{{ item['name'] }}"
state: directory
mode: 0755
loop: "{{ terraform_provider_versions }}"
loop_control:
index_var: provider_index
- name: Output terraform provider test project - name: Output terraform provider test project
ansible.builtin.template: ansible.builtin.template:
src: templates/provider_test/main.tf.j2 src: templates/provider_test/main.tf.j2