template: Add option to lstrip_blocks' and fix settingtrim_blocks` inline (#37478)

* template: Add integration tests for `lstrip_blocks'

Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>

* template: Fix passing `trim_blocks' inline

Fix passing `trim_blocks' option to the template module as inline
argument. Previously passing the `trim_blocks' option inline instead of
using the YAML dictionary format resulted in it always being set to
`True', even if `trim_blocks=False' was used.

Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>

* template: Add option to `lstrip_blocks'

Add option to set `lstrip_blocks' when using the template module to
render Jinja templates. The Jinja documentation suggests that
`trim_blocks' and `lstrip_blocks' is a great combination and the
template module already provides an option for `trim_blocks'.

Note that although `trim_blocks' in Ansible is enabled by default since
version 2.4, in order to avoid breaking things keep `lstrip_blocks'
disabled by default. Maybe in a future version it could be enabled by
default.

This seems to address issue #10725 in a more appropriate way than the
suggested.

Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>

* template: Add integration tests for `trim_blocks'

Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>

* template: Check Jinja2 support for `lstrip_blocks'

Since the `lstrip_blocks' option was added in Jinja2 version 2.7, raise
an exception when `lstrip_blocks' is set but Jinja2 does not support it.
Check support for `lstrip_blocks' option by checking `jinja2.defaults'
for `LSTRIP_BLOCKS' and do not use `jinja2.__version__' because the
latter is set to `unknown' in some cases, perhaps due to bug in
`pkg_resources' in Python 2.6.6.

Also update option description to state that Jinja2 version >=2.7 is
required.

Signed-off-by: Alex Tsitsimpis <alextsi@arrikto.com>

lib/ansible/modules/files/template.py

@@ -74,6 +74,13 @@ options:
     type: bool
     default: 'no'
     version_added: '2.4'
+  lstrip_blocks:
+    description:
+      - If this is set to True leading spaces and tabs are stripped from the start of a line to a block.
+        Setting this option to True requires Jinja2 version >=2.7.
+    type: bool
+    default: 'no'
+    version_added: '2.6'
   force:
     description:
       - the default is C(yes), which will replace the remote file when contents

lib/ansible/plugins/action/template.py

@@ -53,7 +53,20 @@ class ActionModule(ActionBase):
         variable_end_string = self._task.args.get('variable_end_string', None)
         block_start_string = self._task.args.get('block_start_string', None)
         block_end_string = self._task.args.get('block_end_string', None)
-        trim_blocks = self._task.args.get('trim_blocks', None)
+        trim_blocks = boolean(self._task.args.get('trim_blocks', True), strict=False)
+        lstrip_blocks = boolean(self._task.args.get('lstrip_blocks', False), strict=False)
+
+        # Option `lstrip_blocks' was added in Jinja2 version 2.7.
+        if lstrip_blocks:
+            try:
+                import jinja2.defaults
+            except ImportError:
+                raise AnsibleError('Unable to import Jinja2 defaults for determing Jinja2 features.')
+
+            try:
+                jinja2.defaults.LSTRIP_BLOCKS
+            except AttributeError:
+                raise AnsibleError("Option `lstrip_blocks' is only available in Jinja2 versions >=2.7")
 
         wrong_sequences = ["\\n", "\\r", "\\r\\n"]
         allowed_sequences = ["\n", "\r", "\r\n"]
@@ -108,8 +121,8 @@ class ActionModule(ActionBase):
                     self._templar.environment.variable_start_string = variable_start_string
                 if variable_end_string is not None:
                     self._templar.environment.variable_end_string = variable_end_string
-                if trim_blocks is not None:
-                    self._templar.environment.trim_blocks = bool(trim_blocks)
+                self._templar.environment.trim_blocks = trim_blocks
+                self._templar.environment.lstrip_blocks = lstrip_blocks
 
                 # add ansible 'template' vars
                 temp_vars = task_vars.copy()
@@ -133,6 +146,7 @@ class ActionModule(ActionBase):
             new_task.args.pop('variable_start_string', None)
             new_task.args.pop('variable_end_string', None)
             new_task.args.pop('trim_blocks', None)
+            new_task.args.pop('lstrip_blocks', None)
 
             local_tempdir = tempfile.mkdtemp(dir=C.DEFAULT_LOCAL_TMP)
 

test/integration/targets/template/files/lstrip_blocks_false.expected

@@ -0,0 +1,4 @@
+    hello world
+        hello world
+        hello world
+    

test/integration/targets/template/files/lstrip_blocks_true.expected

@@ -0,0 +1,3 @@
+hello world
+hello world
+hello world

test/integration/targets/template/files/trim_blocks_false.expected

@@ -0,0 +1,4 @@
+
+Hello world
+
+Goodbye

test/integration/targets/template/files/trim_blocks_true.expected

@@ -0,0 +1,2 @@
+Hello world
+Goodbye

test/integration/targets/template/tasks/main.yml

@@ -120,6 +120,93 @@
         - 'import_as_with_context_diff_result.stdout == ""'
         - "import_as_with_context_diff_result.rc == 0"
 
+# VERIFY trim_blocks
+
+- name: Render a template with "trim_blocks" set to False
+  template:
+    src: trim_blocks.j2
+    dest: "{{output_dir}}/trim_blocks_false.templated"
+    trim_blocks: False
+  register: trim_blocks_false_result
+
+- name: Get checksum of known good trim_blocks_false.expected
+  stat:
+    path: "{{role_path}}/files/trim_blocks_false.expected"
+  register: trim_blocks_false_good
+
+- name: Verify templated trim_blocks_false matches known good using checksum
+  assert:
+    that:
+        - "trim_blocks_false_result.checksum == trim_blocks_false_good.stat.checksum"
+
+- name: Render a template with "trim_blocks" set to True
+  template:
+    src: trim_blocks.j2
+    dest: "{{output_dir}}/trim_blocks_true.templated"
+    trim_blocks: True
+  register: trim_blocks_true_result
+
+- name: Get checksum of known good trim_blocks_true.expected
+  stat:
+    path: "{{role_path}}/files/trim_blocks_true.expected"
+  register: trim_blocks_true_good
+
+- name: Verify templated trim_blocks_true matches known good using checksum
+  assert:
+    that:
+        - "trim_blocks_true_result.checksum == trim_blocks_true_good.stat.checksum"
+
+# VERIFY lstrip_blocks
+
+- name: Check support for lstrip_blocks in Jinja2
+  shell: "{{ ansible_python.executable }} -c 'import jinja2; jinja2.defaults.LSTRIP_BLOCKS'"
+  register: lstrip_block_support
+  ignore_errors: True
+
+- name: Render a template with "lstrip_blocks" set to False
+  template:
+    src: lstrip_blocks.j2
+    dest: "{{output_dir}}/lstrip_blocks_false.templated"
+    lstrip_blocks: False
+  register: lstrip_blocks_false_result
+
+- name: Get checksum of known good lstrip_blocks_false.expected
+  stat:
+    path: "{{role_path}}/files/lstrip_blocks_false.expected"
+  register: lstrip_blocks_false_good
+
+- name: Verify templated lstrip_blocks_false matches known good using checksum
+  assert:
+    that:
+        - "lstrip_blocks_false_result.checksum == lstrip_blocks_false_good.stat.checksum"
+
+- name: Render a template with "lstrip_blocks" set to True
+  template:
+    src: lstrip_blocks.j2
+    dest: "{{output_dir}}/lstrip_blocks_true.templated"
+    lstrip_blocks: True
+  register: lstrip_blocks_true_result
+  ignore_errors: True
+
+- name: Verify exception is thrown if Jinja2 does not support lstrip_blocks but lstrip_blocks is used
+  assert:
+    that:
+      - "lstrip_blocks_true_result.failed"
+      - 'lstrip_blocks_true_result.msg is search(">=2.7")'
+  when: "lstrip_block_support is failed"
+
+- name: Get checksum of known good lstrip_blocks_true.expected
+  stat:
+    path: "{{role_path}}/files/lstrip_blocks_true.expected"
+  register: lstrip_blocks_true_good
+  when: "lstrip_block_support is successful"
+
+- name: Verify templated lstrip_blocks_true matches known good using checksum
+  assert:
+    that:
+        - "lstrip_blocks_true_result.checksum == lstrip_blocks_true_good.stat.checksum"
+  when: "lstrip_block_support is successful"
+
 # VERIFY CONTENTS
 
 - name: check what python version ansible is running on

test/integration/targets/template/templates/lstrip_blocks.j2

@@ -0,0 +1,8 @@
+{% set hello_world="hello world" %}
+{% for i in [1, 2, 3] %}
+    {% if loop.first %}
+{{hello_world}}
+    {% else %}
+{{hello_world}}
+    {% endif %}
+{% endfor %}

test/integration/targets/template/templates/trim_blocks.j2

@@ -0,0 +1,4 @@
+{% if True %}
+Hello world
+{% endif %}
+Goodbye