Using jinja2 instead of loop

loops
jinja2 variables

Loops

In many online examples you will find loops that use "with_items".
This is the old way, it is better to use "loop" according to the best practices.

- name: An old loop using with_items
  ansible.builtin.set_fact:
    small_fact: "[] + [{{ item }}]"
  with_items: "{{ var }}"
  when: "'text' in item"

This will still work, but when using multiple of these with item loops in the same playbook, you will have a variable "item"
that has a value at the start of the loop. It could have some nasty side effects, when this value is used somewhere else in the play.

The new approach looks like this:

- name: A new loop with loop
  ansible.builtin.set_fact:
    small_fact: "[] + [{{ small_var }}]"
  loop: "{{ var }}"
  loop_control:
    loop_var: small_var
  when: "'text' in small_var" 

The above loop is the same as the with_items loop, but if you use different loop_vars, there is no chance of using a value where it shouldn't be.

If loops must itterate over a lot of items, the efficiency is not great and will take a lot of time to complete. Imagine yout playbook has to loop over the output of a previous task and that regisered variable contains 10000 lines.
If you do this with a standard loop (as shown above), it will take a long time.

Example:

- name: Generate 10000 lines of output
  ansible.builtin.shell: |
      for i in $(seq 1 10000);
      do
          echo "${i} this is a line";
      done
  register: _output

- name: Loop over the output to find 55
  ansible.builtin.set_fact:
    line_number: "{{ ansible_loop.index0 | int }}"
  when:
    - "'55' in _line"
  loop: "{{ _output.stdout_lines }}"
  loop_control:
    loop_var: _line
    extended: true
    label: "{{ ansible_loop.index0 }}"

This is a valid playbook and checks out great in code linting and syntax checks.
As with the code it looks nice and structured as we would like to see it.
But, when we execute this playbook, it wil take 10 minutes to complete, giving 9999 times skipped as output, and just 1 OK.

There must be room for improvement here...

The trick here is using a jinja2 filter to parse the output and if we replace the loop with a jinja2 filter, we end up with the following playbook:

-  ansible.builtin.shell: |
      for i in $(seq 1 10000);
      do
          echo "${i} this is a line";
      done
  register: _output

- name: Loop over the output to find 55
  ansible.builtin.set_fact:
    line_number: >-
      {%- for i in range(_output.stdout_lines | length) -%}
        {%- if '55' in _output.stdout_lines[i] -%}
        {{ i | int }}
      {%- endif -%}
      {%- endfor -%}

In the above play, we still loop throug the output, but we do this with a single jinja2 filter, which is much more efficient than an ansible loop.
It also removes all the 9999 non-matching lines in the output.

The performance/runtime difference is vast at these volumes:

playbook	number of records	runtime
loop ansible	10000	600 sec
loop ansible	1000	30 sec
jinja	10000	1,2 sec
jinja	1000	1 sec

WARNING Jinja2 always returns a string value, so formatting is verry important when you need a list or a dict returned.

Example:

In this example we want the same set of files to be templated for every environment.
When you do this in a loop, you generally put the template tasks in a separate file and include this in an inner of two loops (1 for the environments and 1 for the files) to render all files in 1 run.
So you will be using a nested loop.

In the example below, we don't create a nested loop, but generate a string variable, in which all cobinations of both lists are present.
Then we loop over this list to generate the file needed.
There is no nested loop and no include_tasks itteration.
This will improve afficiency a great deal.

- name: Example play
  hosts: localhost
  gather_facts: false
  vars:
    template_files:
      - credentials
      - inventory
      - notifications
      - organization
      - projects
      - roles
      - schedules
      - teams
      - templates
      - users
      - workflows
    environments:
      - dev
      - test
      - accp
      - prod

    tasks:

      - name: Create the loop var
            ansible.builtin.set_fact:
              template_loop: |-
                [
                {%- for env in environments -%}
                  {%- for file in template_files -%}
                  '{{ env }},{{ file }}'
                  {%- if not loop.last -%},
                {%- endif -%}
                {%- endfor -%}
                {%- if not loop.last -%},
                {%- endif -%}
                {%- endfor -%}
                ]

      - name: "Template all files loop"
        ansible.builtin.template:
          src: "{{ curr_file.split(',')[-1] }}.yml.j2"
          dest: "/tmp/{{ project_name }}/group_vars/{{ curr_file.split(',')[0] }}/{{ curr_file.split(',')[-1] }}.yml"
          lstrip_blocks: true
          mode: "0640"
        loop: "{{ template_loop }}"
        loop_control:
          loop_var: curr_file

The "Create the loop var" task, creates a list of all combinations of environments and files.
All fields are comma separated.
A smal piece looks like this:

template_loop: ["dev,credentials","dev,inventory","dev,notifications", .... etc.. ]"

The "Template all files loop" uses a standard single loop. We don't need a include_tasks to create the templated files.
This makes the playbook more readable and more efficient.

The loop_var "curr_file" is filled with the next part of the template_loop variable.
Like: curr_var: "dev,credentials" To be able to use these variables to create the file we need, we must spit them.
We do that with spit, but we directly set the var with the part we want to use there.
For example: - The first part, the environment: "{{ curr_file.split(',')[0] }}"
- The last part, the filename: '"{{ curr_file.split(',')[-1] }}"
If there are more elements, these can be addressed through a zero based index number.

By applying the above examples, the number of tasks will be reduced, performance will increase and the number of yaml files will decrease.
The code will even be more readable.