Complex looping using jinja2
In some cases you would create a loop within a loop to perform a complex task that could look like this:
In main.yml:
- name: "Loop to write secure credentials"
ansible.builtin.include_tasks:
file: secure_group.yml
vars:
curr_env2: "{{ lcred[0] }}"
creds: "{{ lcred[1] }}"
loop_control:
loop_var: lcred
loop: |-
[
{%- for key, env_item in code_environment_vars.items() -%}
{%- if env_item.credentials != [] -%}
{%- for cred in env_item.credentials -%}
['{{ key }}',{{ cred }}],
{%- endfor -%}
{%- endif -%}
{%- endfor -%}
]
when: creds.credential_type != 'Machine'
no_log: true
The jinja code in the above task creates a list of comma separated items from a dict, as specified in the automate organization addition.
Link to vars structure
Specificly the list will contain credential items per environment.
And the loop will itterate over this list that is created dynamicly and will include the file with the tasks for every item. This is a loop that is used very much in ansible code, and this is not so efficient, but to be able to run multiple tasks it is sometimes the only way.
In secure_group.yml:
What we did here, is create a vaulted credential in a rather complex manner, and we needed multiple tasks to do it, so we had to use include_tasks.
---
- name: Be sure to clean the file with unencrypted value
block:
- name: Write token to file
ansible.builtin.copy:
content: "{{ creds.encrypt }}"
dest: /tmp/file
mode: "0600"
- name: Encrypt a var and register result
ansible.builtin.command:
argv:
- ./encrypt.sh
- /tmp/file
- dummy
- "{{ team_password }}"
register: _vaulted
changed_when: false
no_log: true
always:
- name: Clean the unencrypted file
ansible.builtin.file:
path: /tmp/file
state: absent
- name: Write the encrypted credential to file
ansible.builtin.blockinfile:
path: "/tmp/{{ team_project_name }}/group_vars/{{ curr_env2 }}/controller_credentials.yml"
insertafter: EOF
marker: ''
block: |
{% filter indent(width=2, first=true) %}
- name: {{ creds.name }}
{% if cred.description is defined %}
description: {{ creds.description }}
{% else %}
description:
{% endif %}
credential_type: {{ creds.credential_type }}
organization: {{ organization_long_name | upper }}
inputs:
{% if creds.credential_type == 'Ansible Galaxy/Automation Hub API Token' %}
auth_url: ''
token: {{ '"{{ ahub_token }}"' }}
url: '{{ creds.url }}'
update_secrets: true
{% endif %}
{% if creds.credential_type == 'Source Control' %}
ssh_key_data: {{ _vaulted.stdout }}
username: {{ creds.username }}
{% endif %}
{% if creds.credential_type == 'Vault' %}
vault_password: {{ _vaulted.stdout }}
{% endif %}
{% if creds.credential_type == 'Container Registry' %}
host: {{ creds.host }}
username: {{ creds.username }}
password: {{ _vaulted.stdout }}
verify_ssl: {{ creds.verify_ssl | lower }}
{% endif %}
{% endfilter %}
- name: Remove empty lines at end of file # noqa: command-instead-of-module
ansible.builtin.shell:
cmd: sed -i -e :a -e '/^\n*$/{$d;N;ba' -e '}' controller_credentials.yml
chdir: "/tmp/{{ team_project_name }}/group_vars/{{ curr_env2 }}"
changed_when: true
Now we know better and merged the above code into 1 task: And it still does the same...
- name: "Loop to write secure credentials" # noqa: jinja[spacing]
ansible.builtin.blockinfile:
path: "/tmp/{{ team_project_name }}/group_vars/{{ lcred[0] }}/controller_credentials.yml"
insertafter: EOF
marker: ''
block: |
{% filter indent(width=2, first=true) %}
- name: {{ lcred[1].name }}
{% if lcred[1].description is defined %}
description: {{ lcred[1].description }}
{% else %}
description:
{% endif %}
credential_type: {{ lcred[1].credential_type }}
organization: {{ organization_long_name | upper }}
inputs:
{% if lcred[1].credential_type == 'Ansible Galaxy/Automation Hub API Token' %}
auth_url: ''
token: {{ '"{{ ahub_token }}"' }}
url: '{{ lcred[1].url }}'
update_secrets: true
{% endif %}
{%- endfilter %}
{% if lcred[1].credential_type == 'Source Control' %}
{% filter indent(width=2, first=true) %}
ssh_key_data: !vault |
{% endfilter %}
{% filter indent(width=10, first=true) %}
{{ lcred[1].encrypt | vault(team_password,inventory_hostname,'') }}
{%- endfilter %}
{% filter indent(width=2, first=true) %}
username: {{ lcred[1].username }}
{%- endfilter %}
{% endif %}
{% if lcred[1].credential_type == 'Vault' %}
{% filter indent(width=2, first=true) %}
vault_password: !vault |
{% endfilter %}
{% filter indent(width=10, first=true) %}
{{ lcred[1].encrypt | vault(team_password,inventory_hostname,'') }}
{%- endfilter %}
{% endif %}
{% if lcred[1].credential_type == 'Container Registry' %}
{% filter indent(width=2, first=true) %}
host: {{ lcred[1].host }}
username: {{ lcred[1].username }}
password: !vault |
{% endfilter %}
{% filter indent(width=10, first=true) %}
{{ lcred[1].encrypt | vault(team_password,inventory_hostname,'') }}
{%- endfilter %}
{% filter indent(width=2, first=true) %}
verify_ssl: {{ lcred[1].verify_ssl | lower }}
{%- endfilter %}
{% endif %}
loop_control:
loop_var: lcred
loop: |-
[
{%- for key, env_item in code_environment_vars.items() -%}
{%- if env_item.credentials != [] -%}
{%- for cred in env_item.credentials -%}
['{{ key }}',{{ cred }}],
{%- endfor -%}
{%- endif -%}
{%- endfor -%}
]
when: lcred[1].credential_type != 'Machine'
no_log: true
As the loop is not altered, we will not explain that.
But the jinja in the blokinfile is rather complex, but if you obide by a few rules, you can write this yourself.
The result of this code is this:
(I deleted some lines, to keep this readable)
The base layout of the file was already templated, this just adds the credentials.
---
controller_credentials:
- name: INFRA_automation_hub_image_pull_secret
description:
credential_type: Container Registry
organization: ORG_INFRA
inputs:
host: rhaap25.homelab
username: ee_pull
password: !vault |
$ANSIBLE_VAULT;1.1;AES256
65316138303936616434353063336661326162303462376434336461323666396637383064616538
3334363164616263650a373932663635633864663139343366343864616464623263386436653632
6565
verify_ssl: false
- name: INFRA_gitlab
description:
credential_type: Source Control
organization: ORG_INFRA
inputs:
ssh_key_data: !vault |
$ANSIBLE_VAULT;1.1;AES256
62616339383061323463616162346534336534326266396662336666333265333163386534646562
3635633330663165633163633165663331643262356530370a336635353066366664363730393435
32626361656666356666306331613633373163383662656664653736653333656232373666363666
613232323138333630326430386333373231
username: AAP_user
- name: INFRA_vault
description:
credential_type: Vault
organization: ORG_INFRA
inputs:
vault_password: !vault |
$ANSIBLE_VAULT;1.1;AES256
66333662343338353062353038326430373161643638636264386336643462373737396664356663
31386565313562366336376665333631653630613034663533656631653930376439
As you can see, all credentials are nicely formatted and vaulted in one loop.The loop will add the credentials in the inventory in a number of directories, that are the environments in the source dict.
Let's breakup the jinja that renders this: Indentation is used here to clarify the code, do not use this in your code...
# We start with the correct indentation for our yaml file
{% filter indent(width=2, first=true) %}
- name: {{ lcred[1].name }}
# an IF statement must complete within the same FILTER
{% if lcred[1].description is defined %}
description: {{ lcred[1].description }}
{% else %}
description:
{% endif %}
credential_type: {{ lcred[1].credential_type }}
organization: {{ organization_long_name | upper }}
inputs:
{% if lcred[1].credential_type == 'Ansible Galaxy/Automation Hub API Token' %}
auth_url: ''
token: {{ '"{{ ahub_token }}"' }}
url: '{{ lcred[1].url }}'
update_secrets: true
{% endif %}
{%- endfilter %}
{% if lcred[1].credential_type == 'Source Control' %}
# the same goes for a FILTER within an IF statement it must complete before ENDIF
{% filter indent(width=2, first=true) %}
ssh_key_data: !vault |
{% endfilter %}
{% filter indent(width=10, first=true) %}
{{ lcred[1].encrypt | vault(team_password,inventory_hostname,'') }}
{%- endfilter %}
{% filter indent(width=2, first=true) %}
username: {{ lcred[1].username }}
{%- endfilter %}
{% endif %}
# Here we start a new IF..
{% if lcred[1].credential_type == 'Vault' %}
{% filter indent(width=2, first=true) %}
vault_password: !vault |
{% endfilter %}
{% filter indent(width=10, first=true) %}
{{ lcred[1].encrypt | vault(team_password,inventory_hostname,'') }}
{%- endfilter %}
{% endif %}
# Here we start a new IF
{% if lcred[1].credential_type == 'Container Registry' %}
{% filter indent(width=2, first=true) %}
host: {{ lcred[1].host }}
username: {{ lcred[1].username }}
password: !vault |
{% endfilter %}
{% filter indent(width=10, first=true) %}
{{ lcred[1].encrypt | vault(team_password,inventory_hostname,'') }}
{%- endfilter %}
{% filter indent(width=2, first=true) %}
verify_ssl: {{ lcred[1].verify_ssl | lower }}
{%- endfilter %}
{% endif %}
This looks like the number of filter statements could be reduced, but ansible-lint will fail with a load error, without a clear error message. Keep these rules in mind when writing complex jinja code.