Configuration As Code (CaC) Pipeline

If you read the title, you might think that something is being recorded in program
code, but of course we don't. It is not given to every administrator to write programs.
Configuration as code is a bit of a misleading term these days. Before the infra
collection was available, this was of course somewhat correct, because then the
ansible code also had to be created to get the configuration into the system. Since
the community has made the collection available for this, the latter is no longer
necessary, and we only record the content.

You can still ask several questions:

• What exactly do we record?
• Where do we record this?
• How do we record this?
• What do we do this for?
• What can we do with it?

We hope to answer these questions in the book and actually go a step further, by
making a statement now:
"why should we backup automation platform"
We are not going to answer that question, you can do that yourself after reading this
book.

In the first instance, we will briefly summarize the answers to the above questions,
the substantive and technical treatment will follow in later chapters.

What exactly do we record?

In configuration as code, we actually record everything that exists in terms of content
in the automation platform. That is, "Anything we can fill in and click on in the user
interface".

Automation platform 2.6 consists of 4 main components: - Automation gateway
- Automation hub
- Automation controller
- Event driven ansible controller

For each component, we try to record everything that is part of the configuration in
.yaml files, so that we can read it with ansible and load it back into the system via an
API, for example. Now that's easier said than done, but here the community has
already taken a lot of work off our hands by making the infra collections available, so
that we no longer have to create that code to load it. Big thanks to the community.

In this book, we are going to make grateful use of the work that the community has
done for us. In the context of "better to copy well than to have to sweat for it
ourselves", we also use the community collections and we supplement them with
some of our own ansible code plus some pipelines, which we will explain in full in this
book.

But we would tell you exactly what we are capturing, there it comes.

Automation gateway

The automation gateway in version 2.6 is the main entrypoint for automation platform and all user configuration is moved from the separate components to the gateway. This changes the configuration as code a lot since each component had its own user configuration.

In gateway we configure the following:
- Organizations
- Teams
- Users
- Access rights
- Authentication providers - Team mappings - Applications - Ports

Automation Hub

The automation hub is the interface for your internal organization to RedHat and
community collections. The automation hub also determines which parts of the redhat
and/or community collections will be made available to the internal organization. Your
own custom collections and Execution Environments are also stored and managed
here. As a rule, an automation hub only exists once per environment (and can be
highly available) and the content can also be different per environment. The
configuration of the automation hub is made for the benefit of the controller(s) that will
be linked to it. This configuration only exists once per environment and is therefore
the same for all controllers in that environment.

As already mentioned, we record this configuration data in yaml files and what does it
contain:
- What repositories are there (both RedHat, Community and own)
- Which config is synced with?
- Which (redhat) token is used
- What exactly is synced
- What custom namespaces are there?
- Custom collections
- Execution environments

For the description of the recording of this data, we refer to a later chapter. For a
description of how automation hub works, see RedHat's online documentation.

Automation controller

The automation controller is the replacement for its predecessor ansible tower. This
is the heart of your automation environment, where playbook runs are started,
planned and monitored. Without the controller (or similar platform), there is no real
automation environment. For an automation controller, we can define the following for
each environment:
- credential types
- credential input sources
- credentials
- execution environments
- hosts
- instance groups
- inventories
- inventory sources
- labels
- Manifest (License)
- notification templates
- projects
- job templates
- schedules
- settings
- teams
- roles
- workflows

An automation controller is suitable to be used by multiple teams (organizations),
each team can require a different setup with different inventories and credentials. As
a result, it is not convenient to record the entire configuration of the controller in 1 set
of files. If you want to make AAP users end-to-end responsible, you have to give all
users access to the same repository and files, this will not give the desired result and
will be a major source of annoyance.

For the above reason, we have split the setup of the controller into several parts:
- Basic configuration
- Team configurations

As you can see later in this book, the basic setup only occurs once, it contains
everything that needs to be set up on a global level, such as licenses, superusers,
organizations, organization_admin accounts and more. We will discuss the content of
this in another chapter.

Event driven ansible controller

Event driven ansible is at the moment of writing rather new. It is used to gather
events from monitoring, evaluating the event and then run a playbook to remediate
the cause of the event. This way we can create a self-healing infrastructure.

For the event-driven ansible controller, a collection is now also available to record the
configuration in code and then load it. In it, we record the following data:
- credentials
- decision_environments
- projects
- rulebook_activations

For the description of the recording of this data, we refer to a later chapter. For a
description of how Event Driven Ansible works, see RedHat's online documentation.

Where do we record this?

In a large IT organization (Enterprise), we want to prevent configuration information
from being spread over many systems and/or departments. Traditionally, IT
departments within organizations were set up according to silos, where a lot of effort
was put into keeping documents and knowledge within the department and hidden
from other departments (indispensability principle). Nowadays, fortunately, this is no
longer the case, because there are generally no passwords in the documents
anymore, fortunately a certain security awareness has ensured that. But it is
precisely this fragmentation of knowledge by the organization that has become
(almost) fatal for many companies. At the time they were needed, the one who knew
'everything' was no longer employed, it had not been transferred... For these
reasons, we really only want one place where the configuration is, a source with the
truth, for which we use a version management system, in this case GIT.

We are talking about the configuration files and not the code. The code is maintained
by the community and for that we only have 1 source of truth and that is ansible
galaxy. We will come back to this in later chapters, which would be a possible setup
for the git repositories, based on gitlab. For other git implementations, it shouldn't be
that difficult to rewrite this with the knowledge you have of that git implementation.

How do we record this?

This is the question we want to answer in this book. The code we use to load the
configuration enforces the standard here and we have to follow it. What we can do is
play around with it before it is picked up by the code. That's what we're going to do in
this book. In principle, each part of the configuration will have its own git repository
from which the configuration can be pushed in code to the system. How this is
structured will be explained in the following chapters.

What do we do this for?

There has never been an infallible system created by humans, so why should we bet
that it will be? That's why we do this, the moment something goes wrong somewhere
and the configuration is lost for whatever reason, we need to be able to fix it as
quickly as possible. So that the organization is not or hardly inconvenienced by the
disruption. As an administrator, you can't explain nowadays that you can't fix a
malfunction within a day. For many organizations, there is a major financial loss if the
automation environment does not work. So cover yourself (not with excuses, but with
recovery code)....

What can we do with it?

As mentioned in the previous section, we can define the configuration and have it
pushed to the system via the code. But we've also said that the configuration will be
set up in parts. If you then lose all configuration because your system has been
completely wiped, you still have a lot of work to do and it also needs to be restored in
the correct order. You can already feel it coming, we are going to automate this too,
so that as a real IT person we can put our feet on the desk and shout:

"Look Mammy, No hands".

We're going to take you on a journey to automated recovery, as far as that's possible
right now.

Data & Environments

What we store in the configuration as code is done separately for each environment,
so that it will not be the same in all areas. If we do that for the credentials of the
automation controller, for example, we can do it as follows:

Statement: "We have 2 automation controller environments"
- dev
- test

These environments are the first 2 steps of an DTAP set-up and will be treated as
such. For these environments, we use 1 git environment as a source of truth.

The credentials contain users and passwords that must be known in the controller for
an environment. How exactly these are in it is not that interesting for the story at the
moment, but what we do with them in our solution for the configuration as code is.
We are trying to explain this clearly here with a simplified example. We have a
number of options to capture this data in such a way that it will appear correctly in
any environment when the configuration is applied.

There are 4 accounts in the controllers with the same name in all environments, but
with different passwords, except the git account, which is the same everywhere.

Username	Passwd
Git	Imayreadall
Ansible	{ssh_key per environment}
hub_token	{token per environment}
vault_pw	a_vault_password

BEWARE!
In this chapter, I will take you along in my journey towards the described solution, so
don t start writing code @option 1. Read the full chapter to be able to follow my
journey.

Option 1:

We create a repository for each environment on the git server, in the CaC group:
- "CaC/dev_controller_config.git"
- "CaC/"test_controller_config.git"

In each repository we then create a file credentials.yml in which the above credentials
are in the correct yaml format (and not like here in the example). Below is the file in
the dev repository:

---
controller_credentials:
  - Name: Git
    Password: Imayreadall
  - name: ansible
    ssh_key: |
      ----start dev key---
      ---end key---

- name: hub_token
    Token: a_token_of_dev_automationhub
  - name: vault_pw
    password: a_vault_password

Below is the file in the test repository:

---
controller_credentials:
  - Name: Git
    Password: Imayreadall
  - name: ansible
    ssh_key: |
      ----start test key---
      ---end key---

- name: hub_token
    Token: a_token_of_test_automationhub
  - name: vault_pw
    password: a_vault_password

In each repository there is the same pipeline that reads the configuration and
configures it in the controller of that particular environment, the only difference
between the pipelines of the two repositories, is the difference in the login data for
the controllers. This is a possibility that can work very well, provided you have the
administration in order and the working methods are also strictly implemented. If any
configuration update is always in all repositories, it will be implemented correctly and
emerged. However, this method is complex, labor-intensive and error-prone. In short,
not so obvious to use, there must be better ways...

This method can be used with some modification with the
infra.controller_configuration

Option 2:

We only create a repository for the configuration and create folders with the files for
the environments based on an inventory structure, so that ansible can read them as
such.

.
└── group_vars
    ├── dev
    │   └── credentials.yml
    └── test
        └── credentials.yml

The content of the files is still the same as it was in the first option, but now you have
created a single source of the data, which is managed in one place. To make a
change, multiple repositories do not need to be updated. So it has already become
simpler and less error-prone. If you also create 2 branches with the names of the
environments in this repository, you also have the option to apply the changes per
environment in your pipeline. This gives the possibility to test the change in the "dev"
environment, before promoting it to the "test" environment. So this set-up already has
a number of advantages over the first option.

This method can be used perfectly with the infra.controller_configuration collection,
without modifications.

But there is still a lot of data "double" in it, which in turn has a chance of error for
differences between the environments. What we would like is for that which must
exist in all environments to be defined even once.

What are we going to do:

We're going to use the possibilities of an inventory, but with its own twist (or piece of
magic, if you will). The structure remains that of an inventory, but we are going to
tinker a bit with the content of the files and the pipeline, before we feed it to the
collection in the format that the collection wants.

We extend the folder structure with an "all" folder, which we also see in almost every
inventory. That folder also has the same function as the one in an inventory,
everything that is here must be everywhere (i.e. in every environment).

.
└── group_vars
    ├── all
    │   └── credentials.yml
    ├── dev
    │   └── credentials.yml
    └── test
        └── credentials.yml

The contents of the files are now different in each folder, we still use the data as
specified in the first option, nothing else changes. Below is the file in the
group_vars/all folder:

---
controller_credentials_all:
  - Name: Git
    Password: Imayreadall
  - name: vault_pw
    password: a_vault_password
Below is the file in the group_vars/dev folder:
---
controller_credentials_dev:
  - name: ansible
    ssh_key: |
      ----start dev key---
      ---end key---

- name: hub_token
    Token: a_token_of_dev_automationhub

Below is the file in the group_vars/test folder:

---
controller_credentials_test:
  - name: ansible
    ssh_key: |
      ----start test key---
      ---end key---

- name: hub_token
    Token: een_token_of_test_automationhub

Now we have purposely placed these files directly below each other, so that they can
easily be compared with each other. We immediately see a difference on the second
line, the name of the variable has been given an addition of the folder (or branch). As
a result, none of the variables can be used directly for the infra collection, but this is
by design.
What is also striking is that the variables that should be the same in all environments
are now in the "all" and nowhere else. So there is only 1 version of this variable. The
variables that are environment-specific are in the environment where they belong and
nowhere else.

In a complete configuration, think about how much data this would save if all files had
this setup.

Anyway, we still have to make this data suitable for the infra collection, otherwise it
will be of no use to us. We do that by making the playbook that we call with our
pipeline, not just the call to the role:

  role: infra.controller_configuration.dispatch

but a pre_task that merges the variables into the desired version:

  set_fact:
    controller_credentials: controller_credentials_all + controller_credentials_[branch]  

By passing the branch to the playbook, it is possible to add the correct variable to the
all, giving it the full contents of the file in option 1. By not using addition, but doing a
merge, it is even possible to change the standard created in the "all" in an
environment, by giving variables in the structure a different value. This allows you to
make optimal use of the inventory principle. Exactly how this was solved can be read
in the pipelines themselves, but the idea is clear.

The variable after the merge with, for example, the "dev" branch:

controller_credentials:
  - Name: Git
    Password: Imayreadall
  - name: ansible
    ssh_key: |
      ----start dev key---
      ---end key---

  - name: hub_token
    Token: a_token_of_dev_automationhub

- name: vault_pw
    password: a_vault_password

This is the structure that has been applied throughout the book, for all variables used
through configuration as code. This is the basis of everything that is mentioned and
discussed in this book. How does the playbook know which environment to
aggregate the variables from? Read on below, where you can find out how the
pipeline and the variables work.

GitLab pipelines

What is a pipeline: A gitlab pipeline is a piece of code that is initiated every update
(push or merge request) of a repository. We say started here explicitly, because that
doesn't mean that anything is done by the code. The configuration and code of a
gitlab pipeline is (by default) in a file ".gitlab-ci.yaml". In order for a gitlab pipeline to
start, a number of conditions must be met in gitlab:
- A GitLab runner must be configured

A runner has to have all commands you ll use installed on the host or in the
container image, depending on the implementation.
- The runner must be linked to the group/project
- The repository should not contain a .gitlab-ci.yaml with the pipeline code, this code should be in a separate project for security reasons.

.GitLab-ci-yaml

The actions performed by a pipeline are located by default in the .gitlab-ci.yaml file in
the repository. In this file, we define when which action should be performed. The
pipeline is triggered with every update of the repository and depending on the content
of the .gitlab-ci.yaml, actions will be performed.
In the example below, the gitlab runner that executes this, is on a host (or vm), there
is no need for an image tag.

Here's an example:

# List of pipeline stages
stages:
  - Verify ansible code

Verify_ansible_code:
  stage: Verify ansible code
  script:
    - ansible-lint

In this case, the pipeline only has 1 "stage" and it is always executed. The command
that executes the pipeline can be found under "script", in this case "ansible-lint". This
is about the simplest form of a pipeline that can be made. However, this form also
has the greatest risk of performance problems. Because the pipeline is always
running with every update, when used by many teams, the runner will quickly be
overloaded by the large number of updates. It is therefore more convenient to temper
it a bit here.

In the example below the gitlab runner is on a container platform like Kubernetes or
docker, hence the image tag is now needed to pull an image.

Another example:

# Defaults
image: docker.homelab:5000/cac-image:latest

# List of pipeline stages
stages:
  - Verify ansible code
only:
  - dev

Verify_ansible_code:
  stage: Verify ansible code
  script:
    - ansible-lint

In the example above, each update starts a new container on OpenShift (this is
because the gitlab-runner is implemented there) that is based on the rh-python
image. This can also be done on a docker container platform, the operation is exactly
the same. However, the image must then be retrieved from another registry. That
image is a minimal Linux container with ansible installed in it, so that it is suitable for
running ansible code (see elsewhere how these images are created).
In this case, we use the same pipeline definition as above, only now we don't always fire it at every
update, but only at an update in the "dev" branch, this prevents that when updates to
a feature branch or a merge request to, for example, the test environment, the code
is not needlessly run through the linting again. By combining different stages, one
can Create a highly complex pipeline that can perform many tasks automatically.

In the above example a cac-image is used.
To create the (docker)image, see here: cac-image

The keywords "stage" and "script" are reserved words in a gitlab pipeline definition.
Multiple stages can be created and dependencies between stages can be
determined. There are too many possibilities to describe here. All documentation can
be found on the gitlab.com website. We're going to show another version of the
pipeline, and that's the one we use for most repositories. With the knowledge gained
here, all the pipelines in this book can be read and explained.

A more advanced pipeline is below:
We've talked about DTAP before. We therefore want this order to be enforced in the
pipeline, where necessary. In order to enforce this sequence in the pipeline
completely correctly and only perform a merge upon successful execution, a number
of settings in gitlab need to be adjusted compared to the default settings. Also, for an
Enterprise environment, at least a premium/Enterprise version is required.

configure_from_merge

In the event of an update to the git repository where this pipeline is located, the
pipeline will be triggered, but will not do anything when creating a new (feature)
branch. Only when the new branch is brought to the dev branch via a merge request,
the code will be executed after the merge button is pushed. A specific order is not
really enforced here, but the pipeline is only run on the commit after a merge request. The advised sequence should be this:
- New branch to dev
- From dev to test
- From test to accp
- From accp to prod

configure_from_trigger

The code that is here is not executed during a code update, but under another
condition, which we will come back to later in the chapter recovery.

# Pull the ansible config as code image
image: docker.homelab:5000/cac-image:1.3

# List of pipeline stages
stages:
 - Configure automation-platform

configure_from_merge:
  tags:
    - shared
  stage: Configure automation-platform
  rules:
    - if: '($CI_COMMITT_BRANCH == "dev" || 
            $CI_COMMITT_BRANCH == "test" ||
            $CI_COMMITT_BRANCH == "accp" ||
            $CI_COMMITT_BRANCH == "prod") &&
            && $CI_PIPELINE_SOURCE == "push" &&
            && $CI_COMMIT_MESSAGE =~ /Merge branch/i'
  script:
    - echo "From pipeline - Start rhaap configuration on '$CI_COMMIT_BRANCH' 
Environment"
    - ansible-playbook main.yml
      -i inventory.yaml
      -e instance=aap_$CI_COMMIT_BRANCH
      -e branch_name=$CI_COMMIT_BRANCH
      --vault-password-file <(echo ${VAULT_PASSWORD})

configure_from_trigger:
  tags:
    - shared
  stage: Configure automation-platform
  rules:
    - if: '$CI_PIPELINE_SOURCE == "pipeline"'
  script:
    - echo "Pipeline triggered by '$CI_PIPELINE_SOURCE' ref"
    - echo "From pipeline - Start controller recovery on '$CI_COMMIT_REF_NAME' 
Environment"
    - ansible-playbook main.yml
      -i inventory.yaml
      -e instance=aap_$CI_COMMIT_REF_NAME
      -e branch_name=$CI_COMMIT_REF_NAME
      --vault-password-file <(echo ${VAULT_PASSWORD})

What is remarkeble (if you look closely) is that the script sections are slightly
different in detail. We are going to explain one of the two, the other you should be
able to read with the explanation of the first. Some of the variables used are different,
this is because of the time at which the code is executed. Let's take the bit aside for a
moment, that's a bit clearer:

  script:
    - echo "Pipeline triggered by '$CI_PIPELINE_SOURCE' ref"
    - echo "From pipeline - Start rhaap recovery on '$CI_COMMIT_REF_NAME' 
Environment"
    - ansible-playbook main.yml
      -i inventory.yaml
      -e instance=aap_$CI_COMMIT_REF_NAME
      -e branch_name=$CI_COMMIT_REF_NAME
      --vault-password-file <(echo ${VAULT_PASSWORD})

This is the part of the pipeline that does the actual work, all that remains is the
prerequisite. We'll first explain the variables in each part of the pipeline, which will
make things a lot clearer.

Variable name	origin	Definition
CI_COMMIT_REF_NAME	gitlab commit	This is always filled by gitlab itself when pushing/merging to a branch
CI_MERGE_REQUEST_TARGET_BRANCH_NAME	gitlab merge	In the case of a merge request, this variable has the name of the target branch in it
CI_MERGE_REQUEST_SOURCE_BRANCH_NAME	gitlab merge	In the case of a merge request, this variable contains the name of the origin branch
CI_PIPELINE_SOURCE	GitLab CI/CD	This variable indicates the source of a pipeline run, which can be a merge request, push or a pipeline event
VAULT_PASSWORD	ci/cd fresh	Because secrets always have to be encrypted with ansible vault, we give the vault password to the playbook in this way, which is also used in the logs. This vault password can be set per repository, which means that all secrets are stored in different repositories with a different vault.

Are we going to replace the variable with their values in the code "as if we were the
pipeline", this gives the following result:

echo "Pipeline triggered by 'dev' ref"
echo "From pipeline - Start rhaap recovery on 'dev' Environment"
ansible-playbook main.yml -i inventory.yaml -e instance=aap_dev -e
branch_name=dev --vault-password-file <(echo IamthePassword)

Then, all of a sudden, it seems like a simple piece of bash code generated by the
pipeline, and in the end, it is.
In each chapter the pipeline is repeated a bit, if you
understand what has been explained above, you can skip it in most cases.

That's all you need in terms of pipeline knowledge, with the examples shown above it
is possible to capture many automation questions in a pipeline. At least as far as
ansible automation platform is concerned and much more.

Back

Back to Site