External Vault

Secrets must be kept in a safe place (not unencrypted in GIT).
When starting with automation, it can be a temporary solution to store secrets in GIT that are encrypted, using ansible vault.
But when automation grows, the maintenance is growing and will lead to playbook errors and so on, because may be not aall credentials are correctly updated or even stored in a single source of truth.
This is where an external vault comes to the rescue, if correctly implemented.

In this example, described on this site, we use openbao, a opensource vault like HashiCorp Vault.

Install Openbao:
Implementing OpenBao

organizing secrets

Once openbao is installed, we can store secrets, but how do we separate the secrets into organizations, because when you create a secret in the root, it will be accessable for all that connect to the vault. We have created a rhaap installation with 2 environments and multiple organizations, this should be replicated in the vault, to be able to separate the secrets between organizations.

The power of keeping this organized in a standardized structure, is that you can program access to the credentials, if you do not organize you credentials in the vault,this will become a daunting task to automate this.

Example:

dev:
  base:
  org1:
  org2:
  org3:
prod:
  base:
  org1:
  org2:
  org3:

In the above example, we have created 2 so called "namespaces" from the root. These are the rhaap environments we created. As the secrets must be different id each environment, we created separate namespaces for them.
In each environment namespace, we create a namespace for each organization to store credentials for the organization specific. Credentials that must be shared in the whole environment, can be created in the environment namespace. The "base" namespace is special, there is no "base" organization, this is where we store secrets we need for the base configuration. So we pass the "org_name = base" variable to the pipeline.

The names for some credentials that are present in the pipelines for the organizations, must have the same name, see the below example:

Example 2:

dev:
  base:
    base_users:
      coll_get: this_is_the_password
      org1-admin: otherdevpass
      org2-admin: thirddevpass
      org3-admin: secretdevpass
  org1:
    admin_user:
      username: org1-admin
      password: otherdevpass
  org2:
    admin_user:
      username: org2-admin
      password: thirddevpass
  org3:
    admin_user:
      username: org3-admin
      password: secretdevpass
prod:
  base:
    base_users:
      coll_get: prod_password
      org1-admin: otherprodpass
      org2-admin: secretprodpass
      org3-admin: verysecretprodpass
  org1:
    admin_user:
      username: org1-admin
      password: otherprodpass
  org2:
    admin_user:
      username: org2-admin
      password: secretprodpass
  org3:
    admin_user:
      username: org3-admin
      password: verysecretprodpass

The name of this credential is the same every time, but the content is different, so each environment and every organization have the credential available which can be used in a pipeline. Imagine these to be the credentials for the organization admin for the configuration as code for each organization.
By just varying the namespace and environment as vaiables in the pipeline, you can get the correct account for the organization from the vault.
The base is a special, it holds all the local automation paltform users together with their passwords, to be able to create the gateway_user_accounts variable and create all users in one run.

The use of this external vault setup is documented on this site.