How to secure the ansible key
When we use RedHat Ansible Automation Platform (RHAAP) or AWX, we need an ansible private
key on all systems managed by RHAAP or AWX. If poorly configured we can potentially
risk to give root permissions to any user with access to RHAAP or AWX.
With the ansible private key there is a lot of power available to the potential hacker/user. In many organizations it gives access to systems and even more to the root privileges.
In this document we describe the risks, and the mittigation of those risks, using
ansible and RHAAP.
The risk
As most system admins know, with a private_ssh_key exposed, we have a serious problem.
Even more, when this key gives accces to all hosts and passwordless sudo.
This would compromise the entire production environment. In my experience there are many
companies at risk.
The documentation of RHAAP tells us the key can't be decrypted from the controller or a
playbook running, that is correct. But the key is also present on the system being
targetted by a playbook. And if I can run playbook as the ansible user (wich is mostly the
case), I can create a playbook to:
- copy the private_keyfile to /tmp
- make the file readable to the world
- cat the file
This way I get the content off the private key file in the output of my playbook to copy
anyware. If I copy this key to a linux host I have access to, I can configure ssh so, that
this key will be used to connect to other linux hosts as the "ANSIBLE" user with root access.
without ever being asked for a password.....
The vaulted solution
To add several layers of security, follow the steps below, ensuring your ansible setup keeps working.
Add a complex passphrase to the ansible private key
As the ansible keypair is often only created once (and never replaced), this should be done right
the first time. Using a complex passphrase wil enhance security greatly. RHAAP can use this out of
the box, adding the passphrase to the configuration of the ansible credential.
In configuration as code this is done by the ssh_key_unlock option (see the example below for more info).
- ssh_key_unlock
When we now extract the private key with an ansible playbook, we still get the key, but we won't
get the passphrase to unlock the key. Using the key has become virtually impossible, even when we
could brute force the passphrase.
Ensure the ansible account has a "complex" password.
Ensure the complex password setting for the ansible user, be sure to make the password different
than the passphrase for the key, it would add nothing to security if you use the same password.
Only the complex password doesn't help without changing a bit of configuration in the system.
The sudoers file for the ansible user must (as per hardening guideline) never contain the "NOPWD"
directive. So, if its there, remove it! Ans add a complex password to the ansible account, this will be used as the
become_password from rhaap.
Configure the new passwords in your RHAAP/AWX environments, ensuring each environment in Your DTAP has an unique set.
To configure this through the infra.controller_configuration:
- name: ansible
description: This is a secure ansible credential using passwords on all options
credential_type: Machine
organization: Default
inputs:
become_method: sudo
become_username: ''
ssh_key_data: !vault |
93284210394338921830912487394834908490484109
32094812348921234894679847894728974892748466
13838930293839218393801980102198301380380381
93284210394338921830912487394834908490484109
32094812348921234894679847894728974892748466
13838930293839218393801980102198301380380381
93284210394338921830912487394834908490484109
32094812348921234894679847894728974892748466
13838930293839218393801980102198301380380381
93284210394338921830912487394834908490484109
32094812348921234894679847894728974892748466
13838930293839218393801980102198301380380381
93284210394338921830912487394834908490484109
32094812348921234894679847894728974892748466
13838930293839218393801980102198301380380381
3294890381293081203
username: ansible
ssh_key_unlock: !vault |
09238430297893823092138398130921381209381903
12098232109381203912830913810938210938109380
23098487093481
become_password: !vault |
93284210394338921830912487394834908490484109
32094812348921234894679847894728974892748466
13838930293839218393801980102198301380380381
3294890381293081203
Ensure all data is vaulted in your git repositories, it helps with security.
The external vault solution
An even better option is storing your key data and passwords in an external vault, like HashiCorp or other..
then the above file may look like this:
- name: ansible
description: This is a secure ansible credential using passwords on all options
credential_type: Machine
organization: Default
As you can see, there is no definition of the inputs for the credential.
There is some additional configuration needed to make this work in configuration as code.
You will need to configure the following:
- A credential to access the external secrets vault
- The credential with name and type (like the example above).
- The credential input fields (in controller_credential_input_sources.yml).
As an example we will show you how the ansible credential could be defined to use the hashicorp vault.
The external vault credential
To be able to access the external vault we need a credential:
In controller_credentials.yml we need to add the following credential for hashicorp:
controller_credentials:
- name: Default_hashivault
description: HashiCorp Vault Secret Lookup example using token auth
organization: Default
credential_type: HashiCorp Vault Secret Lookup
inputs:
url: "{{ vault_url }}"
token: "{{ vault_token }}"
namespace: "{{ vault_namespace }}"
api_version: v1
default_auth_path: token
As you can see, we have a number of variables in this definition: These variables will be added by the pipeline, so we don't have these credentials in the definition.
| Variable | Description |
|---|---|
| vault_url | The url of the vault server, including the port |
| vault_token | The token with permissions to read the vault secrets |
| vault_namespace | The namespace in the vault where the sectrets are defined |
In controller_credentials.yml we add the definition of the credential without inputs:
- name: ansible
description: This is a secure ansible credential using passwords on all options
credential_type: Machine
organization: Default
In the file controller_credential_input_sources.yml we add a source for every input field for each credential we want to pull from the secrets vault.
- source_credential: Default_hashivault
target_credential: ansible
input_field_name: ssh_key_data
description: Fill the ansible ssh_key from HashiCorp Vault
metadata:
secret_backend: kv
secret_path: data/ansible
auth_path: token
secret_key: ssh_private_key
- source_credential: Default_hashivault
target_credential: ansible
input_field_name: username
description: Fill the ansible username from HashiCorp Vault
metadata:
secret_backend: kv
secret_path: data/ansible
auth_path: token
secret_key: username
- source_credential: Default_hashivault
target_credential: ansible
input_field_name: become_method
description: Fill the ansible become method from HashiCorp Vault
metadata:
secret_backend: kv
secret_path: data/ansible
auth_path: token
secret_key: become_method
- source_credential: Default_hashivault
target_credential: ansible
input_field_name: become_password
description: Fill the ansible become password from HashiCorp Vault
metadata:
secret_backend: kv
secret_path: data/ansible
auth_path: token
secret_key: become_password
- source_credential: Default_hashivault
target_credential: ansible
input_field_name: ssh_key_unlock
description: Fill the ansible key passphrase from HashiCorp Vault
metadata:
secret_backend: kv
secret_path: data/ansible
auth_path: token
secret_key: passphrase
For each input field in the credential you want pulled for the secrets vault there needs to be a definition. Fields that are not used, won't need to be defined.
Each input definition consists of the following:
| Key | Description |
|---|---|
| source_credential | The name of the credential that gives access to the external secrets vault. |
| target_credential | The name of the credential to write the field to |
| input_field_name | The name of the input field to fill in |
| description | A free format text decription for this input field |
| metadata | The dictionary holding the secret path to read |
| secret_backend | What engine was used to create the secret in hashicorp vault |
| secret_path | The path in the secret store to the secret ( includes the secret name ) |
| auth_path | The authentication method to use for the vault, if you defined token as default, this can be omitted |
| secret_key | what key to retrieve from the secret in the secret vault |
This file will become huge with many credentials in the external store, but is essential for security.
SSH certificates
Even better is to have automation platform use certificate based SSH logins, this requires a signed certificate to login to a server.
By keeping the certificate lifetime a short as possible, security is made simple, no keys scattered all around. Enabled users can
just have their ssh key signed and are able to login, with restrictions applied, if needed.