It’s inevitable that a growing organization actively using Red Hat Ansible Automation Platform will face the challenge of managing the configuration of the Ansible Automation Platform as a code. The reasons leading to that may include various requirements for change management, consistency, and repeatability, to name a few.
There is plenty of information on the configuration as code (CaC) approach, such as the benefits and challenges, but reiterating them is outside of the scope of this article. Going forward, we assume that the reader is well aware of why CaC is needed and how this approach can make a positive impact on the whole organization.
If you're responsible for managing Ansible Automation Platform and have been tasked with implementing CaC, this series may be useful.
Part 1 (this article) is about first steps on the path of managing an existing Ansible Automation Platform instance as CaC: setting up Ansible Automation Platform accounts, collections, credentials, projects, and job templates required to run the automation, exporting configuration of some objects, handling secrets and special strings in the CaC, and managing configuration drift.
Part 2 will discuss completing the transition: exporting all objects, formatting configurations for readability, verification, access restrictions, and Git management.
Part 3 will deal with migrating configurations from AWX 24 to Ansible Automation Platform 2.5.
Part 4 will discuss migrating smart inventories (deprecated in future Ansible Automation Platform releases) to constructed inventories.
Part 5, the final article in this series, will discuss migrating configurations from Ansible Automation Platform 2.4 to Ansible Automation Platform 2.5.
Collections requirements
For the purposes of this article, we assume the following:
- You are using Ansible Automation Platform version 2.4 or 2.5 with Private Automation Hub.
- You have a basic understanding of Ansible Automation Platform and its configuration via the web interface.
- You are familiar with Ansible collections and their role in automation workflows.
For mature organizations, the luxury of starting the Ansible Automation Platform configuration from scratch and using CaC right from the beginning is rare. Most likely there is already an existing instance of Ansible Automation Platform with hundreds or even thousands of different objects.
Therefore, the very first requirement is the ability to export existing configurations in a structured format that can be used to maintain platform configuration directly or with minimal adjustments.
To keep configurations up to date, the obvious requirement will be an ability to manage configuration drift. In other words, we need to be able to remove objects that are not needed anymore by removing them from the code, without the need to create additional cleanup playbooks.
To cover both requirements, we will use the configify.aapconfig collection available from ansible-galaxy.
Let’s start by creating an empty Git repository in the code management system of your choice and creating a collections/requirements.yml file with the following content:
---
collections:
- name: configify.aapconfigAssuming that the Ansible Automation Platform controllers are able to access the internet and that Private Hub is synchronizing Red Hat Certified Collections, this file should be enough to satisfy all collection requirements.
For isolated environments (or if certified collections are missing in the Private Automation Hub), the following collections will need to be either synchronized to the Hub or imported there manually:
ansible.controller(Red Hat Hub)infra.ah_configuration(Red Hat Hub)ansible.platform(Red Hat Hub)ansible.utils(Galaxy)ansible.hub(Galaxy)
Consult the most recent version of the collection’s galaxy.yml file for specific version requirements.
Collection synchronization or import is outside of the scope of this article. Check the official Red Hat documentation for detailed steps.
At this point, all the required collections should be downloaded automatically during the project synchronization, assuming that the Ansible Automation Platform organization we are going to use for CaC jobs has been assigned the correct credentials for accessing Galaxy and Private Automation Hub.
We will create a project during the following steps.
Accounts and tokens requirements
To configure the Ansible Automation Platform controller and hub, you must have an account with full administrative access. In Ansible Automation Platform 2.4 there are two separate sets of accounts for each of the components. The recommended practice is to have a separate account for automation tasks. So we will need two new administrative accounts for Ansible Automation Platform 2.4.
Once you've created those accounts, let’s generate API tokens for them. In Controller and Gateway, this is done from User Details | Tokens. Tokens should have read/write scope. Make sure to copy the values as they will be shown only once. We will use them in Ansible Automation Platform credentials during the following steps.
Unfortunately for Private Hub, we will have to use passwords for now. As of the time of writing, some of the dependency collections do not support tokens for some modules.
The recommendation for separate automation accounts is driven by manageability and auditing considerations. For Hub, since token retrieval operation resets the token, a separate account reduces the risk of token invalidation by another process or user.
The choice to use API tokens instead of account passwords helps to keep Ansible Automation Platform configuration tasks independent from any account password rotation, simplifies management, and improves security, since there is no need to store token values once they’ve been supplied to a credential. At the same time, this choice simplifies secret rotation which you can do by generating a new token, replacing it in automation credentials at your convenience while the old token is still valid, and deleting the old token later.
Ansible configuration requirements
To finalize the preparation, let’s log in to Ansible Automation Platform and create the following:
Credential of type Red Hat Ansible Automation Platform for accessing Controller, specifying Controller URL, and API token created at the previous step.
For Ansible Automation Platform 2.5, the controller URL should be set to Gateway URL because of unified authentication.
Credential type for future hub credential.
In the input field, specify variables for URL, username, and token:
fields: - id: hub_url type: string label: HubURL - id: hub_user type: string label: HubUser - id: hub_password type: string label: HubPassword secret: trueIn the Injector Configuration field, expose these variables as environment variables:
env: AH_HOST: '{{ hub_url }}' AH_USERNAME: '{{ hub_user }}' AH_PASSWORD: '{{ hub_password }}'To be able to connect to nodes using self-signed SSL certificates, you may need to expose
AH_VERIFY_SSL. For Ansible Automation Platform 2.5, make sure that the Hub URL is set to Gateway URL.- Credential of the type created at the previous step, specifying the Hub URL and the automation username and password.
For Ansible Automation Platform 2.5, we will need credential type and credential for Gateway. Follow the previous two steps and expose the following environment variables:
GATEWAY_HOSTNAME GATEWAY_API_TOKEN (GATEWAY_VERIFY_SSL)- An inventory without any hosts. Ansible Automation Platform configuration jobs will run against localhost but an inventory can be empty since localhost is implied for each Ansible Automation Platform Inventory by default.
A project of type Git, pointing to the previously created repository. If the repository requires authentication, create a source control credential and associate it with the project.
Once configured, trigger a project synchronization and verify the logs to confirm successful collection retrieval.
For Ansible Automation Platform 2.4, specify the correct API URL prefix under Settings > Job settings > Edit > Extra Environment Variables:
{ "CONTROLLER_OPTIONAL_API_URLPATTERN_PREFIX": "/api/" }
Export objects
At this point, we should be able to perform configuration exports. Let’s start with simpler objects: organizations and users.
Add two playbooks in the Git repository created earlier: a wrapper to export organizations calling the corresponding playbook from the configify.aapconfig collection.
---
- name: Run playbook to export organizations
import_playbook: configify.aapconfig.aap_audit_organizations.ymlAs well as another playbook for user export:
---
- name: Run playbook to export users
import_playbook: configify.aapconfig.aap_audit_users.ymlSynchronize the project created earlier and create a job template using credentials, inventory, and project. For the playbook, select the one for organizations export and launch the job.
Once the job finishes, the last task contains a structure describing the existing organization configuration, which looks like this:
"controller_objects_organizations": [
"{'name': 'Default', 'descr': '', 'creds': ['Ansible Galaxy', 'Private Hub']}",
"{'name': 'Org B', 'descr': '', 'creds': ['Credential B']}",
"{'name': 'Org C', 'descr': 'Org C', 'creds': ['Credential C', 'Credential C2']}"
]Let’s remove the double quotes generated by the Ansible Automation Platform to get a list of dictionaries:
controller_objects_organizations: [
{'name': 'Default', 'descr': '', 'creds': ['Ansible Galaxy', 'Private Hub']},
{'name': 'Org B', 'descr': '', 'creds': ['Credential B']},
{'name': 'Org C', 'descr': 'Org C', 'creds': ['Credential C', 'Credential C2']}
]This structure is going to become one of the variables in CaC describing Ansible Automation Platform configuration.
Next run the playbook for user export to get the controller_objects_users variable:
controller_objects_users: [
{'username': 'UserB', 'first_name': 'bbb', 'last_name': '', 'email': '',
'superuser': False, 'auditor': False, 'pass': '$encrypted$'},
{'username': 'UserB_Superuser', 'first_name': 'bbbsss', 'last_name': '', 'email': '',
'superuser': True, 'auditor': False, 'pass': '$encrypted$'},
{'username': 'UserC', 'first_name': 'ccc', 'last_name': 'ccc', 'email': '',
'superuser': False, 'auditor': False, 'pass': '$encrypted$'},
{'username': 'UserC_Auditor', 'first_name': 'cccaaa', 'last_name': 'ccaa', 'email': '',
'superuser': False, 'auditor': True, 'pass': '$encrypted$'},
{'username': 'admin', 'first_name': '', 'last_name': '', 'email': 'admin@example.com',
'superuser': True, 'auditor': False, 'pass': '$encrypted$'},
{'username': 'automation', 'first_name': 'automation', 'last_name': '', 'email': '',
'superuser': True, 'auditor': False, 'pass': '$encrypted$'}
]Verify configurations
Once the configurations have been retrieved, let’s verify they are valid and can be used for CaC going forward. To do that, we need to create a new file in the repository with configurations obtained at the previous step and add another wrapper playbook:
- name: Include vars
hosts: localhost
gather_facts: false
tasks:
- name: Include variables
ansible.builtin.include_vars: all_aap_objects
tags: always
- name: Run playbook to configure AAP
import_playbook: configify.aapconfig.aap_configure.ymlThis playbook adds variables with objects from the file to the play and runs the Ansible Automation Platform configuration playbook from the aapconfig collection.
Synchronize the project and run the job template with the new playbook in check mode. Since we are verifying organizations and user configurations, let’s run the job with the following tags:
controller_config_organizations_apply
controller_config_users_applyOf course, the expectation is that there should be no tasks reporting a change:
PLAY [Include vars] ************************************************************
TASK [Set controller hostname] *************************************************
ok: [localhost]
TASK [Include all vars] ********************************************************
ok: [localhost]
PLAY [Configure AAP] ***********************************************************
TASK [Determine AAP version] ***************************************************
ok: [localhost]
TASK [configify.aapconfig.controller_config : USERS - Create users (2.5)] ******
skipping: [localhost] => (item= | user: UserB)
skipping: [localhost] => (item= | user: UserB_Superuser)
skipping: [localhost] => (item= | user: UserC)
skipping: [localhost] => (item= | user: UserC_Auditor)
skipping: [localhost]
TASK [configify.aapconfig.controller_config : USERS - Make user an auditor if required (2.5)] ***
skipping: [localhost] => (item= | user: UserB)
skipping: [localhost] => (item= | user: UserB_Superuser)
skipping: [localhost] => (item= | user: UserC)
skipping: [localhost] => (item= | user: UserC_Auditor)
skipping: [localhost]
TASK [configify.aapconfig.controller_config : USERS - Create or modify users (pre 2.5)] ***
ok: [localhost] => (item= | user: UserB)
ok: [localhost] => (item= | user: UserB_Superuser)
ok: [localhost] => (item= | user: UserC)
ok: [localhost] => (item= | user: UserC_Auditor)
TASK [configify.aapconfig.controller_config : ORGANIZATIONS - Create or modify organizations (2.5)] ***
skipping: [localhost] => (item= | organization: Org B)
skipping: [localhost] => (item= | organization: Org C)
skipping: [localhost]
TASK [configify.aapconfig.controller_config : ORGANIZATIONS - Create (pre 2.5) or modify organizations] ***
ok: [localhost] => (item= | organization: Org B)
ok: [localhost] => (item= | organization: Org C)
PLAY RECAP *********************************************************************
localhost : ok=5 changed=0 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 This confirms that our configurations work and are correct.
Handling objects with secrets
Note that the user accounts exported from Ansible Automation Platform earlier don't contain any passwords. This is because secrets are encrypted in Ansible Automation Platform and can not be retrieved. There are several ways to address this challenge in CaC:
- We can choose not to handle passwords in configuration and instead rely on users or administrators to manually update the credentials they own. This approach may be enough at the beginning or as a temporary solution, but in the long run, it will stand in the way of achieving full automation from start to finish.
- Alternatively, we could use Ansible Vault. Passwords and other secrets can be encrypted and used directly in CaC. This solution may be suitable for smaller deployments, but it doesn’t seem to be scalable for larger environments. It also requires additional scripts should regular password rotation be needed.
- The third approach seems to address the downsides of the two previous ones. The secrets will be stored in an external secret management system and retrieved using a lookup plug-in during the playbook run. For example, for HashiCorp, such retrieval looks similar to the following:
'pass': lookup('community.hashi_vault.hashi_vault', 'secret=secret/hello:value auth_method=approle role_id=hashi_roleid secret_id=hashi_secretid')Certainly, this solution adds to the requirements. Specifically, we will need to add community.hashi_vault collection to requirements.yml and create additional Ansible Automation Platform credential type and credential to store role_id and secret_id values.
One of these approaches needs to be taken with all Ansible Automation Platform objects containing secrets, such as users, credentials, and some notification profiles.
By default, the configify.aapconfig collection doesn’t update secrets. When such change is required, run the configify.aapconfig.aap_configure.yml playbook specifying in the job template’s variables as follows:
replace_passwords: trueThis switch makes all tasks that handle objects with secrets not idempotent since the Ansible Automation Platform has no way of telling if secrets changed.
Handling objects with special characters
Let’s create another wrapper playbook to export credential types and run it:
- name: Run playbook to get credential types
import_playbook: configify.aapconfig.aap_audit_credential_types.ymlThe output will look similar to the following:
controller_objects_credential_types: [
{'name': 'Credential Type B', 'descr': '',
'inputs': {'fields': [{'id': 'usernameB', 'type': 'string', 'label': 'Username'},
{'id': 'passwordB', 'type': 'string', 'label': 'Password'}]},
'injectors': {'extra_vars': {'configifyadpass': '{{ passwordB }}',
'configifyaduser': '{{ usernameB }}'}}},
{'name': 'Credential Type C', 'descr': '',
'inputs': {'fields': [{'id': 'usernameC', 'type': 'string', 'label': 'Username'},
{'id': 'passwordC', 'type': 'string', 'label': 'Password'}]},
'injectors': {'extra_vars': {'configifyadpass': '{{ passwordC }}',
'configifyaduser': '{{ usernameC }}'}}},
{'name': 'Hub type', 'descr': '',
'inputs': {'fields': [{'id': 'hub_url', 'type': 'string', 'label': 'HubURL'},
{'id': 'hub_user', 'type': 'string', 'label': 'HubUser'},
{'id': 'hub_pass', 'type': 'string', 'label': 'HubPass', 'secret': True}]},
'injectors': {'env': {'AH_HOST': '{{ hub_url }}',
'AH_USERNAME': '{{ hub_user }}',
'AH_API_TOKEN': '{{ hub_pass }}'}}}
]Double curly brackets in the output will certainly be a problem when running configuration playbooks since {{ }} in the Ansible Automation Platform represents a variable, while values from the output should be handled as strings.
The solution is simple. We need to add a key !unsafe in front of each problematic value as follows:
{'name': 'Hub type', 'descr': '',
'inputs': {'fields': [{'id': 'hub_url', 'type': 'string', 'label': 'HubURL'},
{'id': 'hub_user', 'type': 'string', 'label': 'HubUser'},
{'id': 'hub_pass', 'type': 'string', 'label': 'HubPass', 'secret': True}]},
'injectors': {'env': {'AH_HOST': !unsafe '{{ hub_url }}',
'AH_USERNAME': !unsafe '{{ hub_user }}',
'AH_API_TOKEN': !unsafe '{{ hub_pass }}'}}}Another area of concern is quotes and backslashes in object fields such as names and usernames. Consider the following export:
controller_objects_credentials_organizational: [
{'name': 'Credential with \"quotes\"', 'org': 'Org B', 'descr': '', 'type': 'Machine',
'inputs': {'password': 'HIDDEN', 'username': 'domain\\\\user'},
'src_input_field_name': '', 'src_credential': '', 'src_metadata': ''}The Ansible Automation Platform added backslashes in front of the double quotes in the credential name and multiple backslashes in the Active Directory username. It will consider such a configuration as a new credential during the import, and it will be created with a username containing multiple backslashes. To avoid this, simply remove unwanted characters before applying the configuration:
controller_objects_credentials_organizational: [
{'name': 'Credential with "quotes"', 'org': 'Org B', 'descr': '', 'type': 'Machine',
'inputs': {'password': 'HIDDEN', 'username': 'domain\user'},
'src_input_field_name': '', 'src_credential': '', 'src_metadata': ''}Manage configuration drift
Finally, we need to make sure the tool we use can handle these two cases:
- If there is an object or setting in the Ansible Automation Platform that is missing in the CaC, we would like to know about that and be able to delete them if we choose so.
- If there is an object that is not needed anymore, we want to simply remove it from CaC and watch the removal from Ansible Automation Platform during the next run.
Let’s verify that both situations are handled properly using organizations configuration retrieved earlier as an example. To do that, we are going to manually create a "rogue organization" in Ansible Automation Platform via the web interface and remove "Org C" from the file with objects in the repository.
Now let’s run the configuration playbook specifying a controller_config_organizations tag. The job should report both organizations as rogue:
<…>
TASK [configify.aapconfig.controller_config : ORGANIZATIONS - Notify on rogue organizations] ***************************************************************
skipping: [localhost] => (item= | organization: Default)
skipping: [localhost] => (item= | organization: Org B)
changed: [localhost] => (item= | organization: Org C) => {
"msg": "Shouldn't be there"
}
changed: [localhost] => (item= | organization: Rogue Organization) => {
"msg": "Shouldn't be there"
}
<…>
TASK [configify.aapconfig.controller_config : ORGANIZATIONS - Delete rogue organizations (pre 2.5)] ************************************************************
skipping: [localhost] => (item= | organization: Default)
skipping: [localhost] => (item= | organization: Org B)
skipping: [localhost] => (item= | organization: Org C)
skipping: [localhost] => (item= | organization: Rogue Organization)
skipping: [localhost]
<…>By default it deletes nothing. To force deletion, run the job specifying an extra variable:
delete_objects: trueFinal thoughts
This concludes the first part of the series. Using the right tools for configuring Ansible Automation Platform with the configuration as code approach makes this task easy and rewarding.
Stay tuned for the next article where we'll discuss CaC automation from start to finish and aspects of access and Git processes you should consider afterwards.
