Synchronize instance tags from Amazon EC2 and Microsoft Azure with Red Hat Insights

Tagging is a common feature offered today as part of system and platform management tools. Its goal is to provide a consistent way to organize, search and filter managed entities, thus improving reporting and overall operations management efficiency. A common challenge when dealing with different tools and platforms is the lack of tagging governance. Tags assignment and values can quickly run out of control unless best practices and rules are instored and honored in an organization. Often, organizations choose to define their taxonomy in a central configuration management database (CMDB) and use a discovery process to flag inconsistent tagging standards or associations in various platforms or management tools in use.

In this article, we investigate how Red Hat Insights can synchronize its system tags with external sources. Red Hat Insights is a Software-as-a-Service offering that enables users to obtain actionable intelligence regarding their Red Hat Enterprise Linux environments, helping to identify and address operational and vulnerability risks before an issue results in downtime. 

We provide examples for public cloud providers such as Amazon Elastic Compute Cloud (EC2) and Microsoft Azure. Both cloud providers offer system tagging or labeling to store metadata information. This is used to help categorize, search, and filter their inventory based on criteria relevant to the business (e.g., environment, team, cost center, workload, etc.) Our goal is to leverage existing tags set in these infrastructure providers to minimize friction to get going with our SaaS management offering. We automatically ingest tags from the cloud providers into Red Hat Insights and use them for managing and reporting in a consistent manner.

What are System tags in Red Hat Insights?

Red Hat Insights stores metadata tags as part of the System Inventory and offers them as a way to search and filter systems across applications. System tags consist of a key value pair storing information about the system and its context. They are set or imported from various sources. These include the Insights Client with a dedicated file for tags, Red Hat Satellite during its inventory synchronization, and the automated workload identification process performed by Insights to detect applications running on a system (e.g. SAP, Microsoft SQL, Red Hat Ansible Automation Platform). 

Tags can be used in many different ways in Insights. You may, for example, want to get a list of findings from Advisor application affecting your production systems (assuming you have tagged your systems with their environment.) You may also want to use tags as part of Policies conditions, or get a list of all systems owned by one or more specific users. All these use cases (and more) can be fulfilled with system tags in Red Hat Insights.

Setting system tags using Insights Client is achieved by editing the /etc/insights-client/tags.yaml file on the system itself. This file gets collected by Insights Client and its content processed every time the client runs (daily by default.) Values of the tags set in the file are used to replace existing values in Inventory. As such, system tags for a particular system get updated in Inventory at each insights client run.

An example of the tags.yaml file is presented below.

---
group: My Group
Location: 
- Geneva
- Switzerland
Application: RHEL Desktop
Environment: Production
Expiration Date: 
Team: PMs
Owner: Jerome Marc
Slack: jmarc
Contact: xxx

The tags.yaml configuration file consists of a YAML file containing tag value pairs. Tags can get assigned multiple values (as described in the example file above for Location values). 

Once processed, the file results in the assignment of multiple system tags. They are displayed in all applications with a system list by a tag icon next to the system name itself e.g., System Inventory, Advisor, Vulnerability, Compliance, and etc. Clicking on the tag icon opens up a window with all recorded tags for the system (Figure 1).

Values of system tags are used to filter the tables within Red Hat Insights. All possible tab names and values are presented when selecting a filter by tags (Figure 2). One or more tag value pairs can be selected in your filter. In the case of multiple selection, the OR operator is evaluated between tag value pairs.

Tag value pairs can also be used as part of the Global Filtering capabilities to set which tags should be used for filtering consistently across all areas of Red Hat Insights. This global filter is available at the top of the page (Figure 3). Once set, all applications inherit the filters and only present the results according to the list of filtered systems.

Additional information on settings and using system tags in Red Hat Insights can be found in the product documentation.

Amazon EC2 tags

Tag metadata is a common tool to manage Amazon Web Services (AWS) resources. AWS Tags documentation describes how to create and manage them on the cloud platform. Additionally, AWS published a Best Practices for Tagging AWS Resources whitepaper providing guidance on tagging strategy, use cases, and best naming conventions to adopt.

We are specifically interested in retrieving the tags from the instance running insights client. We plan to populate the tags.yaml file with AWS tags and get them uploaded via insights client.

Note that the instance must be enabled to retrieve metadata. The procedure is documented in Allow access to tags in instance metadata and must be followed for the following to work.

The get the instance tags for an instance documentation describes the commands to run to retrieve metadata tags from Linux instances. This can be achieved in two steps. First we grab the name of all tags set on the instance. Then we use the names to query for each individual value of the tag. It is important to note that AWS EC2 tags are limited to one single value, unlike insights-client tags.

Retrieving Amazon EC2 tags from a script

Our plan is to create a Shell script that runs on the instance to query for its metadata and retrieve all associated tags. The script populates the tags.yaml file with AWS tags. We plan to run the script every time prior to running insights client. 

First we create a /usr/bin/get-aws-tags.sh shell script file with executable permissions.

$ sudo touch /usr/bin/get-aws-tags.sh
$ sudo chmod +x /usr/bin/get-aws-tags.sh
$ ls -lal /usr/bin/get-aws-tags.sh
-rwxr-xr-x. 1 root root 592 Mar 21 15:38 /usr/bin/get-aws-tags.sh

The content of /usr/bin/get-aws-tags.sh should look like the following.

#!/bin/sh
TAG_FILE=/etc/insights-client/tags.yaml
if [ ! -e $TAG FILE ]; then echo '---' > $TAG_FILE; fi
TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && TAGS=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/tags/instance`
sed -i '/^aws\/.*:/d' $TAG_FILE
for TAG in $TAGS
do
  VALUE=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/tags/instance/$TAG`
  echo aws/$TAG: $VALUE >> $TAG_FILE
done

The script queries for each AWS tag and gets its associated name. It then appends the tag to the end of the /etc/insights-client/tags.yaml file. Note that the AWS tags are cleaned up from the file initially to ensure invalid tags are removed and no duplicate is present.

AWS tags get prefixed with aws/ to ensure they differ from regular insights-client tags. If no insights-client tag is present and/or if the /etc/insights-client/tags.yaml file does not exist, we create it and ensure it begins with --- (three dashes).

You can confirm that the script works successfully by running it manually and validating that the AWS tags are appended at the end of the /etc/insights-client/tags.yaml file. The file should now contain a new line for each of the AWS tags associated with the instance. For example:

$ cat /etc/insights-client/tags.yaml 
---
aws/Name: ral-db02.hcc-lab.com
aws/cost-center: xxxx
aws/owner: jspinks
aws/service-phase: prod
aws/sleep_schedule: true

Running our script prior to insights client

When installed on Red Hat Enterprise Linux (RHEL) 7 and later, insights-client schedules a systemd timer /usr/lib/systemd/system/insights-client.timer that runs daily by default on the system.

We can use an override.conf file for the systemd timer to add an ExecStartPre condition to the service. The condition runs our /usr/bin/get-aws-tags.sh script. We make sure that the timer doesn’t fail if something goes wrong with our script by prefixing our script path with a ‘-’ (dash) character. The following commands are used to create the file and display its content.

$ sudo mkdir /etc/systemd/system/insights-client.service.d
$ sudo touch /etc/systemd/system/insights-client.service.d/override.conf
$ sudo cat /etc/systemd/system/insights-client.service.d/override.conf
[Service]
ExecStartPre=-/usr/bin/get-aws-tags.sh

Now that the override.conf file is in place, we can reload the daemon and restart the timer with the following commands.

$ sudo systemctl daemon-reload
$ sudo systemctl restart insights-client.service

We can also validate the status and outcome of the script by querying systemctl for insights-client.

$ systemctl status insights-client
● insights-client.service - Insights Client
     Loaded: loaded (/usr/lib/systemd/system/insights-client.service; static)
    Drop-In: /etc/systemd/system/insights-client.service.d
             └─override.conf
     Active: active (running) since Thu 2024-03-21 15:45:50 UTC; 6s ago
TriggeredBy: ● insights-client.timer
       Docs: man:insights-client(8)
    Process: 17181 ExecStartPre=/usr/bin/get-aws-tags.sh (code=exited, status=0/SUCCESS)
   Main PID: 17182 (insights-client)
      Tasks: 2 (limit: 300)
     Memory: 43.5M (high: 1.0G max: 2.0G available: 980.4M)
        CPU: 2.018s
     CGroup: /system.slice/insights-client.service
             ├─17182 /usr/libexec/platform-python /usr/bin/insights-client --retry 3
             └─17194 /usr/libexec/platform-python /usr/lib/python3.9/site-packages/insights_client/run.py --retry 3

You should see in the output a process executed as ExecStartPre with a 0/SUCCESS exit status.

Validating the end-to-end process

Our last validation consists of confirming AWS tags are present in our system inventory in Red Hat Insights. The list of tags contains all tags retrieved from different sources (e.g. insights-client or Satellite). We can use the filter with our aws/ prefix to get a list of all tags coming from AWS via insights-client (Figure 4). 

Now that our system tags are synchronized between AWS and Red Hat Insights, we can use the same tags in both platforms to filter our views and services, perform actions on specific systems and/or run automation according to system tags.

In the remaining sections of this article, we provide the script required to perform the same operations with Microsoft Azure and retrieve system tags.

Microsoft Azure tags

Microsoft Azure encourages similar practices than AWS for managing and using system tags. They document conditions and limitations of using system tags in Use tags to organize your resources and management hierarchy. In particular, Azure offers a similar access to tag metadata from each instance. The procedure is documented in Azure Instance Metadata Service for virtual machines.

Retrieving Microsoft Azure tags from a script

We follow the exact same approach as AWS to populate our tags.yaml file with tags coming from Azure instances. The content of /usr/bin/get-azure-tags.sh script should look like the following.

#!/bin/sh
TAG_FILE=/etc/insights-client/tags.yaml
if [ ! -e $TAG FILE ]; then echo '---' > $TAG_FILE; fi
TAGS=`curl -s -H Metadata:true --noproxy "*" "http://169.254.169.254/metadata/instance/compute/tags?api-version=2021-02-01&format=text"`
sed -i '/^azure\/.*:/d' $TAG_FILE
IFS=';'
for tag in $TAGS
do
  echo "azure/$tag" | sed 's/:/: /' >> $TAG_FILE
done

The script queries for Azure tags at once and retrieve a semicolon separated string of name value pairs. It then parses the string and appends each tag to the end of the /etc/insights-client/tags.yaml file. Note that the Azure tags are cleaned up from the file initially to ensure invalid tags are removed and no duplicate is present.

Similar to AWS, Azure tags get prefixed with azure/ to ensure they differ from regular insights-client tags. If no insights-client tag is present and/or if the /etc/insights-client/tags.yaml file does not exist, we create it and ensure it begins with --- (three dashes).

You can confirm that the script works successfully by running it manually and validating that the Azure tags are appended at the end of the /etc/insights-client/tags.yaml file. The file should now contain a new line for each of the Azure tags associated with the instance. For example:

$ cat /etc/insights-client/tags.yaml 
---
azure/cost-center: xxxx
azure/environment: production
azure/owner: jmarc

The same procedure as AWS can be followed to run the script prior to insights-client as part of its systemd timer. Once done, Azure tags will automatically be updated and synchronized with Red Hat Insights every time the client runs. We can use the same tags in both platforms to filter our views and services, perform actions on specific systems and/or run automation according to system tags.

Conclusion 

This article provides a way to synchronize system tags from instances running on Amazon EC2 or Microsoft Azure cloud services with Red Hat Insights. The shell scripts offered for each cloud provider are available on a GitHub repository for ease of use. With this mechanism in place, one can use the same taxonomy and naming convention used as part of their cloud management platforms within Red Hat Insights. Further, system tags are useful to obtain additional context and pass on further information to the platform. This can in turn be used for automation and/or driving operations processes.

We welcome feedback to help improve and grow our product. Please share your experience with us by using the Feedback form located on the right side of the Hybrid Cloud Console.