Using Falcon to cleanup Satellite host records that belong to terminated OSP instances

Overview

In an environment where OpenStack instances are automatically subscribed to Satellite, it is important that Satellite is notified of terminated instances so that is can safely delete its host record. Not doing so will:

• Exhaust the available subscriptions, leading to unsubscribed hosts not being able to apply updates and security errata.
• In the event that an emergency security errata needs to be deployed across the organization, Satellite administrators would be unable to determine if a host was either off or terminated, leading to uncertainty with their security posture.

In smaller environments, where one team is responsible for both OSP and Satellite, it's possible to have one system administrator do this by using their administrator level access across both systems to determine which host records can be safely deleted in Satellite when the corresponding instance no longer exists.

This approach, however, does not scale as the number of instances is launched and terminated daily increases across the environment. Larger environments also lead to different teams being responsible for different software suites, and administrator level credentials would seldom be granted to a single person.

One approach to solving this problem in an automated manner is to have Satellite periodically poll OpenStack to determine if a given instance's UUID still exists and should it not, remove the host record.

Some assumptions before we begin:

• Instances launched are automatically subscribed to Satellite via rhsm.
• UUID of the instance is passed to Satellite during the registration process, found by default under the hosts virt::uuid fact on Satellite.
• An instance/VM/physical box that can connect to the Keystone/nova endpoints and that can be polled by Satellite.

Designing the API

Falcon is a remarkably simple Python web API framework that can be quickly deployed with minimal effort. The API was designed to return status codes depending on the status of the instance's UUID being checked using the call http://hostname/check/, where the following return codes are used:

200 = instance exists, don't delete host record

400 = bad UUID request - UUID not formatted correctly

404 = instance does not exist, delete host record

500 = unable to reach OSP

With the API designed, it's now a simple coding exercise to have the API:

1. Check that the UUID provided is valid.
2. Contact OSP and search for the provided UUID.
3. Return a status code based on the search result.

Using Keystone v3

The keystoneauth session Python API is used to create a token that is then passed to Nova via the novaclient Python API. The following functions will be used later to query OSP:

from novaclient import client as nova_client
from keystoneclient.v3 import client as keystone_client
from keystoneauth1.identity import v3
from keystoneauth1 import session
import sys

def get_osp_token():
  try:
    auth = v3.Password(user_domain_name=default, username=admin, password=XXXX, auth_url=https://osp.endpoint:35357, project_domain_name=default, project_name=admin)
    sess = session.Session(auth=auth, verify="./cacert.pem")
    return sess

  except session.exceptions.http.Unauthorized:
    print ("Credentials incorrect")

  except session.excpetions.connection.ConnectFailure:
    print ("Unable to reach OSP Server")

  except:
    print ("Unexpected error:, sys.exc_info()[0])

Using the token generated by the above get_osp_token() function, the following generate_id_list() function will generate a list of all the instance UUID's that exist in OSP:

def generate_id_list(token):
  nova = nova_client.Client('2', session=token)
  instance_list = nova.servers.list(detailed=True, search_opts= {'all_tenants': 1,})
  instance_id_list = [instance.id.lower() for instance in instance_list]
  return instance_id_list

The Falcon API

Beginning with Falcon's simple Learning by Example, we utilize the above functions to create our API:

#instanceapi.py
import falcon

class CheckUUID(object):

  def on_get(self, req, resp, uuid):
    if not uuid_valid(uuid):
      resp.status = falcon.HTTP_400
      resp.body = (uuid+' is not a valid UUID that can be parsed\n')
      return

    osptoken = get_osp_token()
    id_list = generate_id_list(osptoken)

    if not id_list:
      resp.status = falcon.HTTP_500
      resp.body = ('Server Down\n')
      return

    uuid = uuid.lower()

    if uuid in id_list:
      resp.status=falcon.HTTP_200
      resp.body =('The UUID '+uuid+' exists in OSP\n')
      return

    # no match found
    resp.status = falcon.HTTP_404
    resp.body = ('The UUID '+uuid+' does not exist in OSP\n')

# main block
app = falcon.API()
check = UUIDCheck()
app.add_route('/check/{uuid}', check)

Using Gunicorn to serve the API

With the working code, it's now a simple matter of deploying the code to a host that can be polled by Satellite.

Using Gunicorn, a simple WSGI server is started which serves the Python code.

# gunicorn --workers=4 --bind=0.0.0.0:80 check:app

A simple systemd service file will allow the API to start in the event the host is restarted:

#/usr/lib/systemd/system/instanceapi.service

[Unit]
Description=Instance API Frontend
After=network.target

[Service]
Environment="PATH=/root"
WorkingDirectory=/root
ExecStart=/usr/bin/gunicorn instanceapi:app -b 0.0.0.0:80 -w 4 --access-logfile /var/log/access.log

[Install]
WantedBy=multi-user.target

# systemctl enable instanceapi.service; systemctl start instanceapi.service

Checking Satellite Hosts

Satellite can now check iterate over the host records and determine if it can be safely removed. To get the UUID of a specific host from Satellite, you can use the hammer CLI:

# hammer fact list --search "host=<hostname> and fact=virt::uuid"

The UUID returned by hammer can then be passed to API, where the return code will indicate whether the instance still exists.

Conclusion

Without requiring the sharing of administrator credentials between Satellite and OpenStack deployments, it's now possible with the use of a simple API to determine if an instance on OSP still exists.

Whether you are new to Linux or have experience, downloading this cheat sheet can assist you when encountering tasks you haven’t done lately.