ansible share image

In today's fast-paced world, server downtime can have severe consequences for businesses. Ensuring high availability and rapid recovery is essential for maintaining uninterrupted services. In this article, we will explore how to create a self-healing server using the event-driven architecture of Red Hat Ansible Automation Platform and integrate it with Alertmanager for efficient monitoring and alerting.


  • Install Ansible Automation Platform and Ansible Rulebook.
  • Podman and podman-compose
  • These ports must be opened on the server side: 5000, 9090, 9093, 22.

The concepts of event driven and self healing

The event-driven architecture of Ansible Automation Platform enables servers to respond to events and take predefined actions automatically. It utilizes event-driven automation and monitoring to detect and remediate issues in real time, leading to a self-healing infrastructure.

To learn more about the concept of Event-Driven Ansible, please read my previous article. You can pull the code from our GitHub repository.

1. Install Prometheus and Alertmanager

Launching the containers with podman-compose will enable us to install prometheus and Alertmanager, using the following podman-compose.yaml file.

version: '3'

    image: prom/prometheus:v2.30.3
      - 9090:9090
      - prometheus-data:/prometheus
    command: --web.enable-lifecycle  --config.file=/etc/prometheus/prometheus.yml

    image: prom/alertmanager:v0.23.0
    restart: unless-stopped
      - "9093:9093"
      - "./alertmanager:/config"
      - alertmanager-data:/data
    command: --config.file=/config/alertmanager.yml --log.level=debug



The Alertmanager for Event-Driven Ansible should be configured in the receiver section, as shown in the alertmanager.yml file. Include the IP address of the server from which the rulebook is triggered in the webhook section. For instance, if you are running rulebooks on your local machine, the time IP should be, or if you have a remote server, then add the public IP of that server http:// 123.345.9.56:5000/alerts.


  group_by: [ alertname ]
  receiver: 'EDA' # default receiver
  repeat_interval: 24h

  - name: 'EDA'
      - url: ''

To launch the file, use the following command:

podman-compose up -d

Check that Alertmanager and Prometheus are running:

podman ps
CONTAINER ID   IMAGE                       COMMAND                  CREATED          STATUS          PORTS                                       NAMES
254000d2a108   prom/alertmanager:v0.23.0   "/bin/alertmanager -..."   15 seconds ago   Up 13 seconds>9093/tcp, :::9093->9093/tcp   self-healing-server_alertmanager_1
277f1c6da0cd   prom/prometheus:v2.30.3     "/bin/prometheus --w..."   15 seconds ago   Up 14 seconds>9090/tcp, :::9090->9090/tcp   self-healing-server_prometheus_1

Search for the site and check that prometheus is up and running. After accessing Prometheus, check out the Alertmanager dashboard:

2. Write the rulebook

The basic principle of any rulebook is source - rule - action. So the following rulebook also contains the same conditions. For a self-healing use case, we will use rules with conditions to trigger rulebooks for specific conditions.

- name: Automatic Remediation of a webserver
  hosts: localhost
    - name: listen for alerts
        port: 5000
    - name: server down
      condition: event.alert.labels.job == "server" and event.alert.status == "firing"
          name: remediation-playbooks/server-playbook.yml

    - name: Storage full on server
      condition: event.alert.labels.job == "storage" and event.alert.status == "firing"
          name: remediation-playbooks/storage-playbook.yml

    - name: memory full on server
      condition: event.alert.labels.job == "memory" and event.alert.status == "firing"
          name: remediation-playbooks/memory-playbook.yml

    - name: ssh server down
      condition: event.alert.labels.job == "ssh" and event.alert.status == "firing"
          name: remediation-playbooks/ssh-playbook.yml

    - name: CPU full on server
      condition: event.alert.labels.job == "cpu" and event.alert.status == "firing"
          name: remediation-playbooks/cpu-playbook.yml

For a self-healing server, we must list all conditions or scenarios where we can anticipate the server facing issues like full storage, memory fully utilized, and so on.

Accordingly, we have to find the remediation of that issue and create an Ansible Playbook that resolves that issue after triggering that issue without manual intervention.

Create the inventory file with localhost as host:


3. Run Ansible Rulebook

Use the ansible-rulebook command to run the rulebook:

ansible-rulebook --rulebook ansible-rulebook.yaml -i inventory -v
05:13:46,294 - - INFO - Starting sources
05:13:46,294 - - INFO - Starting rules
05:13:46,294 - ansible_rulebook.engine - INFO - run_ruleset
05:13:47 496 [main] INFO org.drools.ansible.rulebook.integration.api.rulesengine.AbstractRulesEvaluator - Start automatic pseudo clock with a tick every 100 milliseconds
05:13:48,402 - ansible_rulebook.engine - INFO - load source filters
05:13:48,403 - ansible_rulebook.engine - INFO - loading eda.builtin.insert_meta_info
05:13:48,887 - ansible_rulebook.engine - INFO - Calling main in ansible.eda.alertmanager
05:13:48,890 - ansible_rulebook.engine - INFO - Waiting for all ruleset tasks to end
05:13:48,890 - ansible_rulebook.rule_set_runner - INFO - Waiting for actions on events from Automatic Remediation of a webserver
05:13:48,890 - ansible_rulebook.rule_set_runner - INFO - Waiting for events, ruleset: Automatic Remediation of a webserver
05:13:48 891 [drools-async-evaluator-thread] INFO - Async channel connected

It will wait for the Alertmanager alert to fire and status changes to firing. Then only the Ansible rule will trigger. Conditions also play a major role. We are using the and functionality of two functions. The first one is for firing status, and the second one is for job label match.

We can assign different labels to different applications and we can trigger the rulebook based on status.

4. Trigger the alerts of Alertmanager

The status of alert changed from stable to firing condition as shown in Figure 1. In labels as a job, we can change the application name and trigger the specific remediation playbook.

A screenshot of the Alertmanager rules dashboard showing triggered rules.
Figure 1: The Alertmanager rules dashboard showing triggered rules.

Go back to the Ansible Rulebook CLI terminal where the rulebook was run. The logs show that the rule was triggered and the remediation playbook was run.

2023-06-20 05:26:46,907 - aiohttp.access - INFO - [20/Jun/2023:05:26:46 +0000] "POST /alerts HTTP/1.1" 202 164 "-" "Alertmanager/0.23.0"
2023-06-20 05:26:46,935 - ansible_rulebook.rule_generator - INFO - calling restart web server
2023-06-20 05:26:46,938 - ansible_rulebook.rule_set_runner - INFO - call_action run_playbook
2023-06-20 05:26:46,938 - ansible_rulebook.rule_set_runner - INFO - substitute_variables
2023-06-20 05:26:46,938 - ansible_rulebook.rule_set_runner - INFO - action args: {'name': 'say-what.yml'}
2023-06-20 05:26:46,938 - ansible_rulebook.builtin - INFO - running Ansible playbook: say-what.yml
2023-06-20 05:26:46,942 - ansible_rulebook.builtin - INFO - ruleset: Automatic Remediation of a webserver, rule: restart web server
2023-06-20 05:26:46,942 - ansible_rulebook.builtin - INFO - Calling Ansible runner
2023-06-20 05:26:46,943 - aiohttp.access - INFO - [20/Jun/2023:05:26:46 +0000] "POST /alerts HTTP/1.1" 202 164 "-" "Alertmanager/0.23.0"

5. Containerize Event-Driven Ansible functionality

You can also create containers on Event-Driven Ansible using the following Containerfile:

RUN microdnf install java-17 python3 gcc python3-devel -y && microdnf clean all && python -m ensurepip --upgrade && pip3 install ansible ansible-rulebook asyncio aiokafka aiohttp aiosignal
ENV JAVA_HOME="/usr/lib/jvm/jre-17"
RUN mkdir /eda-ansible
RUN ansible-galaxy collection install ansible.eda
WORKDIR /eda-ansible
COPY. /eda-ansible
CMD ansible-rulebook -i inventory --rulebook ansible-rulebook.yaml  --verbose

Integrate Event-Driven Ansible in the podman-compose file. Compose files will create a container and run it.

version: '3'
    - 5000:5000
    - prometheus
    - alertmanager

To run Prometheus, Alertmanager, and the event-driven rulebook, use the following:

podman-compose up -d

You will get the same results.

Continue your automation journey with Ansible Automation Platform

Get started with Ansible Automation Platform by exploring interactive hands-on labs. Download Ansible Automation Platform at no cost and begin your automation journey.