Even if you have never shipped out an agent yourself, you would hope that the companies behind the agents you're using have tested the security of the model and harness. Traditionally, as described in the Red Hat blog post Introducing asago: Open source AI safety and governance orchestration, onboarding a new agent looks like this:
- Policy development: Policy documents are created to encode the organization's AI dos and don'ts.
- Risk extraction: Someone reads the documents and identifies the theoretical risks they describe.
- Risk triaging: Someone decides which of the risks apply to the agent being onboarded and whether or not they can be tested.
- Scenario generation and red teaming: Technical teams build test scenarios for the risks and use red teaming frameworks to test the agent.
- Iteration: The results are interpreted and safety features are implemented as needed. The agent is retested until it is production ready.
Connecting policy development and risk extraction is in itself a difficult process. After an organization's legal team drafts the policy, security reviews it, and leadership signs off on it, it sits in a shared drive as a legal document rather than as technical requirements that engineers and security teams can implement easily. The agents that organizations have running in production may have been tested and checked against the policy once or twice, but probably are not being checked continuously.
Similar issues arise with the handoff process for each of these steps, as there is a noticeable lack of a consistent audit trail. These issues, while they can be fixed manually, can be time consuming and require dedicated effort from someone on the team. This process could potentially take months or more. Up until now, there has been no automated process capable of verifying that an agent is actually following the policy requirements. asago, an open source, community-governed framework that aims to automate the process from AI governance policy to production-ready AI systems, aims to fix this problem. Let's see how it can bridge some of the gaps in the process of securing AI agents.
Gap 1: Your policy and your agent live in separate worlds
For most organizations, their AI governance policy takes the form of a legal policy document. While it can use technical language, it often speaks in broad ideas. For example, here is a snippet from St. John University's policy, which you can find in asago's policy_examples folder:
"The University commits to the ethical use of AI in accordance with its professional conduct and non-discrimination policies. These technologies must not be used to create content that is inappropriate, discriminatory, deceptive, or otherwise harmful to others or to the University."
What is inappropriate? What is deceptive? And how do we make sure our agents don't create that kind of content? Ultimately, these documents are made for humans to read, understand, and interpret. How can we derive concrete risks from such legal policy documents, and then compare an agent against those risks?
asago's policy mapper addresses this issue. It reads the governance documents and extracts testable requirements by mapping the policy language to recognized risk frameworks. These include the IBM AI Risk Atlas for a broad taxonomy for identifying and categorizing AI risks, the OWASP LLM Top 10 for a security-focused view of risks specific to LLM applications, and the NIST AI Risk Management Framework for a broader framework for governing, measuring, and managing AI risk. Together, these mappings give organizations a structured way to move from policy language to identifiable risks.
For example, a policy clause like "must not be used to create content that is inappropriate, discriminatory, deceptive, or otherwise harmful to others or to the University" can be mapped to specific risks like unfair discrimination and misrepresentation, output bias, decision bias, and more. Each mapping includes the evidence of the risk from your policy, how confident the policy mapper is that the risk applies to the policy, and the references to the taxonomy the risk is drawn from. The result is a systematic connection from the policy to the risks to the tests and controls, so when auditors ask how AI requirements are being handled, you can show them the risks extracted from the document, the tests generated from them, the results, and the deployed controls. The structured scenario YAML files created by the policy mapper are then used later by downstream asago components like the scenario generator to create technology-agnostic test scenarios.
Gap 2: Policy to testing takes manual translation
Even if you know what risks apply to your agent, someone still needs to look at the risk descriptions and think about what kinds of harmful scenarios your agent could find itself in. This kind of work takes time and is prone to mistakes; different engineers might interpret the same risk differently, omit small edge cases, or miss huge scenarios entirely.
asago's scenario generator helps to automate this translation. After the policy mapper identifies the relevant risks, the scenario generator creates high-level harmful scenarios that the target agent could find itself in based on those risks. These scenarios also include the kinds of techniques that can be used to attack the agent.
For example, a harmful scenario for an agent could be "Exposure of customer Personally Identifiable Information (PII) via prompt injection." The risk here is the exposure of customer PII, and the possible technique is prompt injection. These test scenarios are systematically created by the scenario generator based on an agent's actual attack surface rather than what a human reviewer happened to think of during review. These test scenarios are later used by downstream asago components to create runnable tests for the agent.
Gap 3: Scenarios don't translate directly into executable tests
Test scenarios are just descriptions of the security risks that you're testing for. The next step is to actually run tests against your agent based on those scenarios. For comprehensive coverage, hundreds to thousands of tests need to be written. This becomes a huge bottleneck in the process and creates massive work for the engineer who needs to write them. A red teaming tool like garak also has specific input formats, evaluation structures, and ways to define what success or failure looks like. As a result, someone has to manually translate each scenario into the specific format that the testing platform expects for potentially thousands of tests.
asago's artifact generator automates this translation by taking the scenario YAML files generated earlier by the scenario generator, classifying the injection surfaces, and generating specific red teaming tests. Consider, for example, the "Exposure of customer Personally Identifiable Information (PII) via prompt injection" scenario we discussed in the previous section. This description needs to become a structured test artifact that can actually be run against your agents with adversarial probes, detector rubrics, and metadata that the testing platform can parse. The artifact generator creates a series of prompt injection tests that can be run with garak against your agent to test for PII leakage. It also creates these tests for all of the scenarios created to establish comprehensive coverage and compliance with the AI policies.
What this means for your agents
After running all 3 components, you can use an orchestration platform like EvalHub for the execution layer. EvalHub will trigger the garak scan to actively run the adversarial probes against the agent endpoint. With the raw safety metrics, the recommender reviews the results and produces guardrails or configuration recommendations. Then the user applies the recommendations and re-runs the evaluation against the same scenario until all of the risks pass. This whole process is illustrated in Figure 1. As of September 2026, the recommender is still in the development stage, but the other components, including the policy mapper, scenario generator, and artifact generator, are codebases that you can run today.

If you're building AI agents, you're probably facing some version of the gaps we've described here. While asago does not eliminate the need for security experts, it automates the mechanical work of extracting policy requirements, generating test scenarios, running tests, and providing mitigation recommendations.
Getting started
Check out the components below along with some introductory material about asago.
Components:
Blog post:
Press release: