01 / The goal
The goal
The goal of this project is to make AI agent systems easier to understand and safer to use. I’m exploring practical ways to manage what agents can access, check the messages they send, and keep a record of their actions, then sharing small, repeatable labs so others can learn and build on them.
02 / The problem
The problem
An AI agent is software that can ask an AI model questions and use tools to take action. For example, it might ask a model to summarize a document, then use a file tool to find that document.
When agents connect directly to models and tools, it can be hard to keep track of what they can access or what they’ve done. If something goes wrong, it may be unclear which agent made a request.
03 / The approach
The approach
I put a gateway between the agents, the AI model, and the tools. Think of it as a checkpoint: each request passes through it, where it can check who’s asking, whether the request is allowed, and what should be recorded.
I used agentgateway, an open-source project. In this lab, it checks each agent’s identity, limits which tools it can use, looks for certain sensitive information, and records requests.
How the setup works
- Agent: the program making a request. In my lab, it’s a small Python script.
- Gateway: the checkpoint that checks requests.
- AI model: the service that generates an answer. I use a pretend model for the lab.
- Tools: services the agent can use, such as email, files, or databases. My lab uses a demo tool server.
The agent sends a request to the gateway. The gateway checks it before passing it to the model or a tool.
AI agent
The one asking
In my lab: a small Python script
In real life: a terminal agent such as Claude Code, Codex CLI or Gemini CLI
Gateway
The security guard
1. Who is asking?
2. Is it allowed?
3. Is it safe to send?
4. Write it in the log
In my lab and in real life: agentgateway
AI model
The brain that answers
In my lab: a fake model
In real life: Claude Opus, Claude Sonnet, GPT-5, Gemini 2.5 Pro
Tools (MCP)
What the agent can do
In my lab: a demo tool server
In real life: email, files, GitHub, databases
04 / Architecture
Architecture
How to read it
- Controller and proxy are separate. The controller turns Kubernetes resources into configuration; the proxy is what traffic passes through.
- One policy point. Agents never talk to the model or the tool server directly.
- Agents are identities. orchestrator and worker are tokens I sign myself; the gateway trusts only the public key.
- The provider key stays with the gateway. Agents present their own identity and never hold the model provider's key.
05 / Six areas I looked at
Six areas I looked at
I used the six functions of the NIST Cybersecurity Framework (CSF) 2.0 to organize the project:
- Govern: Who sets the rules?
- Identify: What systems and risks are involved?
- Protect: How can harmful actions be prevented?
- Detect: How will we know when something happens?
- Respond: What should we do if something goes wrong?
- Recover: How do we return to normal?
GOVERN
Who owns this and what are the rules?
GOVERN
Who owns this and what are the rules?
What I did
- I keep every rule as a file in Git.
- I pinned the versions of the gateway, the Gateway API and the MCP server.
- Every resource has an owner label.
- Only namespaces I labelled can publish routes on the gateway.
- The token signing key stays out of Git.
Base practices
- Assign ownership: every agent, tool server, model and policy has a named owner and a risk tier.
- Write the rules of use: which data may go to which model, which actions need a human approval.
- Change policy only through review: pull request, second reviewer, CI validation, pipeline deploy.
- Control the supply chain: an approved catalogue of models and MCP servers, pinned and reviewed before upgrade.
- Gate onboarding: a team publishes through the gateway only after registering its agent and tools.
- Keep evidence and report: Git history answers "who allowed this and when".
IDENTIFY
What do I have and what can go wrong?
IDENTIFY
What do I have and what can go wrong?
What I did
- I listed the backends, routes and policies straight from the cluster and wrote a one-page threat model.
- Then I ran my attack script against the unprotected setup to get a baseline: no identity needed, a secret reaches the model, every agent sees every tool, and one tool returns the server's environment with a password I planted.
Base practices
- Keep a live inventory generated from gateway configuration and logs, not a spreadsheet.
- Find shadow usage: teams calling model APIs or MCP servers directly.
- Classify tools (read-only, writing, destructive) and data by sensitivity; flag agents that combine untrusted input, private data and a way to send data out.
- Threat-model every agent path before go-live with one template.
- Measure the baseline and repeat after each significant change.
PROTECT
What stops a bad action?
PROTECT
What stops a bad action?
What I did
- I added JWT authentication on the gateway, so every call carries an agent identity.
- I wrote one tool rule in CEL: anyone can use echo, only the orchestrator can use get-sum, nobody can use get-env.
- I added prompt guards that reject a prompt containing an AWS key or a card number and mask emails and keys in answers.
- The gateway injects the provider key from a Secret, so the agents never hold it.
- Each agent gets 5 LLM requests per minute.
matchExpressions:
- 'mcp.tool.name == "echo" || (jwt.sub == "orchestrator" && mcp.tool.name == "get-sum")'Base practices
- Give every agent its own short-lived identity from the corporate identity provider; no shared keys.
- Least privilege on tools: deny by default, allow per agent, human approval for destructive actions.
- Hold provider keys centrally and rotate them.
- Limit rate and spend per identity.
- Inspect content in both directions; start rules in audit mode, then enforce.
- Close the bypass: the gateway must be the only route to models and tools.
- Train the people who build and review agents.
DETECT
How do I know something happened?
DETECT
How do I know something happened?
What I did
- I read one access log for both model calls and tool calls.
- Each line has the agent identity, the route, the status and a reason; tool calls also have the tool name.
- I also checked the Prometheus metrics for MCP requests, guardrail checks and token usage.
2026-10-04T15:02:58.323097Z error request gateway=agentgateway-system/agentgateway-proxy listener=http route=agentic-lab/llm src.addr=127.0.0.1:53068 http.method=POST http.host=localhost http.path=/v1/chat/completions http.version=HTTP/1.1 http.status=429 jwt.sub=worker protocol=http error="rate limit exceeded" reason=RateLimit duration=0msBase practices
- Centralise gateway logs in the security monitoring platform.
- Define a short list of alerts from a baseline: bursts of denied calls, any guardrail rejection, first use of a tool, repeated rate-limit hits.
- Watch cost and volume per agent.
- Trace end to end with OpenTelemetry.
- Keep an audit trail for decisions an agent contributes to.
- Test the detections on a schedule.
RESPOND
What do I do about it?
RESPOND
What do I do about it?
What I did
- My scenario: the worker agent's token is stolen.
- I apply a deny policy I prepared in advance.
- It blocks that one identity on every route while the orchestrator keeps working.
- Then I rotate the signing key so every old token stops working, and I commit the change under an incident reference.
Base practices
- Write runbooks for agent incidents: stolen credential, malicious tool, prompt injection, runaway agent.
- Prepare and test containment levers in advance: block one agent, hide one tool, close one route.
- Revoke and rotate centrally.
- Scope the incident from the gateway logs.
- Record and communicate under an incident reference.
- Exercise it and measure time to contain.
RECOVER
How do I get back to normal?
RECOVER
How do I get back to normal?
What I did
- I delete the cluster and rebuild it from the repository only, then I run the proof script again.
- The rebuild time is my recovery time for this setup.
- The one thing that is not in Git is the signing key, so I re-issue it.
Base practices
- Make everything rebuildable from Git through a pipeline.
- Roll back by reverting a commit, then run automated control tests before reopening traffic.
- Set recovery objectives and prove them with a timed rebuild.
- Protect what is not in Git: keys and secrets.
- Be able to undo what agents changed.
- Feed lessons back into the rules, the threat model and the policies.
06 / What I tested
What I tested
I ran a script before and after turning on the gateway’s protections. The tests showed that the gateway could, for example:
- Block requests from callers without an identity.
- Stop a request containing a sample secret from reaching the model.
- Limit which tools each agent could use.
- Block access to a tool containing sensitive information.
- Slow down an agent making too many requests.
In the results below, “Blocked” is the expected outcome for requests that should not be allowed. The lab guide includes the full output from each check.
| # | Check | Before | After |
|---|---|---|---|
| 1 | Caller with no identity reaches the LLM | Allowed (200) | Blocked (401) |
| 2 | Expired token | Allowed (200) | Blocked (401) |
| 3 | Orchestrator with a valid token reaches the LLM | Allowed | Allowed |
| 4 | Prompt containing an AWS key | Reaches the model | Blocked (403) |
| 5 | Model answer containing an email and a key | Returned as is | Masked |
| 6 | Tools each agent can see | Both see all 13 | Orchestrator 2, worker 1 |
| 7 | Worker calls a tool reserved for the orchestrator | Allowed | Blocked |
| 8 | Any agent dumps the tool server's environment | Password exposed | Blocked |
| 9 | Orchestrator calls its own tool | Allowed | Allowed |
| 10 | Caller with no identity opens an MCP session | Allowed (200) | Blocked (401) |
| 11 | Worker loops eight LLM calls | All served | Throttled after 5 (429) |
From my run
These are exemple from running the lab on my laptop. The full output for each step is in the lab guide.
A FAIL means the open behaviour the baseline script expects is gone.
[FAIL] PROTECT 1. Caller with no identity reaches the LLM -> allowed (no policy yet)
HTTP 401
[FAIL] PROTECT 4. Prompt containing an AWS key -> allowed (no policy yet)
HTTP 403 Blocked by gateway: the prompt contains a secret or card number.
[FAIL] PROTECT 5. Model answer containing an email and a key -> returned as is
HTTP 200 content":"Sure, the admin contact is <EMAIL_ADDRESS> and the key is <masked>.","role":"assistant"},"index":0,"
[FAIL] IDENTIFY 6. Tool inventory seen by each agent
orchestrator sees 2: ['echo', 'get-sum'] | worker sees 1: ['echo']
[FAIL] PROTECT 10. Caller with no identity opens an MCP session -> allowed (no policy yet)
HTTP 401
[FAIL] PROTECT 11. Worker loops 8 LLM calls in a row -> all served
200 200 200 200 200 429 429 4292026-10-04T15:02:58.323097Z error request gateway=agentgateway-system/agentgateway-proxy listener=http route=agentic-lab/llm src.addr=127.0.0.1:53068 http.method=POST http.host=localhost http.path=/v1/chat/completions http.version=HTTP/1.1 http.status=429 jwt.sub=worker protocol=http error="rate limit exceeded" reason=RateLimit duration=0ms
2026-10-04T15:02:58.326208Z error request gateway=agentgateway-system/agentgateway-proxy listener=http route=agentic-lab/llm src.addr=127.0.0.1:53074 http.method=POST http.host=localhost http.path=/v1/chat/completions http.version=HTTP/1.1 http.status=429 jwt.sub=worker protocol=http error="rate limit exceeded" reason=RateLimit duration=0ms
2026-10-04T15:02:58.331921Z error request gateway=agentgateway-system/agentgateway-proxy listener=http route=agentic-lab/llm src.addr=127.0.0.1:53076 http.method=POST http.host=localhost http.path=/v1/chat/completions http.version=HTTP/1.1 http.status=429 jwt.sub=worker protocol=http error="rate limit exceeded" reason=RateLimit duration=0ms07 / What I plan to improve next
What I plan to improve next
This is a small demonstration, and there are several things I’d improve before using an approach like this in a real system:
- Use Kubernetes network policies to stop agents from reaching the model or tools except through the gateway.
- Use a proper identity service instead of a script to identify agents.
- Add guardrails to check messages passed between agents for unsafe or sensitive content.
- Improve the guardrail rules to catch more patterns of prompt injection and data leakage.
- Encrypt traffic between parts of the setup.
- Send logs somewhere they can be collected and reviewed.
feedback
Questions or corrections? Open an issue on the repo.