start here
Before you start
Words I use
- Agent
- a program that uses an AI model to decide what to do next, then acts by calling tools or other agents.
- LLM
- a large language model. The agent sends it text (a prompt) and gets text back.
- Tool server (MCP)
- a service that offers actions an agent can call, such as reading a file or adding two numbers. MCP, the Model Context Protocol, is the standard way an agent discovers and calls those tools.
- Kubernetes cluster
- a system that runs programs packaged as containers and keeps them running. I describe what I want in YAML files, called manifests, and Kubernetes makes it so.
- Namespace
- a folder inside the cluster that keeps one team's workloads apart from another's.
- Label
- a key and value tag on a resource, such as
owner: agentic-lab. Labels let me find and group things. - Gateway
- the single entry point all agent traffic goes through.
- Proxy
- the program behind the gateway that every request passes through. It checks each call, applies the rules and writes a log line.
- Controller
- the program that watches the cluster and configures the proxy. No traffic passes through it.
- Custom resource
- a new resource type added to Kubernetes. agentgateway adds its own types for backends and policies.
A note on versions
I use agentgateway release v1.6.0. The release number and the API version are two different things. agentgateway adds its own resource types to Kubernetes, and those types are at API version v1alpha1. That is why every policy file in this lab starts with apiVersion: agentgateway.dev/v1alpha1.
Kubernetes API versions move through three levels: alpha, beta and stable. v1alpha1 means the first alpha version: the types work, but their fields and names can still change between releases without a guarantee of backward compatibility. For this lab it means one thing: pin the version, and expect to adjust the manifests when you upgrade.
agentgateway is an open-source project hosted by the Agentic AI Foundation, which is part of the Linux Foundation.
Why: I use kind because it runs a Kubernetes cluster inside Docker. The whole lab stays on my laptop and I can delete it with one command.
In plain words
- Docker
- Docker runs containers, which are small packaged programs with everything they need inside.
- kind
- kind stands for Kubernetes in Docker. It starts a whole Kubernetes cluster as containers on my laptop, so nothing touches a cloud account.
- kubectl
- The command line I use to talk to the cluster: create things, list them, read their logs.
- Helm
- A package manager for Kubernetes. One command installs an application and all its parts at a version I choose.
brew install kind kubectl helmCheck: kind version, kubectl version --client and helm version --short each print a version, and Docker is running.
Why: I decide the rules before I deploy anything: every rule is a file in Git, versions are pinned, every resource has an owner, and the signing key stays out of the repository.
In plain words
- What Govern means
- Before anything runs, I decide who owns what and what the rules are. It is the cheapest moment to do it.
- Namespaces and labels
- In this lab there is one namespace,
agentic-lab, and I play every role. In an organisation, each application team gets its own namespace, labelled with the team's name. The team deploys its agents, tool servers and routes there, from its own repository. - The platform team
- The cluster, the gateway controller and the Gateway itself belong to the platform team. They carry the platform team's owner label and are deployed from the platform team's own repository, separate from the application teams' repositories. Application teams use the gateway but cannot change it.
- One label opens the door
- The Gateway only accepts routes from namespaces labelled
agentgateway-access: "true". The platform team sets that label when it onboards a team, so nobody can publish through the gateway without being registered first. - Owner labels
- Every resource carries an
ownerlabel. With one command I can answer "whose is this?" for anything in the cluster. - Pinned versions
- I write the exact version of every component. The lab I rebuild next month is then the same one I tested today.
- The signing key
- The signing key is the secret I use to create agent identities. Anyone who holds it can pretend to be any agent, so
.gitignorekeeps it out of the repository.
git clone https://github.com/fsclyde/agentic-gateway-lab.git && cd agentic-gateway-lab
cat .gitignore
grep -rn "owner:\|csf-function:\|@2026\|v1.6" manifests README.md | head -20# Python environment for the token and proof scripts
python3 -m venv venv && source venv/bin/activate
pip install PyJWT cryptographyRun source venv/bin/activate again in every new terminal you use for the Python scripts.
Check: manifests/01-gateway.yaml only accepts routes from namespaces labelled agentgateway-access: "true".
Why: The Gateway API adds the standard Gateway and HTTPRoute resource types. The first Helm chart adds agentgateway's own types (backends and policies) and the second installs the controller. When I apply a Gateway resource, the controller creates the proxy for me.
In plain words
- Teaching Kubernetes new types
- Kubernetes comes with built-in resource types. It can also learn new ones through custom resource definitions (CRDs). A CRD gives the cluster a new type with its own fields and validation.
- The Gateway API
- The first
kubectl applyinstalls the Gateway API, a standard set of types (Gateway, HTTPRoute) from the Kubernetes project for describing how traffic enters a cluster. - agentgateway's own types
- The first Helm chart installs agentgateway's custom types in the API group
agentgateway.dev, versionv1alpha1.AgentgatewayBackendsays where traffic goes: a model or a tool server.AgentgatewayPolicyholds the rules: who may call, what is inspected, how many requests are allowed. These custom types are what let me write every restriction in this lab as a plain YAML file. - The controller
- The second Helm chart installs the controller. No request passes through it. It watches the cluster for gateways, routes, backends and policies and turns them into configuration for the proxy.
- Why a proxy gets created
- When I apply the Gateway resource I am telling the cluster "I want an entry point". The controller sees it and creates the proxy for me, as a Deployment and a Service called
agentgateway-proxy. I never install the proxy by hand. - Why I need a proxy
- The proxy is the program every request passes through. An agent sends its call to the proxy, not to the model or the tool server. Because it sits in the middle, the proxy can check who is calling, apply the rules, keep the model provider's key away from the agent and write a log line. Without a proxy in the path there is no single place to enforce or record anything.
kind create cluster --name agentic
kubectl apply --server-side \
-f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.2/standard-install.yaml
helm upgrade -i agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds \
--create-namespace --namespace agentgateway-system --version v1.6.0
helm upgrade -i agentgateway oci://cr.agentgateway.dev/charts/agentgateway \
--namespace agentgateway-system --version v1.6.0 --wait --timeout 5m
kubectl apply -f manifests/00-namespace.yaml -f manifests/01-gateway.yaml
kubectl wait --for=condition=Programmed gateway/agentgateway-proxy -n agentgateway-system --timeout 5m
kubectl get pods,svc -n agentgateway-systemCheck: pods agentgateway (controller) and agentgateway-proxy are Running.
Why: I want to know what I have and what can go wrong before I choose controls. The "before" result is my baseline. Without it the "after" result means nothing.
In plain words
- What Identify means
- I want to know what is deployed in the cluster before I protect it. I cannot secure something I do not know is running.
- What I deploy
- A mock LLM, which is a small stand-in model so the lab needs no API key, and an MCP tool server. The server offers harmless tools such as
echoandget-sum, and one dangerous tool,get-env, which returns the server's environment variables. I planted a fake password there on purpose. - Backends and routes
- A backend tells the gateway where a model or a tool server lives. A route connects a path on the gateway (
/v1for the model,/mcpfor the tools) to a backend. - Port-forward
- It opens a tunnel from port 8080 on my laptop to the proxy inside the cluster, so my scripts can reach the gateway.
- Taking inventory
- The
kubectl getcommand lists every gateway, route, backend and policy in every namespace. Because all agent traffic is declared as resources, the cluster itself is my inventory. - Discovery tools
- On a real cluster I would not rely on one command.
kubectl api-resourceslists every resource type the cluster knows. k9s and Headlamp let me browse everything that is running. Scanners such as Kubescape and Trivy report what is deployed together with its misconfigurations and vulnerabilities. Gateway logs complete the picture: they show what is really called, not only what is declared. - The threat model
- A one-page list of what could go wrong on each path and which control answers it.
- The baseline
tokens.py initcreates the signing key pair.check.py --stage beforethen runs eleven attacks against the unprotected gateway and records the result. This is my "before" picture.
kubectl apply -f manifests/10-mock-llm.yaml -f manifests/11-mcp-server.yaml -f manifests/12-backends-routes.yaml
kubectl rollout status deploy/mock-llm deploy/mcp-everything -n agentic-lab --timeout 5m
# second terminal, leave running
kubectl port-forward -n agentgateway-system service/agentgateway-proxy 8080:80kubectl get gateway,httproute,agentgatewaybackend,agentgatewaypolicy -A
cat docs/threat-model.md
source venv/bin/activate # if not already active
python3 scripts/tokens.py init
python3 scripts/check.py --stage beforeCheck: 11/11 pass in the "before" sense: no identity needed, the AWS key reaches the model, every agent sees all tools, and get-env returns the planted DEMO_DB_PASSWORD.
What I got
Gateway: http://localhost:8080 Expecting: open state
[PASS] PROTECT 1. Caller with no identity reaches the LLM -> allowed (no policy yet)
HTTP 200
[PASS] PROTECT 2. Expired token -> allowed (no policy yet)
HTTP 200
[PASS] PROTECT 3. Orchestrator with a valid token reaches the LLM -> allowed
HTTP 200
[PASS] PROTECT 4. Prompt containing an AWS key -> allowed (no policy yet)
HTTP 200 {"model":"mock-model","usage":{"prompt_tokens":12,"completion_tokens":9,"total_t
[PASS] PROTECT 5. Model answer containing an email and a key -> returned as is
HTTP 200 content":"Sure, the admin contact is admin@example.com and the key is AKIAIOSFODNN7EXAMPLE.","role":"assistant
[PASS] IDENTIFY 6. Tool inventory seen by each agent
orchestrator sees 13: ['echo', 'get-annotated-message', 'get-env', 'get-resource-links']... | worker sees 13: ['echo', 'get-annotated-message', 'get-env', 'get-resource-l
[PASS] PROTECT 7. Worker calls a tool reserved for the orchestrator (get-sum) -> allowed (no policy yet)
{"jsonrpc": "2.0", "id": 3, "result": {"content": [{"type": "text", "text": "The sum of 2 and 3 is 5."}]}}
[PASS] PROTECT 8. Any agent dumps the tool server's environment (get-env) -> allowed (no policy yet)
environment returned, including DEMO_DB_PASSWORD
[PASS] PROTECT 9. Orchestrator calls get-sum -> allowed
{"jsonrpc": "2.0", "id": 4, "result": {"content": [{"type": "text", "text": "The sum of 2 and 3 is 5."}]}}
[PASS] PROTECT 10. Caller with no identity opens an MCP session -> allowed (no policy yet)
HTTP 200
[PASS] PROTECT 11. Worker loops 8 LLM calls in a row -> all served
200 200 200 200 200 200 200 200Why: I apply one control at a time and re-run the proof script after each one, so I see exactly which control stops which attack.
In plain words
- 4a, authentication
- Authentication answers "who is calling?". Each agent presents a JWT, a signed token that carries its name. The policy tells the gateway to accept only tokens signed by my key. The gateway holds the public key only, so it can verify tokens but cannot create them.
- 4b, authorization
- Authorization answers "is this caller allowed to do this?". The rule is written in CEL, a small language for one-line conditions. Anyone can use
echo, only the orchestrator can useget-sum, andget-envmatches no rule, so it disappears from the tool list and is refused if called. - 4c, prompt guards
- The gateway reads the text going to the model and the text coming back. A prompt containing an AWS access key or a card number is rejected. Emails and keys in the model's answer are masked before the agent sees them.
- 4d, rate limit
- Each agent identity gets five model requests per minute. An agent stuck in a loop cannot use up the budget for everyone else.
- 4e, the provider key
- The agent presents its own token. The gateway replaces it with the model provider's key, which it reads from a Kubernetes Secret. The agent never holds the provider key, so a compromised agent cannot leak it.
# 4a. Identity: every request needs a signed token (flips checks 1, 2, 10)
python3 scripts/tokens.py policy > manifests/20-jwt-policy.yaml
kubectl apply -f manifests/20-jwt-policy.yaml
# 4b. Authorization: which agent may use which tool (flips checks 6, 7, 8)
kubectl apply -f manifests/21-mcp-tool-rules.yaml
# 4c. Data protection: inspect prompts and answers (flips checks 4, 5)
kubectl apply -f manifests/22-llm-guardrails.yaml
# 4d. Limits: a budget per agent identity (flips check 11)
kubectl apply -f manifests/23-llm-rate-limit.yaml
python3 scripts/check.py# 4e. See the provider key swap
ORCH=$(python3 scripts/tokens.py mint orchestrator)
curl -s localhost:8080/v1/chat/completions -H "authorization: Bearer $ORCH" \
-H 'content-type: application/json' \
-d '{"model":"mock-model","messages":[{"role":"user","content":"hello"}]}'
kubectl logs -n agentic-lab deploy/mock-llm --tail=3Check: 11/11 pass in the secured sense. The mock LLM's log shows the gateway's provider key, not your agent token. If you run the script twice within a minute a check can fail with 429; wait 60 seconds.
What I got
To watch the controls take effect, I re-ran the baseline script (--stage before) after applying them. A FAIL here is the result I want: the open behaviour it expects is gone.
[FAIL] PROTECT 1. Caller with no identity reaches the LLM -> allowed (no policy yet)
HTTP 401
[FAIL] PROTECT 4. Prompt containing an AWS key -> allowed (no policy yet)
HTTP 403 Blocked by gateway: the prompt contains a secret or card number.
[FAIL] PROTECT 5. Model answer containing an email and a key -> returned as is
HTTP 200 content":"Sure, the admin contact is <EMAIL_ADDRESS> and the key is <masked>.","role":"assistant"},"index":0,"
[FAIL] IDENTIFY 6. Tool inventory seen by each agent
orchestrator sees 2: ['echo', 'get-sum'] | worker sees 1: ['echo']
[FAIL] PROTECT 10. Caller with no identity opens an MCP session -> allowed (no policy yet)
HTTP 401
[FAIL] PROTECT 11. Worker loops 8 LLM calls in a row -> all served
200 200 200 200 200 429 429 429Why: A control that blocks silently is only half the job. Because every call goes through the proxy, I get one log stream for model calls and tool calls.
In plain words
- One log line per call
- Every request through the proxy produces one line with the agent's name (
jwt.sub, taken from its token), the route, the status and the reason. - Reading the status codes
- 401 means the caller has no valid identity. 403 means the caller is known but not allowed. 429 means the caller sent too many requests.
- Metrics
- Metrics are counters the proxy exposes in the Prometheus format, a common standard for monitoring. They show totals over time, such as tool calls, guardrail checks and tokens used.
kubectl logs -n agentgateway-system deploy/agentgateway-proxy --tail=200 | grep -E '401|403|429'
kubectl logs -n agentgateway-system deploy/agentgateway-proxy --tail=200 | grep 'tools/call'
# third terminal
kubectl port-forward -n agentgateway-system deploy/agentgateway-proxy 15020:15020
curl -s localhost:15020/metrics | grep -E 'agentgateway_(mcp_requests|guardrail_checks|gen_ai_client_token_usage_count)'Check: each refused call shows the agent (jwt.sub), the route, the status and a reason.
What I got
2026-10-04T15:02:58.323097Z error request gateway=agentgateway-system/agentgateway-proxy listener=http route=agentic-lab/llm src.addr=127.0.0.1:53068 http.method=POST http.host=localhost http.path=/v1/chat/completions http.version=HTTP/1.1 http.status=429 jwt.sub=worker protocol=http error="rate limit exceeded" reason=RateLimit duration=0ms
2026-10-04T15:02:58.326208Z error request gateway=agentgateway-system/agentgateway-proxy listener=http route=agentic-lab/llm src.addr=127.0.0.1:53074 http.method=POST http.host=localhost http.path=/v1/chat/completions http.version=HTTP/1.1 http.status=429 jwt.sub=worker protocol=http error="rate limit exceeded" reason=RateLimit duration=0ms
2026-10-04T15:02:58.331921Z error request gateway=agentgateway-system/agentgateway-proxy listener=http route=agentic-lab/llm src.addr=127.0.0.1:53076 http.method=POST http.host=localhost http.path=/v1/chat/completions http.version=HTTP/1.1 http.status=429 jwt.sub=worker protocol=http error="rate limit exceeded" reason=RateLimit duration=0msWhy: I want to stop one agent fast without stopping the others, then remove the attacker's access for good.
In plain words
- The scenario
- Someone has stolen the worker agent's token and is using it.
- Contain
- I apply a deny policy that targets one identity,
jwt.sub == "worker", on both routes. The worker is refused everywhere and the orchestrator keeps working. - Eradicate
- Blocking a name is not enough, because the stolen token is still valid. I rotate the signing key: I create a new key pair and give the gateway the new public key. Every token signed with the old key stops working.
- Close
- I remove the temporary block and commit the change under an incident reference, so the Git history records what I did and why.
# contain: block that one identity on every route
kubectl apply -f manifests/30-respond-block-worker.yaml
# eradicate: rotate the signing key so every old token stops working
python3 scripts/tokens.py init
python3 scripts/tokens.py policy > manifests/20-jwt-policy.yaml
kubectl apply -f manifests/20-jwt-policy.yaml
# close the incident
kubectl delete -f manifests/30-respond-block-worker.yaml
git add -A && git commit -m "LAB-001: rotate signing key after worker compromise"Check: while the block is applied the worker gets 403 and the orchestrator still gets 200; after rotation old tokens get 401.
What I got
$ curl -s -o /dev/null -w "%{http_code}\n" localhost:8080/v1/chat/completions -H "authorization: Bearer $WORK" ...
200
$ kubectl apply -f manifests/30-respond-block-worker.yaml
agentgatewaypolicy.agentgateway.dev/incident-block-worker created
$ curl -s -o /dev/null -w "%{http_code}\n" localhost:8080/v1/chat/completions -H "authorization: Bearer $WORK" ...
403Why: Recover means I can get back to a known-good state. I destroy the cluster, rebuild it from the repository only and prove the controls are back.
In plain words
- Starting from nothing
kind delete clusterremoves everything. Because every rule is a file in Git, I can rebuild the cluster and the policies from the repository alone.- Proving it
- I run the proof script again. If all eleven checks pass, the controls are back, not just the cluster.
- Recovery time
- The time the rebuild takes is my recovery time for this setup.
kind delete cluster --name agentic
# repeat Step 2, then:
kubectl apply -f manifests/10-mock-llm.yaml -f manifests/11-mcp-server.yaml -f manifests/12-backends-routes.yaml
kubectl apply -f manifests/20-jwt-policy.yaml -f manifests/21-mcp-tool-rules.yaml \
-f manifests/22-llm-guardrails.yaml -f manifests/23-llm-rate-limit.yaml
# restart the port-forward, then:
python3 scripts/check.pyCheck: 11/11 again. The rebuild time is your recovery time.
reading
Further reading
- agentgateway documentationagentgateway.dev/docs/kubernetes/latest/
- agentgateway source, tag v1.6.0github.com/agentgateway/agentgateway/tree/v1.6.0
- Kubernetes Gateway APIgateway-api.sigs.k8s.io/
- Model Context Protocolmodelcontextprotocol.io/
- NIST Cybersecurity Framework 2.0www.nist.gov/cyberframework
- OWASP Top 10 for LLM applicationsgenai.owasp.org/llm-top-10/
- Kubernetes API versioning (alpha, beta, stable)kubernetes.io/docs/reference/using-api/#api-versioning
- Agentic AI Foundation projectsaaif.io/projects