# Hide credentials from agents

> Keep API keys out of the agent process entirely. The proxy injects them after the request leaves the sandbox.

The cheapest threat-model improvement you can make to an agent system
is to remove the credential from the agent process. Prompt injection,
model jailbreak, tool-call abuse - every one of them ends at a
process that doesn't hold the key. This guide shows how CLRK delivers
that with four standard objects.

## The threat model in one paragraph

An agent that holds an API key is one prompt away from leaking it.
The defense is to never put the key in the agent's process
environment. The egress proxy holds it, terminates the agent's
outbound TLS connection, swaps a placeholder header for the real one,
and re-emits the request to the upstream over a fresh connection.
The agent sees only `clrk-injected-by-proxy`.

## The four objects

```mermaid
flowchart TB
  S[Kubernetes Secret] -. "secretRef" .-> CIP[CredentialInjectionPolicy]
  CIP -. "parentRefs" .-> APR[AIProviderRoute]
  APR -. "parentRefs" .-> EG[EgressGateway]
  Agent[sandbox] -- "HTTPS with placeholder" --> EG
  EG -- "TLS terminate + swap header" --> Up[api.anthropic.com]
```

- **`Secret`** - standard Kubernetes Secret. Holds the real key.
- **`EgressGateway`** - the TLS-terminating MITM hop every outbound
  connection from the sandbox traverses. Must declare a `Terminate`
  listener (the example sets `tls.mode: Terminate`) so the proxy can
  read and rewrite request headers.
- **`AIProviderRoute`** - declares the upstream and the provider
  parser. Accepted provider values today (a registry-validated
  string, not a kubebuilder enum): `openai`, `anthropic`, `google`,
  `azure-openai`, `bedrock`, `custom`.
- **`CredentialInjectionPolicy`** - attaches the Secret to a route
  and declares where the credential goes (`Header`, `QueryParam`, or
  provider-specific `ProviderAuth` like AWSv4 / GCPServiceAccount).

## Worked example: Anthropic

Pulled from `_examples/echo-bot/manifests/`. The four objects:

```yaml
# 1. The Secret. clrk dev --secret materializes it from your shell env;
#    in prod the operator creates it normally.
apiVersion: v1
kind: Secret
metadata:
  name: anthropic-credentials
data:
  api-key: <base64-of-sk-ant-...>
---
# 2. The EgressGateway with a TLS-Terminate listener. allow-all here
#    because the example only egresses to Anthropic; production
#    deployments typically pair this with deny-all + an L4 allowlist.
apiVersion: clrk.apoxy.dev/v1alpha1
kind: EgressGateway
metadata:
  name: echo-bot
spec:
  defaultPolicy: allow-all
  listeners:
    - name: egress
      protocol: TLS
      tls:
        mode: Terminate
  otlp:
    captureBody:
      maxBytes: 65536
---
# 3. The AIProviderRoute. Declares Anthropic and the endpoints to
#    intercept.
apiVersion: clrk.apoxy.dev/v1alpha1
kind: AIProviderRoute
metadata:
  name: anthropic
spec:
  parentRefs:
    - group: clrk.apoxy.dev
      kind: EgressGateway
      name: echo-bot
  rules:
    - matches:
        - provider: anthropic
          endpoints:
            - /v1/messages
            - /v1/complete
---
# 4. The CredentialInjectionPolicy. Binds the Secret to the route and
#    declares the header to overwrite.
apiVersion: clrk.apoxy.dev/v1alpha1
kind: CredentialInjectionPolicy
metadata:
  name: anthropic
spec:
  parentRefs:
    - group: clrk.apoxy.dev
      kind: AIProviderRoute
      name: anthropic
  secretRef:
    name: anthropic-credentials
  secretKey: api-key
  target: Header
  headerName: x-api-key
```

The agent's HTTPS request to `api.anthropic.com/v1/messages` arrives
at the EgressGateway with whatever `x-api-key` value the agent
supplied (typically the placeholder `clrk-injected-by-proxy`). The
proxy rewrites the header from the Secret and forwards.

## Where in the request lifecycle the swap happens

```mermaid
sequenceDiagram
  participant A as agent (sandbox)
  participant EG as EgressGateway
  participant U as api.anthropic.com
  A->>EG: TLS to api.anthropic.com with placeholder x-api-key
  Note over EG: TLS terminate against per-EG CA leaf
  Note over EG: ext_proc swaps x-api-key from Secret
  EG->>U: fresh TLS, real x-api-key
  U-->>EG: response
  EG-->>A: response (OTLP record emitted)
```

The sandbox trusts the per-EgressGateway CA because the worker
installed its leaf cert as a trust anchor at sandbox boot. The
agent's HTTPS library accepts the EG's MITM cert as if Anthropic
itself had presented it. From the agent's perspective the connection
looks normal.

## Confirm via OTLP, not via /proc

The authority is the captured request, not the agent's process
environment. Two things to verify in the `otel-traces` pane:

1. **The call returned 200.** Anthropic 401s on a wrong key. A 200
   with non-zero `gen_ai.usage.*_tokens` proves the proxy supplied a
   valid credential.
2. **The captured request shows the credential header redacted.**
   Expand the request span's `http.request.headers` event and confirm
   `http.request.header.x-api-key` reads `[redacted]` - CLRK replaces
   known credential headers (`authorization`, `x-api-key`,
   `anthropic-api-key`, etc.) with `[redacted]` before exporting
   telemetry, so the real value never ships either. Header values are
   span event attributes attached unconditionally, not part of the
   body. The `otlp.captureBody.maxBytes: 65536` setting only controls
   the request/response body bytes (captured base64 in the
   `http.request.body` event), not headers.

## Smuggle defense

If the agent script sets its own `x-api-key` header in an attempt to
override the policy, the policy still wins - the proxy rewrites the
header unconditionally based on the route + policy attachment.
There's no agent-controllable opt-out. The swap happens on the egress
ext_proc path (the `egress-extproc` component), so the captured
request in the `otel-traces` / `otel-logs` panes reflects the
injected credential, not the agent-supplied one.

## Key rotation

Rotate the Secret in place. The proxy reads the latest value via the
controller's informer cache, so changes propagate without restarting
the agent or the gateway:

```bash
clrk secret set anthropic-credentials --from-literal=api-key="$NEW_KEY"
```

The next request through the gateway picks up the new value.

## BYOK and per-customer keys

The simplest pattern is one `AIProviderRoute` + one
`CredentialInjectionPolicy` per customer, each bound to its own
`Secret`. Route customer requests through the right TaskAgent (so the
right EgressGateway, so the right policy). This works today with
one-policy-per-route attachment.

Per-request credential selection - a single agent picking which
customer's key to use based on a header on the inbound request - is
not supported as a first-class feature today. Talk to us if you need
it.

## What this does NOT do

- **Does not authenticate the agent's caller.** Inbound auth is your
  problem; see [Authenticate users before
  agents](./authenticate-users-before-agents).
- **Does not encrypt traffic between the agent and the proxy
  end-to-end.** The proxy decrypts in order to rewrite. If your
  compliance regime requires no plaintext at any hop, this approach
  is not for you.
- **Does not protect against an agent leaking the response.** Once
  Anthropic returns content to the agent, the agent owns it. Pair
  with output policy and audit if that matters.
- **Does not work with secrets sourced from `spec.template.spec.env[].valueFrom.secretKeyRef`.**
  That field is silently dropped today. Credential injection is the
  intended path; do not try to surface secrets to the agent process.

## Where to next

- Pair credential injection with a default-deny egress allowlist so
  a stolen-key request can only reach allowed upstreams - see [Lock
  down agent egress](./lock-down-agent-egress).
- Cap how many tokens a single agent (or a misbehaving prompt) can
  burn against the injected key - see [Cap LLM spend per
  agent](./cap-llm-spend-per-agent).
- Trace which request triggered which credentialed call - see [Trace
  requests through agents](./trace-requests-through-agents).