# Metrics API

> Read request rates, error counts, latency percentiles, and byte volumes for your project through typed time-series and snapshot endpoints.

The metrics API serves traffic measurements for a project from the project API server, backed by aggregated telemetry. It has two shapes: a **snapshot** on an object (current totals for one Gateway, route, Proxy, Service, VPC network, or tunnel) and a **series** query (one or more lines over time). Both read the same measurements, so a tile and the chart under it agree.

The rule for picking one: one object and you want its current numbers, read `/metrics` on that object. Many objects, or any view over time, use `metrics/<name>/series`.

## Endpoint and authentication

All requests go to your project API server:

```
https://<project-id>.api.apoxy.dev
```

Authenticate with an API key in the `X-Apoxy-API-Key` header, the same as every other call to this API:

```
X-Apoxy-API-Key: <api-key>
```

`Authorization: Bearer <api-key>` is also accepted. To create a key, see [Authentication](/docs/reference/mcp.md#authentication) in the MCP server reference. Requests without a valid key receive `401 Unauthorized`.

## Resources

The group is `metrics.apoxy.dev/v1alpha1`. All resources are cluster-scoped, and no project identifier appears in any path or parameter because the API server serves exactly one project.

| Resource | Kind | Verbs |
|---|---|---|
| `metrics` | `Metric` | get, list, watch, create, update, delete |
| `metricsources` | `MetricSource` | get, list |
| `metrics/<name>/series` | `MetricSeriesSet` | get |

A `Metric` is a named recipe: a query fragment plus display preferences. The built-in recipes are managed for you; you can also save your own. A `MetricSource` describes what a recipe can read and group by.

## Built-in metrics

Every project starts with these recipes. They carry the `metrics.apoxy.dev/managed: "true"` label.

| Metric | Type | Measures | Unit |
|---|---|---|---|
| `http.requests` | counter | `total`, `status_2xx`, `status_4xx`, `status_5xx` | - |
| `http.errors` | counter | `errors_4xx`, `errors_5xx` | - |
| `http.latency` | histogram | `p50`, `p95`, `p99` | `ms` |
| `http.bytes` | counter | `bytes_in`, `bytes_out` | `By` |
| `log.severity` | counter | `total`, `errors`, `warnings` | - |
| `upstream.connections` | gauge | `active`, `peak` | - |
| `tls.handshakes` | counter | `handshakes` | - |
| `network.bytes` | counter | `bytes` | `By` |
| `network.packets` | counter | `packets` | - |
| `network.drops` | counter | `drops` | - |
| `network.keepalives` | counter | `keepalives` | - |
| `network.tunnels` | gauge | `active`, `peak` | - |
| `tunnel.rtt` | gauge | `rtt`, `rtt_peak` | `s` |

`http.requests` and `http.errors` are plain counts and carry no unit, so they contribute nothing to the `units` map in a response. `upstream.connections` reports live upstream connections, and `tls.handshakes` counts downstream TLS connections established in the window.

The `network.*` and `tunnel.*` recipes measure the tunnels of a VPC network rather than HTTP traffic. `network.bytes` and `network.packets` count every packet of payload a tunnel carries, so east-west traffic and non-HTTP protocols show up in them and not in `http.*`. `network.keepalives` counts the keep-alive frames that keep the tunnel path active; they carry no payload, so they are counted apart and a tunnel that carries no traffic reports keep-alives with zero bytes and zero packets. `network.tunnels` reports how many tunnels a network had connected, counted across every relay that serves it; its `peak` adds the largest count each relay saw, so it is an upper bound. `tunnel.rtt` reports the round trip time between an agent and the relay it connects to. Each is readable with `scopeKind=VPCNetwork` or `scopeKind=Tunnel`.

List the catalog with the resolved source and unit for each recipe:

```bash title="terminal"
kubectl get metrics
```

You should see `NAME`, `TYPE`, `UNIT`, and `SOURCE` columns. A `GET` on one name returns the full recipe, including the measures and groupable keys the server derived from it.

<Callout label="Two catalogs">
CLRK has its own fleet metrics catalog under `metrics.clrk.apoxy.dev`, with different recipe names and its own scope kinds. It is a separate product surface. See [Query fleet metrics](/docs/clrk/guides/query-fleet-metrics.md) for that one.
</Callout>

## Metric sources

A source is what a recipe reads. `kubectl get metricsources` lists them with their granularity and retention.

| Source | Granularity | Retention | Notes |
|---|---|---|---|
| `http_1m` | 1m | 7d | HTTP traffic aggregated into one-minute buckets. |
| `http_1h` | 1h | 400d | The same measurements in one-hour buckets. |
| `otel_logs` | row | 90d | Individual log records. |
| `envoy_1m` | 1m | 30d | Proxy connection and TLS stats in one-minute buckets. |
| `relay_1m` | 1m | 30d | Tunnel traffic in one-minute buckets, per network, per relay, and per tunnel. |

The `http.*` recipes read `http_1m`. Series and snapshot reads over longer windows are served at coarser granularity automatically, so history stays available past the finer source's retention without any change to your request.

A snapshot evaluates recipes from several sources, and the sources keep different amounts of history. A window wider than a source's retention drops that source's recipes from the snapshot and reports the rest: `vpcnetworks/{name}/metrics?window=1440h` answers with the `http.*` recipes, which the 400 day rollup serves, and without the `network.*` and `tunnel.*` recipes, which are kept for 30 days. A window no evaluated source can reach is a `400` instead - `tunnels/{name}/metrics` reads `relay_1m` alone, so a 60 day window there names the source and its maximum. Naming such a recipe explicitly with `metric=` is a `400` as well, rather than a quiet drop.

`log.severity` reads raw log records, which is why it is **series-only**: snapshots evaluate the aggregated recipes and skip raw-log ones. A grouped read over raw records is limited to a 24 hour window.

`MetricSource.status.fields` lists each field with a `role`:

- `time` - the bucket column.
- `key` - a groupable dimension. These are exactly the values `groupBy` accepts.
- `measure` - a value column a recipe aggregates.

Each field also reports `discovered`, which is `true` for a field found by sampling log attributes (so it can disappear) and `false` for a fixed column.

## Series

<APIEndpoint method="GET" path="/apis/metrics.apoxy.dev/v1alpha1/metrics/{name}/series" />

Runs one recipe over a window and returns a `MetricSeriesSet`.

### Parameters

| Parameter | Default | Description |
|---|---|---|
| `scopeKind` | `Project` | Owner kind to scope to: `Project`, `Gateway`, `HTTPRoute`, `Proxy`, `Service`, `VPCNetwork`, or `Tunnel`. Takes a kind, not a resource name (`Gateway`, not `gateways`). |
| `scopeName` | - | Name of the owner object. Omit it for `Project`, which reads the whole project. |
| `scopeListener` | - | Narrows a `Gateway` scope to one listener. |
| `since` | `-1h` | Start of the window. An RFC3339 instant or a relative duration. |
| `until` | end of the last complete bucket | End of the window. Same formats as `since`. |
| `window` | - | Shorthand for `since=-<duration>`. Rejected together with `since`. |
| `step` | source granularity | Bucket width. Rounded **up** to the source granularity and echoed in the response. |
| `groupBy` | - | One key field to split the result by. |
| `orderBy` | the recipe's first default column | Measure to rank groups by. |
| `top` | `50` | Maximum number of series to return. `50` is also the ceiling. |

The window is half-open, `[since, until)`. Relative durations accept `h`, `m`, `s`, and `d`: `-6h`, `-7d`, `-1.5d`, and `+30m` all parse. A bare `6h` with no sign does not, and neither does `-1w`.

Buckets count whole steps forward from `since` rather than from the clock, so the first point carries `since`, and a `step` equal to the window returns exactly one point per series.

### Group-by keys

`groupBy` takes exactly one key:

| Key | Description |
|---|---|
| `gateway` | Gateway that served the request. |
| `listener` | Gateway listener that served the request. |
| `route_kind` | Route object kind. |
| `route_name` | Route object name. |
| `route_rule` | Rule index inside the route. |
| `backend_kind` | Backend object kind. |
| `backend_name` | Backend object name. |
| `backend_revision` | Compute revision that served the request. Empty for traffic that is not served by a revision. |
| `status_class` | Response status class (`2`, `4`, `5`). |
| `method` | HTTP request method. |

The `network.*` and `tunnel.*` recipes read a different source and take their own keys:

| Key | Description |
|---|---|
| `network` | VPC network the traffic crossed. |
| `tunnel` | Tunnel that carried it. One tunnel is one agent connection. |
| `relay` | Relay that carried it. A network is served by more than one relay. Empty on a metric that is not counted per relay. |
| `direction` | `rx` or `tx`. Empty on a metric that counts no traffic. |

A key that the recipe's source does not carry returns `400` with the valid keys listed. The authoritative list for any recipe is its `status.keys`.

### Response

```json
{
  "kind": "MetricSeriesSet",
  "apiVersion": "metrics.apoxy.dev/v1alpha1",
  "metric": "http.requests",
  "scopeKind": "Gateway",
  "scopeName": "prod",
  "since": "2026-08-20T15:00:00Z",
  "until": "2026-08-20T21:00:00Z",
  "step": "5m0s",
  "dataUpTo": "2026-08-20T21:00:00Z",
  "truncated": false,
  "totalCount": 7,
  "units": {},
  "series": [
    {
      "labels": { "route_name": "api" },
      "points": [
        { "timestamp": "2026-08-20T15:00:00Z", "values": { "total": 5210, "status_4xx": 140, "status_5xx": 12 } },
        { "timestamp": "2026-08-20T15:05:00Z", "values": { "total": 5188, "status_4xx": 131, "status_5xx": 9 } }
      ]
    }
  ]
}
```

| Field | Description |
|---|---|
| `metric`, `scopeKind`, `scopeName` | Echo the resolved query. `scopeName` is empty for a project-wide read. |
| `since`, `until` | Resolved window bounds. |
| `step` | Applied bucket width, serialized as a duration string (`"5m0s"`). |
| `dataUpTo` | End of the last complete bucket. |
| `truncated` | `true` when more groups matched than `top` returned. |
| `totalCount` | How many groups had data in the window. |
| `units` | Measure name to display unit, echoed from the catalog. Empty when no measure carries a unit. |
| `series[].labels` | Group key and value. Empty for an ungrouped read, which returns exactly one series. |
| `series[].points[]` | Buckets in timestamp order, each with the recipe's measures. |

Points are **sparse**: a bucket with no traffic is omitted rather than returned as zero. Compare a missing bucket against `dataUpTo` to tell missing data from a partial trailing bucket.

### Fleet views

A one-bucket series is how you get one row per object. Set `step` equal to the window and group by the owner key:

```
metrics/http.requests/series?scopeKind=Project&groupBy=gateway&since=-1h&step=1h
```

That returns one point per Gateway, ranked by `orderBy` and bounded by `top`. The same shape with `scopeKind=Gateway&groupBy=route_name` gives one row per route.

## Snapshots

A snapshot is every applicable recipe evaluated over one window for one object, in a single call. It mounts as a `metrics` subresource on the owner.

| Owner | Path | Response kind | `include` tokens |
|---|---|---|---|
| Gateway | `/apis/gateway.apoxy.dev/v1/gateways/{name}/metrics` | `GatewayMetrics` | `routes` |
| HTTPRoute | `/apis/gateway.apoxy.dev/v1/httproutes/{name}/metrics` | `HTTPRouteMetrics` | `rules`, `backends` |
| Proxy | `/apis/core.apoxy.dev/v1alpha2/proxies/{name}/metrics` | `ProxyMetrics` | none |
| Service | `/apis/compute.apoxy.dev/v1alpha1/services/{name}/metrics` | `ServiceMetrics` | `revisions` |
| VPCNetwork | `/apis/vpc.apoxy.dev/v1alpha1/vpcnetworks/{name}/metrics` | `VPCNetworkMetrics` | `services`, `tunnels` |
| Tunnel | `/apis/vpc.apoxy.dev/v1alpha1/tunnels/{name}/metrics` | `TunnelMetrics` | none |

Every snapshot carries `timestamp`, `window`, `since`, `until`, `dataUpTo`, a `metrics` map keyed by recipe name, and a `units` map. The window parameters are the same as for series.

```json
{
  "kind": "GatewayMetrics",
  "apiVersion": "metrics.apoxy.dev/v1alpha1",
  "metadata": { "name": "prod" },
  "timestamp": "2026-08-20T21:00:00Z",
  "window": "24h0m0s",
  "since": "2026-08-19T21:00:00Z",
  "until": "2026-08-20T21:00:00Z",
  "dataUpTo": "2026-08-20T21:00:00Z",
  "metrics": {
    "http.requests": { "total": 1842113, "status_2xx": 1790220, "status_4xx": 48120, "status_5xx": 3773 },
    "http.latency": { "p50": 38, "p95": 190, "p99": 412 }
  },
  "units": { "p50": "ms", "p95": "ms", "p99": "ms" },
  "listeners": [
    {
      "name": "https",
      "metrics": { "http.requests": { "total": 1839900, "status_5xx": 3773 } },
      "truncated": true,
      "totalCount": 37
    }
  ]
}
```

### Nesting

The default response is the owner totals plus the first nesting level, with no leaf rows. Add them with `include`:

- `include=routes` on a Gateway adds `routes[]` under each listener.
- `include=rules`, `include=backends` on an HTTPRoute add `rules[]` and `backends[]`.
- `include=revisions` on a Service.
- `include=services`, `include=tunnels` on a VPC network.
- `include=all` means every token for that kind.

`include` may repeat or take a comma-separated list. An unknown token returns `400`.

`truncated` and `totalCount` sit on whichever container was cut. A Gateway cuts routes per listener, so they appear on each listener. `HTTPRouteMetrics`, `ServiceMetrics`, and `VPCNetworkMetrics` cut their own leaf lists, so they appear on the object itself and count every list the read asked for.

A VPC network reports its services and its tunnels from different measurements: `services[]` covers gateway traffic to the network's VPC services, and `tunnels[]` covers everything the tunnels carried. The two do not add up to each other. `network.tunnels` counts the network rather than one connection, so it is reported in the network's own `metrics` map and never as an entry of `tunnels[]`; every entry of `tunnels[]` names a connection, and `totalCount` counts those entries alone.

For complete per-route coverage instead of the top slice, use a one-bucket series grouped by `route_name`.

`orderBy` names the measure a nested list is ranked by. A snapshot can nest by levels that come from different sources, and a level is ranked by the named measure only when a recipe of that level reports it; a level whose recipes do not report it keeps its own default order, which is the request count where there is one. `include=all&orderBy=bytes` on a VPC network therefore ranks `tunnels[]` by bytes and leaves `services[]` in its own order. A measure no evaluated recipe reports anywhere is a `400`.

### Restricting recipes

`metric=<name>` limits which recipes are evaluated. The parameter repeats and does **not** take a comma-separated list:

```
?metric=http.requests&metric=http.latency
```

Snapshots evaluate the managed recipes by default. Naming a recipe of your own with `metric=` evaluates that one too.

`ProxyMetrics` additionally carries `replicas`, which reports `connected` replica counts from the Proxy's own status rather than from traffic. Proxy metrics cover traffic on the Gateways attached to that Proxy.

`ServiceMetrics` revision rows appear as traffic arrives for each revision. Traffic recorded before revision attribution existed is grouped under an empty revision name.

`TunnelMetrics` covers one agent connection. A tunnel's history belongs to that connection: when an agent reconnects it gets a new `Tunnel` with a new name, and its metrics start over with it. Read `vpcnetworks/{name}/metrics?include=tunnels` for a view that spans reconnects, and `tunnels/{name}/metrics` for one connection.

The `Tunnel` object is removed when the agent disconnects, but its measurements are kept for the retention of `relay_1m`. `tunnels/{name}/metrics` keeps answering for a disconnected connection for that long, so the history of a connection outlives the connection itself. A name with no measurements left, and one that never existed, are both a `404`.

## Custom metrics

A `Metric` you create is a saved query. It appears in `kubectl get metrics`, works with `series`, and can be pulled into a snapshot with `metric=`.

```yaml title="metric.yaml"
apiVersion: metrics.apoxy.dev/v1alpha1
kind: Metric
metadata:
  name: v2.errors
spec:
  source: otel_logs
  type: counter
  description: Server errors on the /v2 path prefix.
  prql: |
    filter (url.path | text.starts_with "/v2/")
    filter http.response.status_code >= 500
    aggregate { n = count this }
```

Apply it with `apoxy apply -f metric.yaml`.

The fragment is **aggregate-only**: zero or more `filter` and `derive` steps followed by one `aggregate`. It states no `from`, no `group`, and no `time_bucket`, because the source comes from `spec.source`, the grouping from `groupBy`, the bucket from `step`, and the scope from the scope parameters. A fragment carrying any of those steps is rejected.

The server compiles the fragment when you write it and fills in `status`: the resolved `source`, the output `measures` with their types and units, the groupable `keys`, and a `Compiled` condition. You never write `status` yourself. A fragment that does not compile is never stored, so a broken recipe cannot surface later as a snapshot silently missing a measure.

Leave `spec.source` unset to have the server resolve it from the fields the fragment uses.

<Callout label="Reserved names" variant="warn">
The prefixes `http.`, `log.`, `upstream.`, `tls.`, `network.`, and `tunnel.` are reserved for built-in recipes, as is the exact name `series`. A write to a reserved name is rejected with `403`, so a recipe of yours can never shadow a built-in. The `envoy_1m` and `relay_1m` sources are reserved the same way: their schema moves with the built-in recipes, so a recipe of yours cannot read them.
</Callout>

## Limits

| Limit | Value |
|---|---|
| Series per response (`top`) | 50 |
| Buckets per series | 1500 |
| Total points (buckets x series x measures) | 20000 |
| Minimum `step` | the source granularity, rounded up |
| Maximum lookback on raw records | 31d |
| Maximum window for a grouped raw-record read | 24h |

Exceeding the point budget returns `400` naming all three factors, so you can lower `top`, widen `step`, or ask for fewer measures with `metric=`. Exceeding `top` is not an error: the response returns the top slice and sets `truncated` with `totalCount`.

## Errors

Failures are standard Kubernetes `Status` objects.

| Status | Meaning |
|---|---|
| `400` | Guardrail violation, or an unknown `groupBy` key, `include` token, or `scopeKind`. The message lists the valid values. |
| `403` | Write to a reserved recipe name. |
| `404` | The owner object does not exist. |
| `422` | The recipe does not compile against the current schema. |
| `429` | Concurrent query limit reached. Retry after the interval in the `Retry-After` header. |
| `503` | The metrics backend is unavailable. |

A project with no data yet returns `200` with an empty window rather than an error, so an empty result is not a failure.

## Caching and polling

Responses carry:

```
Cache-Control: max-age=60
```

Poll on the interval the response states rather than faster. Window bounds are aligned to `step` before a response is cached, so two reads a few seconds apart over the same window return the same result.

Treat `dataUpTo` as the edge of settled data. A bucket after it is still filling, and a client that charts it will show an apparent dip that recovers on the next poll.

## Where to next

- Work through the calls end to end in [Query gateway metrics](/docs/guides/query-gateway-metrics.md).
- Ask free-form questions over raw log records with [PRQL through the MCP server](/docs/reference/mcp.md).

---

**Navigation** (Reference)

- Previous: [MCP server](/docs/reference/mcp.md)
- Next: [HTTP APIs](/docs/reference/http-apis.md)
- All pages: [index](/docs/llms.txt)