Metrics API

The metrics API serves traffic measurements for a project from the project API server, backed by aggregated telemetry. It has two shapes: a snapshot on an object (current totals for one Gateway, route, Proxy, Service, VPC network, or tunnel) and a series query (one or more lines over time). Both read the same measurements, so a tile and the chart under it agree.

The rule for picking one: one object and you want its current numbers, read /metrics on that object. Many objects, or any view over time, use metrics/<name>/series.

This API measures traffic. For the health of the proxy process itself, read the envoy.* recipes: see Proxy health in the metrics API. A gateway you run yourself publishes the same signals as Prometheus series on the pod as well, covered in Gateway pod metrics (self-hosted) at the bottom of this page.

Endpoint and authentication

All requests go to your project API server:

$terminalTXTCOPY

https://<project-id>.api.apoxy.dev

Authenticate with an API key in the X-Apoxy-API-Key header, the same as every other call to this API:

$terminalTXTCOPY

X-Apoxy-API-Key: <api-key>

Authorization: Bearer <api-key> is also accepted. To create a key, see Authentication in the MCP server reference. Requests without a valid key receive 401 Unauthorized.

Resources

The group is metrics.apoxy.dev/v1alpha1. All resources are cluster-scoped, and no project identifier appears in any path or parameter because the API server serves exactly one project.

A Metric is a named recipe: a query fragment plus display preferences. The built-in recipes are managed for you; you can also save your own. A MetricSource describes what a recipe can read and group by.

Built-in metrics

Every project starts with these recipes. They carry the metrics.apoxy.dev/managed: "true" label.

http.requests and http.errors are plain counts and carry no unit, so they contribute nothing to the units map in a response. upstream.connections reports live upstream connections, and tls.handshakes counts downstream TLS connections established in the window. http.latency percentiles come from fixed latency buckets, and the largest bound is 60 s, so a percentile that falls past that bound reads as 60000 ms.

The collector stores metrics whose names start with envoy. or apoxy. when it can identify their project. Raw samples remain in ClickHouse for 72 hours, including new metric names under either prefix.

upstream.connections requires a recognized route or Backend. tls.handshakes requires a Gateway and listener. Proxy health metrics require both a Proxy name and a replica name. Samples without this identity remain in raw storage and do not contribute to the API results.

envoy.bandwidth counts the bytes Envoy moved on the wire, which is why it carries traffic that http.bytes cannot see: http.bytes counts request and response bodies at the end of a request, so a WebSocket or another long-lived stream stays at zero until it closes.

The catalog also holds the other envoy.* recipes, which measure the proxy process rather than the traffic through it. See Proxy health in the metrics API.

The network.* and tunnel.* recipes measure the tunnels of a VPC network rather than HTTP traffic. network.bytes and network.packets count every packet of payload a tunnel carries, so east-west traffic and non-HTTP protocols show up in them and not in http.*. network.keepalives counts the keep-alive frames that keep the tunnel path active; they carry no payload, so they are counted apart and a tunnel that carries no traffic reports keep-alives with zero bytes and zero packets. network.tunnels reports how many tunnels a network had connected, counted across every relay that serves it; its peak adds the largest count each relay saw, so it is an upper bound. tunnel.rtt reports the round trip time between an agent and the relay it connects to. network.tunnels is reported on the network alone, so it is read with scopeKind=VPCNetwork; each of the others is readable with scopeKind=VPCNetwork or scopeKind=Tunnel.

List the catalog with the resolved source and unit for each recipe:

$terminalSHCOPY

kubectl get metrics

You should see NAME, TYPE, UNIT, and SOURCE columns. A GET on one name returns the full recipe, including the measures and groupable keys the server derived from it.

Proxy health in the metrics API

The envoy.* recipes measure the proxy process rather than the traffic through it, so a series query and a snapshot answer for its health the way they do for traffic. They are in the catalog of every project on the managed platform. On a gateway you run yourself they come with the apoxy-gateway-ee chart, which installs the metrics pipeline and stores measurements in ClickHouse inside your cluster. Apoxy gives you a pull key. It reads the chart and its images from oci://us-west1-docker.pkg.dev/apoxy-internal/ee.

Two values decide the install: projectID names the project the measurements belong to, and the ClickHouse choice is either the single node the chart brings up or an external DSN pointing at one you already run. The images run on amd64 and arm64 nodes. The single node keeps its own query and metric logs for clickhouse.systemLogs.retentionDays days, 7 by default, and its trace and text logs are off.

All nine read envoy_1m and group by replica by default, so a single replica in trouble stays visible in a fleet reading. envoy.dns_cache groups by cache instead, because a project can run several.

envoy.bandwidth counts connection bytes rather than request bodies, so a long-lived stream shows up while it runs instead of only when it closes. Read it with scopeKind=Gateway for the client side and with scopeKind=Proxy for the upstream side.

ProxyMetrics reports the same recipes per replica in perReplica, next to the replicas counts it takes from the Proxy status. Each entry names one replica and carries that replica's measures alone. A replica that reported nothing in the window is absent from the list rather than present with zeros, and the list is empty until measurements arrive.

In the console

The console reads these recipes on the Proxy tab of a gateway, next to Telemetry. The tab shows how many replicas are connected, the Envoy exits in the selected window, the p99 request latency through the proxy, the active downstream connections against their summed ceiling, the open file descriptors against their summed limit, and the highest heap use of a replica. Under that, one row per replica gives its status, which reads Gone for a replica the Proxy no longer lists, how long it has been connected, its exits in the selected window, its queue overflows, its file descriptors, its DNS cache and its watchdog misses. Telemetry and Proxy share one window, so the range you pick on one tab opens on the other.

The Telemetry tab of the same gateway draws a Bandwidth chart from envoy.bandwidth: the client side for the gateway, the upstream side for its proxy.

Four charts draw the window: active downstream connections against the ceiling they are measured against, request latency with the downstream and the upstream distributions drawn apart, heap pressure as a percent, and pending upstream requests. Every Envoy exit is marked on all four, so a jump in one number lines up with the restart behind it. A filter narrows the charts to the replicas you pick, matching on name, locality, address or status, with presets for the replicas that need attention, the ones that restarted Envoy in the window, and each locality. An empty selection means every replica, and the table always lists them all.

When the numbers are missing or old, the tab says why:

• Proxy health is not available - this apiserver does not serve the metrics API, so there is no per-replica health to read. Read the replicas from the Proxy object instead.
• No Proxy attached - the gateway names no Proxy, so there is no data plane to report on.
• Shared data plane - the gateway runs on the shared data plane. Per-replica health is reported for dedicated Proxies and for gateways you run yourself.
• No replicas reported - no replica is connected, or no replica reported a measurement in the selected window.
• Metrics backend not configured - the API is served but no backend is behind it, and the card names what the install is missing. Replica status keeps coming from the Proxy object.
• Could not read - one of the reads the page needs came back with an error, or the network did. Nothing is drawn under the message, because a failed read says nothing about the replicas. Select Retry to read again.
• Stale data - the newest measurement is old, so the numbers on the page have stopped moving. Select Retry to read again.

Metric sources

A source is what a recipe reads. kubectl get metricsources lists them with their granularity and retention.

The http.* recipes read http_1m. Series and snapshot reads over longer windows are served at coarser granularity automatically, so history stays available past the finer source's retention without any change to your request.

A snapshot evaluates recipes from several sources, and the sources keep different amounts of history. A window wider than a source's retention drops that source's recipes from the snapshot and reports the rest: vpcnetworks/{name}/metrics?window=1440h answers with the http.* recipes, which the 400 day rollup serves, and without the network.* and tunnel.* recipes, which are kept for 30 days. A window no evaluated source can reach is a 400 instead - tunnels/{name}/metrics reads relay_1m alone, so a 60 day window there names the source and its maximum. Naming such a recipe explicitly with metric= is a 400 as well, rather than a quiet drop.

log.severity reads raw log records, which is why it is series-only: snapshots evaluate the aggregated recipes and skip raw-log ones. A grouped read over raw records is limited to a 24 hour window.

MetricSource.status.fields lists each field with a role:

• time - the bucket column.
• key - a groupable dimension. These are exactly the values groupBy accepts.
• measure - a value column a recipe aggregates.

Each field also reports discovered, which is true for a field found by sampling log attributes (so it can disappear) and false for a fixed column.

Series

Runs one recipe over a window and returns a MetricSeriesSet.

Parameters

The window is half-open, [since, until). Relative durations accept h, m, s, and d: -6h, -7d, -1.5d, and +30m all parse. A bare 6h with no sign does not, and neither does -1w.

Buckets count whole steps forward from since rather than from the clock, so the first point carries since, and a step equal to the window returns exactly one point per series.

Group-by keys

groupBy takes exactly one key:

The network.* and tunnel.* recipes read a different source and take their own keys:

A key that the recipe's source does not carry returns 400 with the valid keys listed. The authoritative list for any recipe is its status.keys.

Response

$terminalJSONCOPY

{ "kind": "MetricSeriesSet", "apiVersion": "metrics.apoxy.dev/v1alpha1", "metric": "http.requests", "scopeKind": "Gateway", "scopeName": "prod", "since": "2026-08-20T15:00:00Z", "until": "2026-08-20T21:00:00Z", "step": "5m0s", "dataUpTo": "2026-08-20T21:00:00Z", "truncated": false, "totalCount": 7, "units": {}, "series": [ { "labels": { "route_name": "api" }, "points": [ { "timestamp": "2026-08-20T15:00:00Z", "values": { "total": 5210, "status_4xx": 140, "status_5xx": 12 } }, { "timestamp": "2026-08-20T15:05:00Z", "values": { "total": 5188, "status_4xx": 131, "status_5xx": 9 } } ] } ] }

Points are sparse: a bucket with no traffic is omitted rather than returned as zero. Compare a missing bucket against dataUpTo to tell missing data from a partial trailing bucket.

Fleet views

A one-bucket series is how you get one row per object. Set step equal to the window and group by the owner key:

$terminalTXTCOPY

metrics/http.requests/series?scopeKind=Project&groupBy=gateway&since=-1h&step=1h

That returns one point per Gateway, ranked by orderBy and bounded by top. The same shape with scopeKind=Gateway&groupBy=route_name gives one row per route.

Snapshots

A snapshot is every applicable recipe evaluated over one window for one object, in a single call. It mounts as a metrics subresource on the owner.

Every snapshot carries timestamp, window, since, until, dataUpTo, a metrics map keyed by recipe name, and a units map. The window parameters are the same as for series.

$terminalJSONCOPY

{ "kind": "GatewayMetrics", "apiVersion": "metrics.apoxy.dev/v1alpha1", "metadata": { "name": "prod" }, "timestamp": "2026-08-20T21:00:00Z", "window": "24h0m0s", "since": "2026-08-19T21:00:00Z", "until": "2026-08-20T21:00:00Z", "dataUpTo": "2026-08-20T21:00:00Z", "metrics": { "http.requests": { "total": 1842113, "status_2xx": 1790220, "status_4xx": 48120, "status_5xx": 3773 }, "http.latency": { "p50": 38, "p95": 190, "p99": 412 } }, "units": { "p50": "ms", "p95": "ms", "p99": "ms" }, "listeners": [ { "name": "https", "metrics": { "http.requests": { "total": 1839900, "status_5xx": 3773 } }, "truncated": true, "totalCount": 37 } ] }

Nesting

The default response is the owner totals plus the first nesting level, with no leaf rows. Add them with include:

• include=routes on a Gateway adds routes[] under each listener.
• include=rules, include=backends on an HTTPRoute add rules[] and backends[].
• include=revisions on a Service.
• include=services, include=tunnels on a VPC network.
• include=all means every token for that kind.

include may repeat or take a comma-separated list. An unknown token returns 400.

truncated and totalCount sit on whichever container was cut. A Gateway cuts routes per listener, so they appear on each listener. HTTPRouteMetrics, ServiceMetrics, and VPCNetworkMetrics cut their own leaf lists, so they appear on the object itself and count every list the read asked for.

A VPC network reports its services and its tunnels from different measurements: services[] covers gateway traffic to the network's VPC services, and tunnels[] covers everything the tunnels carried. The two do not add up to each other. network.tunnels counts the network rather than one connection, so it is reported in the network's own metrics map and never as an entry of tunnels[]; every entry of tunnels[] names a connection, and totalCount counts those entries alone.

For complete per-route coverage instead of the top slice, use a one-bucket series grouped by route_name.

orderBy names the measure a nested list is ranked by. A snapshot can nest by levels that come from different sources, and a level is ranked by the named measure only when a recipe of that level reports it; a level whose recipes do not report it keeps its own default order, which is the request count where there is one. include=all&orderBy=bytes on a VPC network therefore ranks tunnels[] by bytes and leaves services[] in its own order. A measure no evaluated recipe reports anywhere is a 400.

Restricting recipes

metric=<name> limits which recipes are evaluated. The parameter repeats and does not take a comma-separated list:

$terminalTXTCOPY

?metric=http.requests&metric=http.latency

Snapshots evaluate the managed recipes by default. Naming a recipe of your own with metric= evaluates that one too.

ProxyMetrics additionally carries replicas, which reports connected replica counts from the Proxy's own status rather than from traffic. Proxy metrics cover traffic on the Gateways attached to that Proxy.

ServiceMetrics revision rows appear as traffic arrives for each revision. Traffic recorded before revision attribution existed is grouped under an empty revision name.

TunnelMetrics covers one agent connection. A tunnel's history belongs to that connection: when an agent reconnects it gets a new Tunnel with a new name, and its metrics start over with it. Read vpcnetworks/{name}/metrics?include=tunnels for a view that spans reconnects, and tunnels/{name}/metrics for one connection.

The Tunnel object is removed when the agent disconnects, but its measurements are kept for the retention of relay_1m. tunnels/{name}/metrics keeps answering for a disconnected connection for that long, so the history of a connection outlives the connection itself. A name with no measurements left, and one that never existed, are both a 404.

Custom metrics

A Metric you create is a saved query. It appears in kubectl get metrics, works with series, and can be pulled into a snapshot with metric=.

$metric.yamlYAMLCOPY

apiVersion: metrics.apoxy.dev/v1alpha1 kind: Metric metadata: name: v2.errors spec: source: otel_logs type: counter description: Server errors on the /v2 path prefix. prql: | filter (url.path | text.starts_with "/v2/") filter http.response.status_code >= 500 aggregate { n = count this }

Apply it with apoxy apply -f metric.yaml.

The fragment is aggregate-only: zero or more filter and derive steps followed by one aggregate. It states no from, no group, and no time_bucket, because the source comes from spec.source, the grouping from groupBy, the bucket from step, and the scope from the scope parameters. A fragment carrying any of those steps is rejected.

The server compiles the fragment when you write it and fills in status: the resolved source, the output measures with their types and units, the groupable keys, and a Compiled condition. You never write status yourself. A fragment that does not compile is never stored, so a broken recipe cannot surface later as a snapshot silently missing a measure.

Leave spec.source unset to have the server resolve it from the fields the fragment uses.

Limits

Exceeding the point budget returns 400 naming all three factors, so you can lower top, widen step, or ask for fewer measures with metric=. Exceeding top is not an error: the response returns the top slice and sets truncated with totalCount.

Errors

Failures are standard Kubernetes Status objects.

A project with no data yet returns 200 with an empty window rather than an error, so an empty result is not a failure.

Caching and polling

Responses carry:

$terminalTXTCOPY

Cache-Control: max-age=60

Poll on the interval the response states rather than faster. Window bounds are aligned to step before a response is cached, so two reads a few seconds apart over the same window return the same result.

Treat dataUpTo as the edge of settled data. A bucket after it is still filling, and a client that charts it will show an apparent dip that recovers on the next poll.

Gateway pod metrics (self-hosted)

A gateway you run in your own cluster exposes Prometheus metrics on the gateway pod. They describe the Envoy process rather than the traffic through it, so they answer what the API above cannot: whether Envoy restarted, how close it is to a limit, and where a request spent its time. The series carry the apoxy_backplane_ prefix, after the process that supervises Envoy in the pod.

Endpoint

/envoy/metrics answers 200 while Envoy is restarting. Envoy's own stats are missing from that response and apoxy_backplane_envoy_up reads 0, so alert on that gauge rather than on a failed scrape.

Chart values:

• backplane.envoyPodMonitor - creates the PodMonitor for /envoy/metrics.
• backplane.prometheusRule - creates the PrometheusRule holding the alerts below.
• backplane.grafanaDashboard - creates the dashboard ConfigMap.
• backplane.autoscaling.podsMetrics - adds type: Pods metrics to the gateway HPA, so you can scale on a series such as envoy_http_downstream_cx_active. Your cluster needs prometheus-adapter to publish the named series as a pods metric.

Envoy restarts

reason takes four values, and code qualifies it:

• exit - the process returned an exit status, which code carries. exit with code="0" is a clean shutdown, and code="1" is usually a configuration Envoy refused at start.
• signal - a signal ended the process, and code carries the signal name.
• oom_kill - the container reached its memory limit and the kernel killed the process.
• start_failed - the process never started, and code is empty.

The same exit appears in two other places. Proxy.status.replicas[] carries envoyRestarts and lastEnvoyExit for each connected replica:

$terminalSHCOPY

apoxy proxy get my-proxy -o yaml

And the gateway logs one structured line per exit, Envoy exited, with the reason, the exit status or signal name, and how long the process ran.

Limits and the series that show pressure

Every limit below has a ceiling you can read, a current value, and usually a pressure signal that moves before the limit bites.

Reading that table:

• The two Envoy pressure gauges are percentages from 0 to 100, not ratios. Compare against 80, not 0.8.
• The heap and container memory rows need a memory limit on the gateway container. Without one there is no ceiling to publish, apoxy_backplane_envoy_limit_memory_bytes is absent, and Envoy's heap monitor has no threshold to measure pressure against.
• Heap and resident memory are different numbers, and neither covers the other. envoy_server_memory_allocated counts Envoy's own allocator and misses every Edge Function library loaded into the same process, so resident memory can climb while heap pressure stays flat. apoxy_backplane_envoy_rss_bytes is the Envoy process alone, while apoxy_backplane_envoy_limit_memory_bytes is the container limit for the whole pod, so the headroom that ratio shows is optimistic. The container limit is the one the kubelet kills against.
• Pending requests are unbounded unless a Backend sets a limit, so a slow upstream grows the queue instead of overflowing it. Watch the level with envoy_cluster_upstream_rq_pending_active, which is what EnvoyPendingQueueHigh alerts on; upstream_rq_pending_overflow and rq_pending_open stay at zero until a Backend sets a finite limit.
• <name> in the DNS cache series is the Backend the dynamic proxy serves, so a Backend named dynamic-proxy gives envoy_dns_cache_dynamic_proxy_num_hosts.
• The apoxy_backplane_net_tcp_* counters cover connections the kernel dropped or reset before Envoy saw them, which is what a client experiences during a restart or a worker stall. No Envoy stat records those.
• The watchdog series are the totals over all threads. A per-thread breakdown is published beside them, either on an envoy_thread_name label or in the metric name.

Time inside the proxy

Three histograms split where a request spends its time:

• envoy_http_downstream_rq_time - the whole request, from the first downstream byte to the last response byte.
• envoy_cluster_upstream_rq_time - the part your upstream owns.
• envoy_cluster_upstream_cx_connect_ms - connection setup to the upstream, which upstream_rq_time does not include.

The difference between the first two is the proxy's own share: filters, Edge Functions, and time waiting in the pending queue. Three access-log fields split that share per request, so a slow tail can be attributed to a single call rather than to a percentile:

• filter_duration - from the first downstream byte to the first byte sent upstream.
• request_tx_duration - from the first downstream byte to the last byte sent upstream.
• response_duration - from the first downstream byte to the first upstream response byte.

In the hosted logs the same values are the attributes http.request.filter_duration_ms, http.request.tx_duration_ms and http.response.duration_ms, next to http.request.duration_ms.

Recommended alerts

The chart's PrometheusRule ships these. Tune the thresholds to your traffic.

EnvoyConfigRejected is the one with no other symptom. Envoy keeps the last good configuration when it rejects an update, so the pod stays healthy and serves stale routes until someone notices.

The same health signals are readable through the metrics API as the envoy.* recipes. See Proxy health in the metrics API.

Scraping the pipeline itself

The apoxy-gateway-ee chart also watches the pipeline that stores those measurements. With monitoring.enabled (the default) it creates three ServiceMonitor objects, <release>-apoxy-ee-apiserver, <release>-apoxy-ee-collector and <release>-apoxy-ee-clickhouse, and a PrometheusRule named <release>-apoxy-ee-pipeline holding the apoxy-gateway-ee.pipeline alert group. Its alerts fire when a rollup falls behind, when the collector refuses or drops datapoints, and when ClickHouse is down or low on disk. All four render only on a cluster that serves monitoring.coreos.com/v1, so an install without the Prometheus operator skips them.

Where to next

• Work through the calls end to end in Query gateway metrics.
• Ask free-form questions over raw log records with PRQL through the MCP server.
• Diagnose a restart with Envoy restarted inside the gateway.