Troubleshooting

This guide helps you diagnose common issues with route attachments, tunnel connectivity, and Kubernetes mirroring.

Routes not attaching

When a route doesn't show up in apoxy gateway get <name>, work through this checklist:

Confirm the right project - check your Apoxy config has the correct project selected.
Confirm the Gateway exists - run apoxy gateway list and verify the Gateway name.
Check parentRefs - the route must reference an existing Gateway by the correct name.
Check the route type - an HTTPRoute won't attach to a TCP-only listener, and vice versa.
Check hostnames - if the Gateway listener restricts hostnames, the route's hostnames must overlap.
Check sectionName - if specified, it must match a listener name on the Gateway.

Inspect both sides:

$terminalSH

apoxy gateway get default
apoxy gateway routes get http/my-route

Compare the route's parentRefs against the Gateway's listener names and protocols.

Tunnel not connecting

If apoxy tunnel run starts but traffic isn't flowing:

Check the TunnelNode exists - run apoxy tunnel list and verify the name matches what your route references.
Check the tunnel is running - the tunnel client must be actively connected. Look for connection status in the TUI or logs.
Check the local service - verify the service is actually listening on the port you specified. Try curl localhost:<port> from the same machine.
Check the route backend - the HTTPRoute's backendRefs must use kind: Backend and reference a Backend that exists.
Check the Backend exists - run apoxy backend get <name> and verify its endpoint FQDN matches {tunnelNodeName}.tun.apoxy.net.

Kubernetes mirroring not working

If you're using apoxy k8s install and resources aren't appearing in Apoxy:

Check the controller is running - look for the Apoxy controller pod in your cluster.
Check the mirror mode - --mirror gateway only mirrors Gateway API resources. Use --mirror all if you need Ingress as well.
Check cluster connectivity - the controller needs outbound access to the Apoxy API.
Check resource types - the controller mirrors Gateway, HTTPRoute, and related types. Custom resources won't be mirrored automatically.

Common error patterns

"no matching parent"

The route's parentRefs don't match any Gateway. Double-check the Gateway name and that it exists in the same project.

Traffic reaches the Gateway but returns 404

The Gateway is accepting the connection but no route matches the request. Apoxy serves a branded 404 page with a trace= line - that value is the request ID, and it appears as x-request-id in your project's access logs. Include it when reporting an issue. Check:

The route's hostnames match the request's Host header.
The route's path matchers (if any) match the request path.
The route is actually attached - verify with apoxy gateway get <name>.

If you see the branded page, the request reached Apoxy and matched your domain - the problem is route matching, not DNS or TLS.

Traffic reaches the tunnel but the backend is unreachable

The tunnel is working but the local service isn't responding. Check:

The port in the route's backendRefs matches where the service is actually listening.
The service is bound to an address the tunnel can reach (e.g., 0.0.0.0 or localhost, not just 127.0.0.1 if the tunnel is in a container).

523 Backend Unreachable

The route matched but its backend hostname doesn't resolve to any connected endpoint - either the hostname has no tunnel bound to it (check for typos in backendRefs), or every agent for that tunnel is disconnected. The access log carries envoy_response_flags: UH. Check:

The backendRefs hostname matches an existing tunnel name exactly.
The tunnel agent is connected - verify with apoxy tunnel list.

A 523 always originates from Apoxy, never from your backend, so it's distinguishable from a 503 your service returns itself. Like the 404 page, the branded 523 page carries a trace= request ID you can correlate with access logs.

Intermittent 503s with `envoy_response_flags: UC`

Sporadic 503s on otherwise-healthy traffic - with UC in the access log and no corresponding backend restarts - are almost always a backend HTTP keep-alive idle timeout that's shorter than Apoxy's pooled-connection idle. Uvicorn (5s) and Gunicorn (2s) defaults are the most common culprits. See Backend keep-alive for per-server tuning.

Envoy restarted inside the gateway

A burst of connection resets across every route at the same instant, with no backend restart behind it, usually means the Envoy process in the gateway pod exited and came back. The pod around it keeps running, so a pod restart count stays at zero and the event is invisible from outside. Confirm it and find the reason:

apoxy proxy get <name> -o yaml - status.replicas[].envoyRestarts and .lastEnvoyExit carry the count and the last exit per connected replica.
The gateway logs one Envoy exited line per exit with the reason, the exit status or signal, and how long the process ran.
On a self-hosted gateway, apoxy_backplane_envoy_up and apoxy_backplane_envoy_exits_total chart the same event. See Gateway pod metrics.

The metrics API reads the same event over a window: envoy.exits reports each replica's exits by reason and code, which separates one bad replica from a fleet-wide restart. exits is the count inside the window, so a series of it spikes in the minute Envoy died; total is the count since the replica started, so a tile of it tells a replica that has ever crashed from one that has not. Read envoy.connections beside it for the downstream connections that replica was carrying, and see Proxy health in the metrics API.

Requests in flight at the exit are lost and never reach an access log. An exit with reason oom_kill means the container passed its memory limit - raise the limit or add replicas.

Getting more information