EC2 kernel-mode VPC tunnels

This guide deploys tunnel agent on Amazon Linux 2023 and exposes a demo backend publicly via Apoxy Edge Gateway. The EC2 doesn't allow any inbound traffic (no inbound security-group rules).

The request path is:

$terminalTXT

HTTPS → Gateway → HTTPRoute → Backend → VPCService → relay → EC2 → service

Check prerequisites

The Apoxy CLI is authenticated to the intended project. Minimum release: v0.21.0
The managed Apoxy project is ready.
AWS CLI credentials can create CloudFormation, EC2, IAM, Secrets Manager, and SSM resources.
The target subnet assigns or permits a public IPv4 address and has outbound internet access.
jq is installed locally.

$terminalSH

apoxy auth --check
apoxy gateway get default
apoxy vpc network get default
apoxy vpc relay list
aws sts get-caller-identity

Deploy the agent

Download the maintained template:

$terminalSH

curl --fail --silent --show-error \
  --output apoxy-tunnel-agent-ec2.yaml \
  https://apoxy-cloudformation-699259675044.s3.us-west-2.amazonaws.com/tunnel-agent/apoxy-tunnel-agent-ec2.yaml

Set the AWS and Apoxy values. Choose the relay nearest the instance from apoxy vpc relay list.

$terminalSH

export AWS_REGION=<your AWS region>
export VPC_ID=<your VPC ID>
export SUBNET_ID=<your subnet ID>
export RELAY_ADDRESS=<your relay address>
export APOXY_VPC_TOKEN="$(
  apoxy vpc network get default -o json | jq -jr '.status.credentials.token'
)"
test -n "$APOXY_VPC_TOKEN"

Deploy the stack:

$terminalSH

aws cloudformation deploy \
  --region "$AWS_REGION" \
  --stack-name apoxy-tunnel-agent \
  --template-file apoxy-tunnel-agent-ec2.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
    VpcId="$VPC_ID" \
    SubnetId="$SUBNET_ID" \
    VpcToken="$APOXY_VPC_TOKEN" \
    AgentName=ec2-private-api \
    VpcName=default \
    RelayAddress="$RELAY_ADDRESS"
unset APOXY_VPC_TOKEN

VpcToken is a CloudFormation NoEcho parameter. The stack stores it in Secrets Manager, copies it to a root-only file during boot, and mounts that file read-only into the agent container. Keep the token out of scripts and CI logs.

Credential visibility

The alpha client does not yet accept a token file directly. The container reads the root-only file at startup, but the expanded token is visible in the agent process arguments to an operator with host-level access. Restrict instance administration and rotate the token after a credential exposure.

The template creates:

An Amazon Linux 2023 EC2 instance and egress-only security group.
IMDSv2 enforcement with a one-hop metadata response limit.
An instance role with only SSM access and permission to read its tunnel secret.
A Docker systemd unit with host networking, NET_ADMIN, and /dev/net/tun.
A dual-stack demo server on port 8080.

Verify the kernel tunnel

Read the instance ID and open an SSM session:

$terminalSH

export INSTANCE_ID="$(
  aws cloudformation describe-stacks \
    --region "$AWS_REGION" \
    --stack-name apoxy-tunnel-agent \
    --query 'Stacks[0].Outputs[?OutputKey==`InstanceId`].OutputValue' \
    --output text
)"
aws ssm start-session --region "$AWS_REGION" --target "$INSTANCE_ID"

On the instance, confirm that both services, the health check, and apoxy0 are ready:

$EC2 sessionSH

sudo cloud-init status --wait
sudo systemctl is-active apoxy-demo-backend.service apoxy-tunnel.service
curl --fail http://127.0.0.1:8081/healthz
ip -brief address show apoxy0
curl --fail http://127.0.0.1:8080/

The overlay addresses are assigned dynamically and appear in the ip output. Back on your workstation, apoxy vpc tunnel list should show an active tunnel labeled tunnel.apoxy.dev/name=ec2-private-api.

Publish the EC2 service

Create a VPCService that selects the agent:

$ec2-vpc-service.yamlYAML

apiVersion: vpc.apoxy.dev/v1alpha1
kind: VPCService
metadata:
  name: ec2-private-api
spec:
  networkRef:
    name: default
  selector:
    matchLabels:
      tunnel.apoxy.dev/name: ec2-private-api

$terminalSH

apoxy apply -f ec2-vpc-service.yaml
apoxy vpc service get ec2-private-api -o yaml

Continue when Ready=True and at least one endpoint is present.

Attach your hostname to the default Gateway. Replace ec2-api.example.com; for an Apoxy-managed zone, also set spec.zone to the zone root.

$ec2-domain.yamlYAML

apiVersion: core.apoxy.dev/v1alpha3
kind: DomainRecord
spec:
  name: ec2-api.example.com
  tls: {}
  target:
    ref:
      group: gateway.apoxy.dev
      kind: Gateway
      name: default

$terminalSH

apoxy apply -f ec2-domain.yaml

Finally, route port 8080 to the VPCService's internal name:

$ec2-route.yamlYAML

apiVersion: core.apoxy.dev/v1alpha2
kind: Backend
metadata:
  name: ec2-private-api
spec:
  endpoints:
  - fqdn: ec2-private-api.default.vpc.apoxy.net
---
apiVersion: gateway.apoxy.dev/v1
kind: HTTPRoute
metadata:
  name: ec2-private-api
spec:
  parentRefs:
  - name: default
    port: 443
  hostnames:
  - ec2-api.example.com
  rules:
  - backendRefs:
    - group: core.apoxy.dev
      kind: Backend
      name: ec2-private-api
      port: 8080

$terminalSH

apoxy apply -f ec2-route.yaml
apoxy domain list \
  --field-selector spec.name=ec2-api.example.com \
  -o yaml
curl https://ec2-api.example.com/

A successful request returns Hello through the Apoxy EC2 kernel tunnel.

Bind real services dual-stack

A VPCService publishes IPv4 and IPv6 overlay addresses. Bind the workload to both families, for example, :: on a dual-stack Linux socket, or provide listeners for each family. An IPv4-only listener can be refused when the data plane selects the IPv6 endpoint.

Clean up

Delete the Apoxy objects before terminating their selected agent:

$terminalSH

apoxy delete -f ec2-route.yaml
apoxy delete -f ec2-domain.yaml
apoxy delete -f ec2-vpc-service.yaml
aws cloudformation delete-stack \
  --region "$AWS_REGION" \
  --stack-name apoxy-tunnel-agent
aws cloudformation wait stack-delete-complete \
  --region "$AWS_REGION" \
  --stack-name apoxy-tunnel-agent