Infra Projects

An infra.apoxy.dev/v1alpha1 Project is the shard-local desired-state record for one Apoxy project.

Desired state

Each infra-apiserver database represents one shard. A Project in that database assigns the project to the shard:

• Object presence assigns the project to the shard.
• Object deletion removes the assignment.
• spec.apiServerEndpoint changes reconnect consumers to the new project API server.
• Health status never adds or removes membership.

The project UUID is metadata.name. spec.shardID and the infra.apoxy.dev/shard-id label protect consumers from wrong-shard wiring and stale database restores.

Write ownership

Write access is split by subresource:

• Project spec and deletion - The Cosmos project lifecycle identity owns creates, updates, and deletes.
• Project status - The infra-apiserver loopback identity owns health observations.
• Reads and watches - Authenticated shard services list and watch Projects for membership.

The setup workflow writes the Project after it persists the shard assignment and before it provisions the namespace, storage, or project API server. A failed provisioning run leaves the Project present. Rerun Project Setup to repair provisioning without reconstructing membership.

Project Teardown deletes the Project before later resource cleanup. It does not delete the Cosmos project row or shard assignment.

Agent CA

Each Project holds the certificate authority (CA) for the VPC agent certificates of that project:

• spec.agentCA.certificates - The CA certificates, as PEM. They permit only agent identities of this project. During a CA rotation, the list also holds the certificate of the next CA or the certificates of the old CAs.
• spec.agentCA.sealedKey and spec.agentCA.wrappedDataKey - The private key of the CA that signs, in sealed form. Only the API server of the project can open it.
• spec.agentCA.nextCertificate, spec.agentCA.nextSealedKey, and spec.agentCA.nextWrappedDataKey - The next CA during a CA rotation: its certificate, as PEM, and its private key in sealed form. They are empty at all other times.
• spec.relayRoots - The CA certificates that relay certificates chain to, as PEM. Empty means the system roots.
• spec.trustSignature - The signature of Cosmos on the Project name, spec.apiServerEndpoint, spec.agentCA.certificates, and spec.relayRoots: the key ID (keyID), the time (signedAt), and the signature bytes (signature). Cosmos writes it with each change of these fields. A relay refuses a change of these fields that has a signature which is not valid, and keeps the last values that it accepted.

Project Setup makes the CA when the Project has none, and it never replaces a CA. Only a CA rotation that an operator starts replaces it. Relays read spec.agentCA.certificates and keep them in their snapshot file. The sealed keys are never in that file.

The API server of the project signs agent certificates with this CA, and gives spec.agentCA.certificates and spec.relayRoots to each agent that enrolls. A relay accepts an agent certificate of a project only when it chains to spec.agentCA.certificates of that project.

Health status

The infra-apiserver probes https://<spec.apiServerEndpoint>/readyz. The first successful response sets Ready=True. Three consecutive failures set Ready=False; one successful probe recovers immediately.

Endpoint changes reset health to Ready=Unknown until the new generation responds. Healthy probe timestamps are persisted at most once per minute to limit database writes and watch traffic.

Consumers treat status as observation only. They keep an unhealthy Project assigned and let existing connection retry logic recover when its API server returns.

Failure recovery

• Setup failed after Project creation - Rerun Project Setup. The workflow reuses the assignment and upserts the same Project.
• Project is missing from infra-apiserver - Rerun Project Setup to recreate it.
• Agent CA is missing - Rerun Project Setup. A restored project gets a new CA.
• Endpoint is wrong - Correct the lifecycle inputs and rerun Project Setup. Do not patch the Project directly.
• Shard move requested - Use the dedicated shard migration workflow. Ordinary Project Setup rejects moves.
• Health probe failed - Repair the project API server or endpoint. Health failure never triggers teardown.

Watch reconnects preserve the last known desired membership. A transport failure does not produce an empty membership snapshot.

Shard services

Shared shard services (backplane, tunnelproxy, relay, and the shared workerd-manager) take their membership only from the infra Project list/watch. Each one requires:

• --shard_id - The shard UUID. It must equal spec.shardID of the Projects the service connects to.
• --infra_addr - The infra-apiserver that holds the shard's Projects. --infra_insecure disables TLS verification for it in development.

A service stays out of service until its first list succeeds. Status-only Project updates do not reconnect project clients. An update that changes only the agent CA or the relay roots does not reconnect them either.