Getting startedGuidesReferenceChangelog
Apoxy:// Docs / Getting started / Local development

Local development

How `clrk dev` brings up a complete CLRK stack on your laptop, and the flags you'll reach for as you iterate.

clrk dev is the entry point for everything you'd do locally: explore the CRDs, develop agents, change egress policy, watch telemetry land. It boots a complete CLRK stack on a local k3d cluster - no existing Kubernetes cluster required - and persists state under ~/.clrk/ so a Ctrl-C followed by another clrk dev reattaches to the same world.

What it brings up

A single clrk dev invocation starts a k3d cluster (one docker container) and applies the controller-manager and a default WorkerPool as in-cluster workloads:

ComponentWhat it isImage
k3d-clrk-dev-server-0The k3d/k3s server docker container with an embedded apiserver, published on localhost at a free port chosen at startup (recorded in ~/.clrk/kubeconfig.host as https://localhost:<port>). With a local registry enabled there is also a clrk-registry docker container.rancher/k3s:v1.34.1-k3s1
controller-managerAn in-cluster Deployment in the clrk namespace running the reconcilers for every CLRK CRD. It also serves the embedded web console (see The web console).us-west1-docker.pkg.dev/apoxy-dev/public/clrk-controller-manager:<8-char-sha> (fallback :latest)
worker-0The sandbox runtime - libcontainer, ORAS image pulls, per-sandbox netns + TAP - running as an in-cluster pod behind the default-workers Deployment in the default namespace. Add more via --workers.us-west1-docker.pkg.dev/apoxy-dev/public/clrk-worker:<8-char-sha> (fallback :latest)

The dev OTLP receiver that feeds the TUI's otel-logs / otel-traces panes runs in-process inside the clrk dev host CLI (port 14318), not in the controller-manager.

A host-side kubeconfig is written to ~/.clrk/kubeconfig.host. Anything clrk does through --local you can also do with plain kubectl by exporting KUBECONFIG to that path. The CLI is a convenience layer, not a wrapper.

$terminalSH
export KUBECONFIG=~/.clrk/kubeconfig.host kubectl get pods -A

clrk dev bootstraps a default WorkerPool in the default namespace; its reconciler creates the default-workers Deployment. You can override its sizing by applying your own WorkerPool with the same name (see Core concepts).

The web console

The controller-manager serves an embedded web console - the same React UI that ships in production - on container port 8086. There's nothing extra to run: clrk dev auto-forwards it to a localhost port (no kubectl proxy, no separate Vite dev server). Find the URL in the EXPOSED SERVICES block of clrk dev status:

$terminalSH
clrk dev status # EXPOSED SERVICES # clrk/clrk-console http://localhost:18086

Open that URL in a browser. The console is same-origin: the controller-manager reverse-proxies the Kubernetes API (/api, /apis) to its in-process apiserver and serves the SPA from the same plain-HTTP port, so there's no CORS to configure and no self-signed-cert warning. It lists TaskAgents, DaemonAgents, and EgressGateways and streams invocation telemetry straight from the API.

In production clrk install creates the same clrk-console ClusterIP Service (container port 8086); pass --console=false to omit it. The console is unauthenticated in v1 (it proxies the unauthenticated apiserver), so reach it with kubectl port-forward svc/clrk-console 8086:8086 -n clrk. The installed NetworkPolicy deliberately does not open port 8086 - doing so would re-expose the apiserver to the pod network through the proxy - so if you front the console with an in-cluster ingress, add 8086 to your own NetworkPolicy and put authentication in front of it.

Starting it

The bare invocation needs no flags - it boots the stack and prints status:

$terminalSH
clrk dev

The everyday form passes a manifest tree and any environment-sourced Secrets:

$terminalSH
ANTHROPIC_API_KEY=sk-ant-... clrk dev \ --apply _examples/echo-bot/manifests \ --secret anthropic-credentials=ANTHROPIC_API_KEY:api-key

--apply is repeatable and accepts files or directories. -R/--recursive walks sub-directories. Apply happens after the apiserver and --secret materializations are ready, so a manifest referencing a Secret you just declared will resolve cleanly.

Surfacing Secrets from the shell

--secret is the dev-only shape for getting host environment variables into the cluster as Opaque Secrets:

$terminalTXT
--secret NAME=ENVVAR[:KEY]
  • NAME - the resulting Secret name in the default namespace.
  • ENVVAR - the host shell variable to read.
  • KEY - the data key inside the Secret. Defaults to ENVVAR lowercased with _ replaced by -. So ANTHROPIC_API_KEY becomes anthropic-api-key unless overridden.

Multiple --secret flags that share a NAME merge into one Secret with multiple keys. The same UX exists for non-dev use via clrk secret set, which accepts --from-env, --from-file, and --from-literal sources and works against any cluster, not just the dev one.

Heads up

--secret reads your shell environment. Don't paste a real production key into a terminal you can't trust - clrk dev itself is fine, but anything that scrapes shell history will see it.

TUI vs headless

When stdout is a TTY, clrk dev renders a multi-pane TUI: component status, container logs, the otel-logs and otel-traces panes that surface ext_proc records, and a manifest tree. The TUI is the fastest way to watch what an agent is doing.

When stdout isn't a TTY (CI, piped output), the TUI auto-disables and clrk dev falls back to streaming structured logs. You can force the headless form with --tui=false.

Iterating without restarting the stack

Three workflows you'll lean on a lot:

Re-apply a manifest

clrk apply -f is server-side-apply against whatever kubeconfig you point it at. With the dev stack running, the canonical form is:

$terminalSH
clrk apply -f path/to/manifests --local

--local resolves to ~/.clrk/kubeconfig.host. An explicit --kubeconfig wins over --local; otherwise standard kubeconfig resolution applies ($KUBECONFIG, then ~/.kube/config), honoring --context. Field manager is clrk-dev.

Hot-reload a freshly-built image

clrk dev push-image pushes a bazel-built OCI tarball into the dev session's local registry and, with --reload, rolls the matching Deployment and blocks until the new pod is running the pushed image:

$terminalSH
clrk dev push-image worker \ --tar bazel-bin/clrk/worker_oci_tarball/tarball.tar \ --reload

The component is worker or controller-manager. This requires the dev session to have a local registry, so launch clrk dev with --registry-image=COMPONENT=clrk-registry:5000/.... You can also trigger a rollout separately from another process with clrk dev reload <component> - much faster than a full clrk dev restart.

Hot-reload from source

--watch (experimental) rebuilds the controller and worker binaries on source changes and triggers the same hot-reload path. Use it when you're actively editing CLRK itself, not when you're authoring agent manifests.

Inspecting health

clrk dev status prints a tab-aligned table, one row per component, followed by an EXPOSED SERVICES block. STATUS is the docker status / Pod phase (e.g. running, Running), not the literal word healthy:

$terminalSH
clrk dev status # COMPONENT STATUS READY UPTIME IMAGE # k3d-clrk-dev-server-0 running yes ... rancher/k3s:v1.34.1-k3s1 # controller-manager Running yes ... ...clrk-controller-manager:... # worker-0 Running yes ... ...clrk-worker:... # # EXPOSED SERVICES # ...

--json emits the same per-component data keyed by component name, plus sibling registryPort and forwards keys. Because those are not component entries, CI consumers should select a specific component key (e.g. jq '.["controller-manager"].ready') rather than iterating every top-level key. --workers N matches the number of workers clrk dev is running so the report doesn't miss replicas.

clrk dev wait-ready blocks until the whole stack is healthy. Defaults to a 2-minute timeout with a 1-second probe interval. Use it in scripts that need to apply manifests right after clrk dev start.

$terminalSH
clrk dev wait-ready --timeout 90s clrk apply -f manifests/ --local

clrk dev logs -f streams every component in one combined stream - the k3d server's docker logs alongside the controller-manager and worker pod logs (streamed via the apiserver). Useful when the TUI is off, or when you want a single rolled tail across components.

Targeting a cluster

CLRK keeps no separate context store - there is no clrk config command. It targets the standard kubeconfig like kubectl, and the only clrk-owned kubeconfig on disk is the dev session's ~/.clrk/kubeconfig.host (reached with --local). Targeting is via global persistent flags:

  • --kubeconfig <file> - an explicit kubeconfig (highest precedence).
  • --local - the dev session's ~/.clrk/kubeconfig.host.
  • --context <name> - pick a context in the resolved kubeconfig.
  • Otherwise standard resolution: $KUBECONFIG, then ~/.kube/config.

clrk apply -f ... --local, clrk agents list --local, and clrk pools list --local always target the dev stack regardless of $KUBECONFIG or --context - that's the point of the flag. (clrk agents and clrk pools are command groups; the runnable forms are clrk agents list/get/... and clrk pools list/get.)

Tear down

Ctrl-C the foreground clrk dev and every container exits. State persists; the next clrk dev reattaches to it. For a fully clean slate:

$terminalSH
rm -rf ~/.clrk

Removing ~/.clrk deletes the apiserver state, all applied manifests, and the cached kubeconfig (~/.clrk/kubeconfig.host). The next start re-creates them from scratch.

When clrk dev is not the answer

clrk dev is for the laptop loop. The same controller-manager and worker images run in production, but you deploy them yourself into a real Kubernetes cluster and apply CLRK manifests against that cluster's apiserver. The CRDs, CLI subcommands, and agent contract are identical - only the host environment changes.

See the Reference for the full CRD and API surface, or the Guides for task-oriented walkthroughs.