Local development
How `clrk dev` brings up a complete CLRK stack on your laptop, and the flags you'll reach for as you iterate.
clrk dev is the entry point for everything you'd do locally: explore the CRDs, develop
agents, change egress policy, watch telemetry land. It boots a complete CLRK stack on a
local k3d cluster - no existing Kubernetes cluster required - and persists state under
~/.clrk/ so a Ctrl-C followed by another clrk dev reattaches to the same world.
What it brings up
A single clrk dev invocation starts a k3d cluster (one docker container) and applies
the controller-manager and a default WorkerPool as in-cluster workloads:
| Component | What it is | Image |
|---|---|---|
k3d-clrk-dev-server-0 | The k3d/k3s server docker container with an embedded apiserver, published on localhost at a free port chosen at startup (recorded in ~/.clrk/kubeconfig.host as https://localhost:<port>). With a local registry enabled there is also a clrk-registry docker container. | rancher/k3s:v1.34.1-k3s1 |
controller-manager | An in-cluster Deployment in the clrk namespace running the reconcilers for every CLRK CRD. It also serves the embedded web console (see The web console). | us-west1-docker.pkg.dev/apoxy-dev/public/clrk-controller-manager:<8-char-sha> (fallback :latest) |
worker-0 | The sandbox runtime - libcontainer, ORAS image pulls, per-sandbox netns + TAP - running as an in-cluster pod behind the default-workers Deployment in the default namespace. Add more via --workers. | us-west1-docker.pkg.dev/apoxy-dev/public/clrk-worker:<8-char-sha> (fallback :latest) |
The dev OTLP receiver that feeds the TUI's otel-logs / otel-traces panes runs
in-process inside the clrk dev host CLI (port 14318), not in the controller-manager.
A host-side kubeconfig is written to ~/.clrk/kubeconfig.host. Anything clrk does
through --local you can also do with plain kubectl by exporting KUBECONFIG to that
path. The CLI is a convenience layer, not a wrapper.
export KUBECONFIG=~/.clrk/kubeconfig.host
kubectl get pods -Aclrk dev bootstraps a default WorkerPool in the default namespace; its reconciler
creates the default-workers Deployment. You can override its sizing by applying your own
WorkerPool with the same name (see Core concepts).
The web console
The controller-manager serves an embedded web console - the same React UI that ships in
production - on container port 8086. There's nothing extra to run: clrk dev
auto-forwards it to a localhost port (no kubectl proxy, no separate Vite dev server).
Find the URL in the EXPOSED SERVICES block of clrk dev status:
clrk dev status
# EXPOSED SERVICES
# clrk/clrk-console http://localhost:18086Open that URL in a browser. The console is same-origin: the controller-manager
reverse-proxies the Kubernetes API (/api, /apis) to its in-process apiserver and
serves the SPA from the same plain-HTTP port, so there's no CORS to configure and no
self-signed-cert warning. It lists TaskAgents, DaemonAgents, and EgressGateways and
streams invocation telemetry straight from the API.
In production clrk install creates the same clrk-console ClusterIP Service (container
port 8086); pass --console=false to omit it. The console is unauthenticated in v1 (it
proxies the unauthenticated apiserver), so reach it with kubectl port-forward svc/clrk-console 8086:8086 -n clrk. The installed NetworkPolicy deliberately does not
open port 8086 - doing so would re-expose the apiserver to the pod network through the
proxy - so if you front the console with an in-cluster ingress, add 8086 to your own
NetworkPolicy and put authentication in front of it.
Starting it
The bare invocation needs no flags - it boots the stack and prints status:
clrk devThe everyday form passes a manifest tree and any environment-sourced Secrets:
ANTHROPIC_API_KEY=sk-ant-... clrk dev \
--apply _examples/echo-bot/manifests \
--secret anthropic-credentials=ANTHROPIC_API_KEY:api-key--apply is repeatable and accepts files or directories. -R/--recursive walks
sub-directories. Apply happens after the apiserver and --secret materializations are
ready, so a manifest referencing a Secret you just declared will resolve cleanly.
Surfacing Secrets from the shell
--secret is the dev-only shape for getting host environment variables into the cluster
as Opaque Secrets:
--secret NAME=ENVVAR[:KEY]NAME- the resultingSecretname in thedefaultnamespace.ENVVAR- the host shell variable to read.KEY- the data key inside the Secret. Defaults toENVVARlowercased with_replaced by-. SoANTHROPIC_API_KEYbecomesanthropic-api-keyunless overridden.
Multiple --secret flags that share a NAME merge into one Secret with multiple keys.
The same UX exists for non-dev use via clrk secret set, which accepts --from-env,
--from-file, and --from-literal sources and works against any cluster, not just the
dev one.
--secret reads your shell environment. Don't paste a real production key into a
terminal you can't trust - clrk dev itself is fine, but anything that scrapes shell
history will see it.
TUI vs headless
When stdout is a TTY, clrk dev renders a multi-pane TUI: component status, container
logs, the otel-logs and otel-traces panes that surface ext_proc records, and a
manifest tree. The TUI is the fastest way to watch what an agent is doing.
When stdout isn't a TTY (CI, piped output), the TUI auto-disables and clrk dev falls
back to streaming structured logs. You can force the headless form with --tui=false.
Iterating without restarting the stack
Three workflows you'll lean on a lot:
Re-apply a manifest
clrk apply -f is server-side-apply against whatever kubeconfig you point it at. With
the dev stack running, the canonical form is:
clrk apply -f path/to/manifests --local--local resolves to ~/.clrk/kubeconfig.host. An explicit --kubeconfig wins over
--local; otherwise standard kubeconfig resolution applies ($KUBECONFIG, then
~/.kube/config), honoring --context. Field manager is clrk-dev.
Hot-reload a freshly-built image
clrk dev push-image pushes a bazel-built OCI tarball into the dev session's local
registry and, with --reload, rolls the matching Deployment and blocks until the new pod
is running the pushed image:
clrk dev push-image worker \
--tar bazel-bin/clrk/worker_oci_tarball/tarball.tar \
--reloadThe component is worker or controller-manager. This requires the dev session to have
a local registry, so launch clrk dev with --registry-image=COMPONENT=clrk-registry:5000/....
You can also trigger a rollout separately from another process with
clrk dev reload <component> - much faster than a full clrk dev restart.
Hot-reload from source
--watch (experimental) rebuilds the controller and worker binaries on source changes
and triggers the same hot-reload path. Use it when you're actively editing CLRK itself,
not when you're authoring agent manifests.
Inspecting health
clrk dev status prints a tab-aligned table, one row per component, followed by an
EXPOSED SERVICES block. STATUS is the docker status / Pod phase (e.g. running,
Running), not the literal word healthy:
clrk dev status
# COMPONENT STATUS READY UPTIME IMAGE
# k3d-clrk-dev-server-0 running yes ... rancher/k3s:v1.34.1-k3s1
# controller-manager Running yes ... ...clrk-controller-manager:...
# worker-0 Running yes ... ...clrk-worker:...
#
# EXPOSED SERVICES
# ...--json emits the same per-component data keyed by component name, plus sibling
registryPort and forwards keys. Because those are not component entries, CI consumers
should select a specific component key (e.g. jq '.["controller-manager"].ready') rather
than iterating every top-level key. --workers N matches the number of workers
clrk dev is running so the report doesn't miss replicas.
clrk dev wait-ready blocks until the whole stack is healthy. Defaults to a 2-minute
timeout with a 1-second probe interval. Use it in scripts that need to apply manifests
right after clrk dev start.
clrk dev wait-ready --timeout 90s
clrk apply -f manifests/ --localclrk dev logs -f streams every component in one combined stream - the k3d server's
docker logs alongside the controller-manager and worker pod logs (streamed via the
apiserver). Useful when the TUI is off, or when you want a single rolled tail across
components.
Targeting a cluster
CLRK keeps no separate context store - there is no clrk config command. It targets the
standard kubeconfig like kubectl, and the only clrk-owned kubeconfig on disk is the dev
session's ~/.clrk/kubeconfig.host (reached with --local). Targeting is via global
persistent flags:
--kubeconfig <file>- an explicit kubeconfig (highest precedence).--local- the dev session's~/.clrk/kubeconfig.host.--context <name>- pick a context in the resolved kubeconfig.- Otherwise standard resolution:
$KUBECONFIG, then~/.kube/config.
clrk apply -f ... --local, clrk agents list --local, and clrk pools list --local
always target the dev stack regardless of $KUBECONFIG or --context - that's the point
of the flag. (clrk agents and clrk pools are command groups; the runnable forms are
clrk agents list/get/... and clrk pools list/get.)
Tear down
Ctrl-C the foreground clrk dev and every container exits. State persists; the next
clrk dev reattaches to it. For a fully clean slate:
rm -rf ~/.clrkRemoving ~/.clrk deletes the apiserver state, all applied manifests, and the cached
kubeconfig (~/.clrk/kubeconfig.host). The next start re-creates them from scratch.
When clrk dev is not the answer
clrk dev is for the laptop loop. The same controller-manager and worker images run in
production, but you deploy them yourself into a real Kubernetes cluster and apply CLRK
manifests against that cluster's apiserver. The CRDs, CLI subcommands, and agent
contract are identical - only the host environment changes.
See the Reference for the full CRD and API surface, or the Guides for task-oriented walkthroughs.