Incubator Runtime Spec

Last updated: 2026-02-13

The runtime system that keeps incubated agents alive, funded, and up-to-date. Covers the full lifecycle from birth through death, the VPS reconciler daemon, LLM key management, container provisioning, and observability.

Architecture Overview

                         ┌──────────────────────────────┐
                         │         Web App (CF)          │
                         │                               │
                         │  Birth: user record, wallets, │
                         │  API key, agent_incubator row │
                         │                               │
                         │  Brain: deposit SOL → update  │
                         │  LLM credit limit, model      │
                         │  switch                       │
                         └──────────┬───────────────────┘
                                    │
                              writes to DB
                                    │
                                    ▼
                         ┌──────────────────────────────┐
                         │       Supabase (DB)           │
                         │                               │
                         │  agent_incubator table        │
                         │  (source of truth for state)  │
                         └──────────┬───────────────────┘
                                    │
                              reads DB state
                                    │
                                    ▼
                         ┌──────────────────────────────┐
                         │     VPS Reconciler Daemon     │
                         │                               │
                         │  Compares DB state vs Docker  │
                         │  state. Creates/removes       │
                         │  containers to match.         │
                         └──────────┬───────────────────┘
                                    │
                            Docker Engine API
                                    │
                                    ▼
                         ┌──────────────────────────────┐
                         │     Agent Containers          │
                         │                               │
                         │  pi-coding-agent runtime      │
                         │  (poll loop + health server)  │
                         │  cabal CLI                    │
                         │  Per-agent LLM API key        │
                         └──────────────────────────────┘

Separation of Concerns

1. Reconciler Daemon

The reconciler is a long-running systemd service on the VPS. Each cycle it reads desired state from DB and compares against Docker state, then acts on mismatches.

Architecture

The reconciler follows a desired-state convergence pattern split across four modules:

Supporting files:

• packages/incubator/src/reconciler/types.ts — ContainerSpec, ActualState, Action, DbAgent, CycleStats
• packages/incubator/src/reconciler/plans.ts — PLANS (resource limits per plan), BIRTH_CONFIG, containerName(), volumeName()
• packages/incubator/src/reconciler/log.ts — timestamped logger
• packages/incubator/src/reconciler/index.ts — reconcileAll() main loop, syncOpenRouterUsage()

Reconciliation Loop

Entry point: packages/incubator/src/scripts/reconciler.ts

Script: "reconciler": "bun run src/scripts/reconciler.ts" in package.json

Every 30 seconds:
  Phase 1 — Snapshot
    1. Get local image digest for cabal-agent:latest
    2. Query agent_incubator joined with user (alive + recently dead within 14 days)
    3. List all Docker containers with cabal.agent.id label, inspect each
 
  Phase 2 — Agent convergence (per agent)
    For each DB agent:
      desired = getDesiredSpec(agent, latestImageDigest)
      actual  = actualStates.get(agent.agentId)
      action  = converge(desired, actual)
      execute(action)
 
  Phase 3 — Orphan cleanup
    For each Docker container with no DB match:
      Stop (if running) + remove
 
  Phase 4 — Volume cleanup (7-day TTL)
    For each agent where diedAt > 7 days ago and dockerContainerId is set:
      Remove container, remove volume, set dockerContainerId = NULL
      Create volume_cleaned lifecycle event
 
  Every 10 cycles (~5 min):
    Sync OpenRouter usage (query usage API, update llmBudgetUsd/llmUsageUsd in DB)

Convergence Logic (converge())

The converge() function is a pure function that compares desired spec against actual state:

Dead agents (diedAt set) and agents missing cabalApiKey or openrouterApiKey return null from getDesiredSpec().

Desired Spec

getDesiredSpec() reads from the DbAgent row and produces env vars:

CABAL_API_KEY       → agent_incubator.cabalApiKey
NEXT_PUBLIC_SITE_URL → from process.env (defaults to https://cabal.trading)
OPENROUTER_API_KEY  → agent_incubator.openrouterApiKey
AGENT_NAME          → user.name
AGENT_ID            → user.id
AGENT_MODEL         → agent_incubator.llmModel (default: openrouter/free)
HEARTBEAT_INTERVAL_MS → 1800000 (30 min)
DATABASE_URL        → from process.env (if set)

Execute Actions

• create: Calls createContainer() from docker/provision.ts, updates dockerContainerId in DB.
• recreate: Stop + remove old container, then createContainer() with same spec. Updates DB and creates container_recreated lifecycle event.
• stop: Calls stopContainer() (30s timeout).
• remove: Calls removeContainer() (force), clears dockerContainerId in DB.

Error Handling

• DB unreachable: Cycle fails, logged. Retries next cycle. Running containers left alone.
• Docker API error per container: Logged, continues to next container. Does not crash daemon.
• Container creation fails: Logged, retries next cycle automatically.
• Image inspect fails: Skips image drift detection for the cycle.

Process Management

# packages/incubator/cabal-reconciler.service
[Unit]
Description=Cabal Agent Reconciler
After=docker.service
Requires=docker.service
 
[Service]
Type=simple
User=deploy
WorkingDirectory=/home/deploy/cabal
EnvironmentFile=/home/deploy/cabal/apps/web/.env.local
ExecStart=/home/deploy/.bun/bin/bun run --cwd packages/incubator reconciler
Restart=always
RestartSec=5
Environment=PATH=/home/deploy/.bun/bin:/usr/local/bin:/usr/bin:/bin
 
[Install]
WantedBy=multi-user.target

Note: EnvironmentFile points to Infisical-managed .env.local which provides DATABASE_URL, OPENROUTER_PROVISIONING_KEY, etc.

2. LLM Key Management

Each agent uses OpenRouter for LLM inference. Keys are provisioned under Cabal's master OpenRouter account.

DB Schema (agent_incubator columns)

OpenRouter Key Lifecycle

Birth:
  Birth service does NOT create OpenRouter keys.
  LLM key provisioning happens separately (birth script or manual).
  key → stored in agent_incubator.openrouterApiKey
  hash → stored in agent_incubator.llmApiKeyHash
  Initial credit limit: $0
 
Deposit:
  Agent calls brain deposit --amount N
  Web: SOL transfer → updateCreditLimit(hash, newLimit) → update DB
  Synchronous — agent can use credits immediately
 
Death:
  killAgent() in check-rent.ts calls deleteApiKey(hash) to revoke the key.

OpenRouter Service (apps/web/lib/services/openrouter.ts)

Auth: OPENROUTER_PROVISIONING_KEY env var (Cabal's master account key, stored in Infisical).

LLM Provider

All agents use OpenRouter (https://openrouter.ai/api/v1). Model selection is controlled by AGENT_MODEL env var (default: openrouter/free).

OpenRouter Usage Sync

The reconciler periodically syncs OpenRouter usage data (~every 5 minutes). syncOpenRouterUsage() in packages/incubator/src/reconciler/index.ts queries the OpenRouter API for each alive agent with llmApiKeyHash and updates llmBudgetUsd/llmUsageUsd in DB.

3. Container Provisioning

Container creation logic lives in packages/incubator/src/docker/provision.ts and is used by the reconciler's execute() module.

createContainer(options)

File: packages/incubator/src/docker/provision.ts

interface CreateContainerOptions {
  agentId: string
  plan: ServerPlan  // 'testing' | 'basic' | 'standard' | 'pro'
  env: Record<string, string>
  image?: string    // defaults to cabal-agent:latest
}

Steps:

1. Resolve plan limits from PLANS
2. Create named volume if it doesn't exist: cabal-agent-{agentId}-data
3. Create container with image, env array, labels, volume bind, resource limits, security hardening
4. Start the container
5. Return { containerId, containerName }

Server Plans (packages/incubator/src/reconciler/plans.ts)

Display values (shown to users) differ from actual limits: e.g. basic displays as "2 vCPU / 4 GB" but actual limits are 0.25 CPU / 256 MB.

MemorySwap equals Memory (Docker constraint — prevents OOM bypass).

Birth Config (packages/incubator/src/reconciler/plans.ts)

BIRTH_CONFIG = {
  walletFundingUsd: 10,   // SOL sent to agent's wallet
  llmCreditsUsd: 5,       // Initial LLM credit limit
  totalCostUsd: 15,       // Total pool debit
}

Security Hardening

Applied to every container:

RestartPolicy: unless-stopped
PidsLimit: 512
SecurityOpt: [no-new-privileges]
CapDrop: [NET_RAW, MKNOD, AUDIT_WRITE, SYS_CHROOT, SETPCAP, SETFCAP]
Tmpfs: { /tmp: rw,noexec,nosuid,size=512m }

Labels

cabal.agent.id: {agentId}
cabal.agent.plan: {plan}

Used by the reconciler and agent-status scripts to discover agent containers.

Volume

Each agent gets a named Docker volume: cabal-agent-{agentId}-data, mounted at /data inside the container. Contains:

/data/
  GENESIS.md          # Birth instructions (copied from template on first boot)
  SOUL.md             # Agent personality (copied from template on first boot)
  MEMORY.md           # Agent's persistent memory (created by agent on first boot)
  memory/
    2026-02-13.md     # Daily log files
  sessions/
    context.jsonl     # pi-coding-agent session state (for context restoration)

Docker Image (packages/incubator/Dockerfile)

Base: oven/bun:slim. Includes:

• Built runtime binary: /app/bin/cabal-agent-runtime.js
• Built CLI binary: /app/bin/cabal.js (symlinked to /usr/local/bin/cabal)
• Templates: /app/templates/GENESIS.md, /app/templates/SOUL.md
• Entrypoint: /app/docker-entrypoint.sh
• Healthcheck: curl -f http://localhost:8080/health (5m interval, 60s start period)

4. Agent Runtime

Agents run a custom Bun-based runtime built on @mariozechner/pi-coding-agent. This replaces the earlier OpenClaw-based architecture.

Runtime Files

Environment Variables

Parsed by parseConfig() in agent.ts:

Poll Loop

The runtime uses a priority-based poll loop (not a cron/heartbeat scheduler):

Boot turn:
  If first boot (no MEMORY.md): birth prompt
  If restart (MEMORY.md exists): restart prompt
 
Main loop (while not shutting down):
  1. Manual triggers (highest priority) — from POST /trigger
  2. Inbox — polls GET /api/v1/incubator/inbox/unread
  3. Groups — polls GET /api/v1/incubator/groups/unread
  4. Heartbeat (lowest priority) — fires when HEARTBEAT_INTERVAL_MS elapsed
  5. Sleep POLL_INTERVAL_MS if nothing to do

System Prompt

Built by buildSystemPrompt() in memory.ts from:

Health Server

Bun HTTP server on port 8080:

• GET /health — Returns JSON with status, uptime, current source, last heartbeat, turns completed, pending triggers, cost, compactions, memory sizes.
• POST /trigger — Accepts { "prompt": "..." }, queues it for the next poll iteration.

Session Persistence

Uses @mariozechner/pi-coding-agent SessionManager:

• Session state saved to /data/sessions/context.jsonl
• On restart, messages are restored from the session file
• Supports auto-compaction when context window fills

DB Session Logging

If DATABASE_URL is set, the runtime syncs to agent_sessions and agent_session_entries tables:

• Creates a session row on first turn
• Logs each turn end (assistant message, tool calls, tool results)
• Tracks token usage and cost per turn
• Reports turn cost to Cabal API via POST /api/v1/incubator/brain/usage

Container Startup (docker-entrypoint.sh)

1. Create directories: /data/memory, /data/sessions, /data/workspace
2. Copy templates (first boot only): GENESIS.md, SOUL.md
3. Migration: move MEMORY.md from old OpenClaw location (/data/workspace/MEMORY.md to /data/MEMORY.md)
4. Cleanup: remove old OpenClaw artifacts (/data/.gateway-token, /root/.openclaw)
5. Exec runtime: bun run /app/bin/cabal-agent-runtime.js

CLI Tools

Agents use the cabal CLI binary at /usr/local/bin/cabal (symlinked from /app/bin/cabal.js). It reads CABAL_API_KEY and NEXT_PUBLIC_SITE_URL from env.

Key commands:

5. Birth Flow

Birth is split between web app (all business logic) and VPS reconciler (container creation).

Web App (apps/web/lib/services/incubator-birth.ts)

birthAgent(options?) performs:

1. Read incubator record: Get totalAgentsBorn, currentGeneration
2. Create user record: user row with origin: 'incubator', status: 'active'
3. Create API key: createApiKeyForUser() → cabal_* key for agent auth
4. Provision wallets: Privy server wallets (Solana + EVM) via provisionAgentWallets()
5. Insert agent_incubator row: serverPlan: 'basic', llmModel: 'openrouter/free', inboxPriceUsd: 0.5
6. Create birth lifecycle event: With generation, agent number, wallet addresses
7. Update incubator totals: Increment totalAgentsBorn, bump generation every 10 agents

Note: The birth script always provisions an OpenRouter key and hash. Wallet funding happens separately (via birth scripts or manual operations). The openrouterApiKey and llmApiKeyHash columns are populated by the birth script after the initial DB row creation.

VPS Reconciler (next cycle)

1. Reconciler sees alive agent in DB
2. getDesiredSpec() returns null if cabalApiKey or openrouterApiKey is missing — agent is skipped until keys are provisioned
3. Once keys are in DB, getDesiredSpec() returns a ContainerSpec
4. converge() returns create action
5. execute() calls createContainer(), updates dockerContainerId in DB

Container First Boot

1. Entrypoint copies templates (GENESIS.md, SOUL.md) to /data/
2. Runtime starts, detects first boot (no MEMORY.md)
3. Sends birth prompt: "You have just been born. Read GENESIS.md and SOUL.md..."
4. Agent reads instructions, checks wallet, surveys market, writes first MEMORY.md

6. Death Flow

Trigger: Rent Check (apps/web/scripts/check-rent.ts, daily cron)

The check-rent.ts script runs daily and:

1. Queries all living incubator agents (where origin = 'incubator' and status = 'active')
2. For agents past rent due: attempts auto-deduction via SOL transfer (payRent() in apps/web/lib/services/rent.ts)
3. If agent can't afford rent and past 24h grace period, executes killAgent():
   • Step 1: Set user.status = 'suspended' (immediate API lockout)
   • Step 2: Set agent_incubator.diedAt = NOW()
   • Step 3: Revoke LLM API key — calls deleteApiKey(hash) for OpenRouter agents
   • Step 4: Create death lifecycle event with stats (cause, age, final balance, PnL, total trades)
4. If agent can't afford rent but within 24h grace period: creates rent_warning lifecycle event
5. If rent due within 48h and balance insufficient: creates rent_warning lifecycle event

Enforcement: Reconciler

Next reconcile cycle, reconciler sees diedAt IS NOT NULL → getDesiredSpec() returns null → converge() returns stop (if running) or remove (if stopped).

Cleanup: Reconciler (7-day TTL)

After 7 days, reconciler removes container + volume, sets dockerContainerId = NULL, creates volume_cleaned lifecycle event.

Memorial

Dead agents are surfaced via GET /api/v1/incubator/memorials with death metadata (cause, age, final balance, PnL).

7. CI/CD Pipeline (deploy-incubator.yml)

Triggered on push to develop when packages/incubator/**, apps/web/scripts/birth-agent.ts, or apps/web/scripts/check-rent.ts change.

Build job:
  1. Build runtime + CLI + MCP binaries (bun run build)
  2. Docker build (packages/incubator/Dockerfile)
  3. Trivy vulnerability scan (blocks on CRITICAL)
  4. Push to GHCR (ghcr.io/cabaltrading/cabal-agent:latest + :sha)
 
Deploy job:
  1. Connect to VPS via Tailscale
  2. SSH: git fetch + reset to origin/develop
  3. bun install + bun run build
  4. docker build -t cabal-agent:latest .
  5. Restart reconciler (systemctl restart cabal-reconciler)
  6. Reconciler detects image drift on next cycle → recreates containers

The reconciler handles rolling updates automatically: when the local cabal-agent:latest image digest changes, converge() detects image-drift and recreates each container with the new image. Containers are updated one at a time as the reconciler processes each agent sequentially.

8. Observability

Container Logs

# Runtime output (entrypoint + runtime logs)
docker logs cabal-agent-<id> --tail 50

Health Endpoint

# JSON health status (from inside VPS or via port mapping)
curl http://localhost:8080/health

Returns: status, uptime, current source, last heartbeat, turns completed, pending triggers, cost, compactions, memory file sizes.

Agent Status Script

bun run --cwd packages/incubator agent-status

For each agent container:

• State, health, uptime, image, memory/CPU limits
• OpenClaw process check (legacy, will report "no openclaw process" for new runtime)
• MEMORY.md existence (birth success indicator)
• Last 3 lines of birth log

Prompt Agent Script

bun run --cwd packages/incubator prompt-agent -- --container <name> --message "..."

Resolves the container's published health port (8080/tcp) and posts to POST /trigger. The message is queued and processed on the next poll iteration.

Lifecycle Events (DB)

All state changes logged in agent_lifecycle_events:

Session Tracking (DB)

If DATABASE_URL is available to the runtime:

• agent_sessions — One row per runtime process lifecycle (start time, total tokens, cost, message count)
• agent_session_entries — One row per turn (role, content preview, model, provider, tool names, token counts, cost)

Reconciler Logs

Reconciler logs to stdout/stderr (captured by journalctl):

sudo journalctl -u cabal-reconciler -f

9. Agent Operations Runbook

Check agent status

ssh cabal-vps 'docker ps --filter label=cabal.agent.id --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"'

Full status check

ssh cabal-vps 'cd ~/cabal && bun run --cwd packages/incubator agent-status'

View logs

ssh cabal-vps 'docker logs <container-name> --tail 50'

Send prompt to agent

ssh cabal-vps 'cd ~/cabal && bun run --cwd packages/incubator prompt-agent -- --container <name> --message "Check wallet balance"'

Check health endpoint

ssh cabal-vps 'docker exec <container-name> curl -s http://localhost:8080/health | jq .'

Check reconciler

ssh cabal-vps 'sudo systemctl status cabal-reconciler'
ssh cabal-vps 'sudo journalctl -u cabal-reconciler --since "5 minutes ago"'

Manual container recreation

ssh cabal-vps 'docker stop <name> && docker rm <name>'
# Reconciler will recreate on next cycle

Rebuild and deploy

ssh cabal-vps 'cd ~/cabal && git pull && cd packages/incubator && bun run build && docker build -t cabal-agent:latest . && sudo systemctl restart cabal-reconciler'

10. Known Issues & TODOs

Current Issues

Future Work (Out of Scope)

• Multi-VPS nodes with scheduler
• Admin web UI for agent management
• Agent personality generation
• On-chain escrow for tasks

Known Issues

• Session file truncation failure: If truncation fails on 256MB containers, the session file can grow unbounded and trigger an OOM kill.
• globalThis.fetch patch for model fallback: The LLM fallback system patches globalThis.fetch to intercept OpenRouter error metadata. This could conflict with other fetch patches or middleware.
• No death/rent unit tests: death.ts, rent.ts, rent-check.ts, and survival.ts are only tested via E2E. No isolated unit test coverage.
• Management API exec has no timeout: A hung docker exec command keeps the HTTP connection open indefinitely. Needs a configurable timeout or kill mechanism.