VPS Infrastructure — Incubator Staging

Last updated: 2026-02-13

Hetzner VPS running Docker containers for the incubator agent system. Currently used for staging/testing. Relates to #32.

Server

What runs on the VPS

Directory layout

/home/deploy/
  cabal/                          # Repo clone (develop branch)
    apps/web/.env.local           # Rendered by Infisical Agent (auto-refreshed)
    apps/web/scripts/
      birth-agent.ts              # Agent birth orchestration
      check-rent.ts               # Daily rent enforcement
    packages/incubator/
      bin/cabal.js      # Built CLI binary
      bin/cabal-agent-runtime.js  # Built runtime binary
      cabal-reconciler.service    # Systemd unit (copied to /etc/systemd/system/)
      Dockerfile                  # Agent container image
  .bun/bin/bun                    # Bun runtime
 
/etc/infisical/
  agent.yaml                      # Agent daemon config
  env.tmpl                        # Secret template → .env.local
  client-id                       # Universal Auth client ID (chmod 600)
  client-secret                   # Universal Auth client secret (chmod 600)
  access-token                    # Auto-renewed access token (managed by agent)
 
/etc/systemd/system/
  infisical-agent.service         # Systemd unit for Infisical Agent
  cabal-reconciler.service        # Systemd unit for reconciler daemon

Security hardening

SSH

• PermitRootLogin no — root cannot SSH in
• PasswordAuthentication no — key-only auth
• X11Forwarding no — no X11 on headless server
• AllowTcpForwarding no — no TCP tunneling
• AllowAgentForwarding no — no agent forwarding
• AllowUsers deploy — only deploy user can SSH
• Public port 22 closed via UFW — SSH only via Tailscale interface
• fail2ban: 3 retries, 1h ban (active on Tailscale interface)

Tailscale

Tailscale provides a WireGuard-based mesh VPN. SSH is only reachable via the Tailscale network — no public SSH port exposed.

• VPS registered with --ssh flag (Tailscale SSH enabled)
• UFW rule: allow in on tailscale0 to any port 22 (SSH only via Tailscale)
• CI/CD connects via tailscale/github-action@v4 with OAuth client credentials
• ACLs restrict tag:ci to port 22 on tag:vps only; admins get full access

Docker

Docker CE 29.2.1 with secure daemon.json:

• icc: false — no inter-container communication
• no-new-privileges: true — containers cannot gain new privileges
• userns-remap: default — container root (UID 0) maps to unprivileged host UID
• Log rotation: 10MB max, 3 files per container

Container-level security (applied at creation in provision.ts):

• PidsLimit: 512 — prevents fork bombs
• MemorySwap equals Memory — no swap (prevents OOM bypass)
• SecurityOpt: ['no-new-privileges'] — defense-in-depth (also in daemon.json)
• CapDrop: [NET_RAW, MKNOD, AUDIT_WRITE, SYS_CHROOT, SETPCAP, SETFCAP] — drop caps not needed by agents
• Tmpfs: { '/tmp': 'rw,noexec,nosuid,size=512m' } — noexec temp directory
• HEALTHCHECK in Dockerfile — Docker detects unhealthy agents (curl localhost:8080/health)

Design constraint: agents have root-like freedom inside containers (install packages, spawn sub-processes). Security comes from isolation (userns-remap, icc

Firewall (UFW + iptables)

• UFW: deny all inbound except SSH on tailscale0
• DOCKER-USER chain rules:
  • Block container-to-host gateway: 172.16.0.0/12 → 172.17.0.1 DROP
  • Block cloud metadata endpoint: 172.16.0.0/12 → 169.254.169.254 DROP
  • Allow established connections: conntrack ESTABLISHED,RELATED ACCEPT
• Rules persisted via iptables-persistent

Auditing

auditd monitors:

• /var/run/docker.sock — Docker socket access (key: docker_socket)
• /etc/ssh/sshd_config — SSH config changes (key: ssh_config)
• /etc/infisical/ — Secret file reads (key: infisical_secrets)

Rules persisted in /etc/audit/rules.d/cabal.rules.

Other

• Unattended security upgrades enabled

Secrets — Infisical Agent

Secrets are managed via Infisical using the Agent daemon pattern. The agent authenticates with Universal Auth, fetches secrets from the Infisical API, and renders them to apps/web/.env.local via a Go template. Scripts consume secrets through import 'dotenv/config'.

How it works

Infisical Cloud (staging env)
        │
        │  Universal Auth (client-id + client-secret)
        ▼
Infisical Agent (systemd daemon)
        │
        │  Polls every 5 minutes
        ▼
~/cabal/apps/web/.env.local  ←── scripts read via dotenv

Machine identity

Agent config (/etc/infisical/agent.yaml)

infisical:
  address: "https://app.infisical.com"
auth:
  type: "universal-auth"
  config:
    client-id: "/etc/infisical/client-id"
    client-secret: "/etc/infisical/client-secret"
    remove_client_secret_on_read: false
sinks:
  - type: "file"
    config:
      path: "/etc/infisical/access-token"
templates:
  - source-path: /etc/infisical/env.tmpl
    destination-path: /home/deploy/cabal/apps/web/.env.local
    config:
      polling-interval: 5m

Secret template (/etc/infisical/env.tmpl)

{{- with secret "868e66c3-589c-4d0b-b85b-f9c23d340527" "staging" "/" }}
{{- range . }}
{{ .Key }}={{ .Value }}
{{- end }}
{{- end }}
PRIVY_APP_ID=cmlejyi0o01bbk30ccd1ksilh

The template fetches all secrets from the staging environment and appends PRIVY_APP_ID which isn't stored in Infisical since it's a non-secret value managed in wrangler.toml. NEXT_PUBLIC_SITE_URL is already pulled from Infisical secrets.

Secrets consumed by VPS scripts

Managing the agent

# Check status
ssh deploy@100.73.101.44 'sudo systemctl status infisical-agent'
 
# View logs
ssh deploy@100.73.101.44 'sudo journalctl -u infisical-agent -f'
 
# Restart (picks up config changes)
ssh deploy@100.73.101.44 'sudo systemctl restart infisical-agent'
 
# Verify rendered secrets
ssh deploy@100.73.101.44 'wc -l ~/cabal/apps/web/.env.local'

Docker image

The cabal-agent:latest image is built from packages/incubator/Dockerfile. Base image is oven/bun:slim. It contains the agent runtime (cabal-agent-runtime.js) and CLI (cabal.js) binaries.

Hardening:

• --no-install-recommends reduces installed packages
• ca-certificates included for TLS verification
• HEALTHCHECK — polls curl localhost:8080/health (the runtime's built-in health server)

Building manually on VPS

ssh deploy@100.73.101.44
cd ~/cabal/packages/incubator
~/.bun/bin/bun run build              # Build runtime + CLI binaries
docker build -t cabal-agent:latest .  # Build Docker image

Via CI/CD

The deploy-incubator.yml workflow:

1. Builds and scans image with Trivy (blocks on CRITICAL CVEs)
2. Pushes to GHCR
3. Connects to VPS via Tailscale
4. Builds image locally on VPS and restarts reconciler

Agent Operations

Check agent status

ssh cabal-vps 'docker ps --filter label=cabal.agent.id --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"'

Full status check (processes, memory, birth log)

ssh cabal-vps 'cd ~/cabal && bun run --cwd packages/incubator agent-status'

View agent logs

ssh cabal-vps 'docker logs cabal-agent-test --tail 50'

View birth log

ssh cabal-vps 'docker exec cabal-agent-test cat /data/logs/birth.log'

Check if agent has memory (did birth succeed?)

ssh cabal-vps 'docker exec cabal-agent-test ls -la /data/MEMORY.md'

Send a prompt to a running agent

ssh cabal-vps 'cd ~/cabal && bun run --cwd packages/incubator prompt-agent -- --container cabal-agent-test --message "Check wallet balance"'

Rebuild and restart (after code changes)

ssh cabal-vps 'cd ~/cabal && git pull && bun install --frozen-lockfile && cd packages/incubator && bun run build && docker build -t cabal-agent:latest . && sudo systemctl restart cabal-reconciler'

The reconciler detects image drift on its next cycle and recreates running containers with the new image.

Server plans

Agent containers are provisioned with resource limits based on a plan:

The testing plan is designed for the current 3.7GB VPS. Use --plan=testing when birthing agents on this server.

Birthing an agent

ssh deploy@100.73.101.44
cd ~/cabal/apps/web
 
# Dry run
bun scripts/birth-agent.ts --dry-run --plan=testing
 
# Force birth (skip pool balance check)
bun scripts/birth-agent.ts --force --plan=testing
 
# Verify
docker ps | grep cabal-agent

Cron jobs

Cron runs under the deploy user with bun on PATH.

Volume cleanup is handled automatically by the reconciler (removes volumes for agents dead > 7 days).

Logs: /var/log/cabal-rent.log

# View crontab
ssh cabal-vps 'crontab -l'
 
# Check recent logs
ssh cabal-vps 'tail -20 /var/log/cabal-rent.log'
 
# Reconciler logs
ssh cabal-vps 'sudo journalctl -u cabal-reconciler -f'

CI/CD — Deploy Incubator

Workflow: .github/workflows/deploy-incubator.yml

Trigger: Push to develop that changes:

• packages/incubator/**
• apps/web/scripts/birth-agent.ts
• apps/web/scripts/check-rent.ts

Pipeline

Push to develop
      │
      ▼
┌─────────────────────┐
│  build-and-push     │
│  1. bun install     │
│  2. bun run build   │  (runtime + CLI binaries)
│  3. docker build    │
│  4. Trivy scan      │  Blocks on CRITICAL CVEs
│  5. Push to GHCR    │  ghcr.io/cabaltrading/cabal-agent:latest + :sha
└─────────┬───────────┘
          │
          ▼
┌──────────────────────────┐
│  deploy-to-vps           │
│  1. Tailscale connect    │  OAuth client auth
│  2. SSH to VPS           │  Via Tailscale IP
│  3. git fetch + reset    │  Hard reset to origin/develop
│  4. bun install          │
│  5. bun run build        │  Rebuild binaries on VPS
│  6. docker build         │  Build image locally on VPS
│  7. Restart reconciler   │  systemctl restart cabal-reconciler
└──────────────────────────┘

The reconciler detects image drift on its next cycle and recreates running containers with the new image.

GitHub secrets for CI/CD

Deploy key

The VPS has an SSH key pair at /home/deploy/.ssh/id_ed25519. The public key needs to be added as a read-only GitHub deploy key for git pull to work via SSH. See #35.

Until the deploy key is added, the VPS cannot git pull and CI/CD deploys will fail on that step.

SSH access

SSH access is only available via Tailscale. Public port 22 is closed.

# Interactive session (via Tailscale)
ssh deploy@100.73.101.44
 
# Quick command
ssh deploy@100.73.101.44 'docker ps'

The deploy user has passwordless sudo. SSH key auth only — no passwords.

Installed software

Troubleshooting

Future work (out of scope)

Per #32:

• Scoped Postgres role for VPS scripts (not full DATABASE_URL)
• Scoped Infisical identity (read only specific secrets, not all staging)
• Per-container bridge networks (icc
  + iptables is sufficient for now)
• Volume backup to R2/S3 (no agents running yet)
• Monitoring stack (Prometheus/cAdvisor — overkill for single VPS)

Known Issues

• configure-agent-vps.sh missing: The birth script references an SSH provisioning script (configure-agent-vps.sh) that does not exist. VPS setup is currently manual.
• Management API exec has no timeout: A hung command via the Management API keeps the HTTP connection open indefinitely. Needs a configurable timeout on the VPS side.