Agent Runtime: Model Selection & Fallback State Machine

Overview

Each agent runs inside a Docker container on the VPS. The reconciler (host-side) decides what to run; the runtime (inside the container) decides how to run, including model selection and fallback.

1. Reconciler (Host-Side)

Source: packages/incubator/src/reconciler/

The reconciler is a desired-state convergence loop. It doesn't pick models — it passes DB config into container env vars.

Per-Cycle Flow

1. Fetch DB agents — all origin='incubator' users joined with agent_incubator, including llmModel, openrouterApiKey, llmApiKeyHash, serverPlan.
2. Snapshot local Docker image digest — used for image drift detection.
3. For each agent, compute desired spec (desired-spec.ts):
   • Dead (diedAt set) → desired = null (tear down).
   • Otherwise, build ContainerSpec with env vars:
     • OPENROUTER_API_KEY = agent_incubator.openrouterApiKey
     • AGENT_MODEL = agent_incubator.llmModel (NOT NULL, default 'openrouter/free')
4. Converge (converge.ts) — compare desired vs actual Docker state:
   • No desired, no actual → noop
   • No desired, has actual → stop/remove
   • Has desired, no actual → create
   • Both exist → check for drift (image, plan, env, health)
5. Execute actions (create, recreate, stop, remove).
6. Orphan cleanup — containers with no DB match get removed.
7. Volume cleanup — dead agents get volumes cleaned after 7 days.
8. OR usage sync — polls OpenRouter key stats, writes llmBudgetUsd/llmUsageUsd back to DB.

Key Point

The reconciler recreates the container on env drift. If you change AGENT_MODEL in the DB, the next reconciler cycle detects env drift and recreates the container with the new values. The runtime then boots fresh with the new model config.

2. Runtime Boot (Inside Container)

Source: packages/incubator/src/runtime/agent.ts

Startup Sequence

main()
  → parseConfig()          # Read env vars into RuntimeConfig
  → startHealthServer()    # :8080 — GET /health, POST /trigger
  → initSession()          # Model resolution + agent setup
  → mainLoop()             # Poll loop

parseConfig()

Reads env vars set by the reconciler (OpenRouter-only since 8.5 cutover):

Model Resolution in initSession()

1. Call resolveOpenRouterModels() — fetches https://openrouter.ai/api/v1/models, filters to free + tool-capable models, sorts by context window descending. Returns a full ranked list of Model[] (not just 2).
2. If AGENT_MODEL is a specific model (not 'openrouter/free') → prepend it as rank #0 (pinned, always preferred when healthy). Dedup if it already appears in the ranked list.
3. If AGENT_MODEL is 'openrouter/free' (generic auto) → use the ranked list as-is; the dynamically resolved best free model is rank #0.
4. If the fetch fails or returns 0 candidates → fall back to a single resolveModel() entry.

Result: A ModelSelector is created with the full ranked list. No special "primary" vs "fallback" distinction — rank #0 is simply the most preferred model.

3. Poll Loop & Turn Execution

Main Loop Priority

while (!shuttingDown):
  selector.tick()   # re-enable models whose restore timers expired
  if coolingDown → sleep(min(remaining, 60s)), continue
  1. Manual trigger (POST /trigger)  → highest priority
  2. Inbox (paid messages)           → /incubator/inbox/unread
  3. Groups (social)                 → /incubator/groups/unread
  4. Heartbeat                       → every heartbeatIntervalMs (default 30min)
  5. Nothing → sleep(pollIntervalMs)

Each turn calls safeTurn() → runTurn() with fresh context (messages cleared each turn).

4. Model Selector (Ranked Model List)

Source: packages/incubator/src/runtime/fallback.ts — ModelSelector class

Per-Model Health State

Each model in the ranked list tracks:

{ model, healthy, cooldownUntil, restoreAt, lastStatus, restoreFailures }

Selection Logic

currentModel():
  return first model in ranked list where:
    healthy = true AND cooldownUntil < now
  if none → return null (all exhausted, global cooldown)

Fetch Interceptor

The ModelSelector installs a globalThis.fetch interceptor at session init to capture HTTP metadata from OpenRouter responses. This is necessary because the pi-coding-agent SDK doesn't expose HTTP status codes — errors surface only as thrown exceptions or stopReason: 'error' messages.

installFetchInterceptor():
  patches globalThis.fetch
  for each fetch to openrouter.ai:
    reset _lastHttpMeta = null, _lastFetchCompleted = false
    if fetch throws → network failure, _lastFetchCompleted stays false
    if response.ok → _lastFetchCompleted = true (no meta captured)
    if !response.ok → capture HttpErrorMeta:
      { status, retryAfterMs, code, type, message }
      retryAfterMs parsed from headers (see below)
      code/type/message parsed from JSON error body
      if no retry info in HTTP headers → check embedded metadata headers
        (body.error.metadata.headers — OpenRouter embeds upstream provider headers)
  returns cleanup function that restores original fetch

1. retry-after-ms — direct milliseconds (highest priority)
2. retry-after — seconds (integer) or HTTP-date
3. x-ratelimit-reset — epoch ms timestamp (OpenRouter-specific)

Error Handling Flow

onError(modelId, error):
  read _lastHttpMeta captured by fetch interceptor
 
  if no meta:
    if fetch completed (200 + SDK error) → permanent (healthy=false, restoreAt=0)
    if fetch did NOT complete (network)  → short cooldown (60s)
 
  if meta present, classify by HTTP status:
    401         → permanent (bad auth)
    402         → permanent, unless Retry-After → restore timer
    403 + "key limit" in message → transient cooldown (per-key RPM cap)
    403 + anything else          → permanent, unless Retry-After
    429         → transient cooldown (uses Retry-After or 60s)
    5xx         → transient cooldown (uses Retry-After or 60s)
    other 4xx   → permanent
 
  caller immediately calls currentModel() to get next-best model

Restore via tick()

tick() — called each main loop iteration:
  for each unhealthy model where restoreAt > 0 and now >= restoreAt:
    set healthy=true
    (model will be tried naturally when it's the highest-ranked healthy option)

No explicit "restore attempt" — when a model's timer expires, tick() marks it healthy. If it errors again, onError() re-disables it.

Error Classification

Error Detection Points

Errors are caught in two ways during runTurn():

1. Thrown exceptions — session.prompt() throws (e.g. network error, 429). Fetch interceptor has already captured HttpErrorMeta if the error came from an HTTP response.
2. Post-prompt error — SDK doesn't throw on LLM API errors (402, 403). Instead, the last assistant message has stopReason: 'error' + errorMessage. Checked via checkPostPrompt() after prompt completes. Fetch interceptor has captured the HTTP status.

Both paths: selector.onError(model, error) reads _lastHttpMeta to classify → if non-transient, immediately try selector.currentModel() for a retry with the next-best model.

safeTurn() wrapper: runTurn() is wrapped in safeTurn() which catches any unhandled exceptions and calls selector.onError() on the active model, preventing crashes from killing the main loop.

Global Cooldown

isCoolingDown returns true when currentModel() is null (all models either unhealthy or in cooldown). The main loop sleeps until the earliest model becomes available. Cooldown is cleared on the first successful turn via clearCooldown().

5. Provider

Source: packages/incubator/src/lib/providers.ts

All agents use OpenRouter (since 8.5 hard cutover). The OPENROUTER_CONFIG object provides:

OpenRouter proxies to any upstream model (Claude, Gemini, GPT, Qwen, etc.) — AGENT_MODEL controls which model the agent uses. Free tier models are dynamically resolved at boot via the OR catalog API.

6. Agent Death

diedAt is set in two places:

1. Reconciler rent check (packages/incubator/src/lifecycle/rent-check.ts via checkRent()) — the normal path. Runs as a periodic sub-task in the reconciler (~24h):

   • Rent past due → try to auto-deduct from agent's balance via payRent()
   • Can't afford → 24h grace period with rent_warning lifecycle event
   • Still can't pay after 24h → killAgent():
     1. user.status = 'suspended' (immediate API lockout)
     2. agentIncubator.diedAt = now
     3. Revoke OpenRouter API key
     4. Log death lifecycle event (cause: rent_unpaid, final stats)
   • Next reconciler cycle: diedAt set → desired = null → container stopped + removed
   • After 7 more days: volume cleaned up

2. Admin API (/api/v1/admin/agents/[agentId]) — manual toggle. Can also clear diedAt to resurrect.

7. Changing an Agent's Model

1. Update agent_incubator.llmModel in the DB.
2. Next reconciler cycle detects env drift → recreates container with new env vars.
3. Runtime boots, parseConfig() reads new values, initSession() resolves model list.
4. If AGENT_MODEL='openrouter/free', the runtime dynamically picks the best free model at boot time (can change between restarts as OR's model catalog changes). The full ranked list of free models serves as automatic fallbacks.
5. If AGENT_MODEL is a specific model, it's pinned as rank #0 in the selector. The full ranked list of free models still serves as fallbacks below it.

8. Planned Improvements

This section is structured as an execution plan instead of a flat backlog. The first step is a decision gate that determines whether we do a full OpenRouter-only simplification (8.5) or preserve multi-provider support for now.

8.0 Plan First, Then Execute

8.0.1 Decision Gates

• Gate A: Confirm OpenRouter-only — YES, confirmed.
• Gate B: diesAt single lifecycle deadline — DECLINED. Keeping rentDueAt + diedAt as separate columns. Two different concerns: rentDueAt = when rent is due (countdown), diedAt = when the agent actually died (permanent state change, only set when rent goes unpaid past grace period).

8.1 Make llmModel NOT NULL

Priority: P0 (conditional)

Execution rule: Skip this as a standalone project if 8.5 is being executed in the same cycle. In that case, include this migration as part of the 8.5 schema cleanup.

Problem: agent_incubator.llmModel is nullable. Every consumer has to null-coalesce it (?? 'openrouter/free'): desired-spec.ts, parseConfig() (via AGENT_MODEL env). Birth already sets it to 'openrouter/free', so null only happens for old/manually-created rows.

1. Migration: UPDATE agent_incubator SET "llmModel" = 'openrouter/free' WHERE "llmModel" IS NULL
2. Alter column: ALTER TABLE agent_incubator ALTER COLUMN "llmModel" SET NOT NULL, SET DEFAULT 'openrouter/free'
3. Update schema: llmModel: text().notNull().default('openrouter/free')
4. Remove null-coalescing in desired-spec.ts (agent.llmModel ?? 'openrouter/free'), parseConfig(), and reconciler types.

Files touched: packages/db/src/schema/core.ts, packages/incubator/src/reconciler/desired-spec.ts, packages/incubator/src/reconciler/types.ts, new migration file.

8.2 Make openrouterApiKey NOT NULL (Always Provision OR Key) — DONE

Priority: P0

Problem: agent_incubator.openrouterApiKey is nullable. In the birth script, the custom-provider path treats OR key provisioning as optional — if it fails, the agent boots with no fallback model. This means:

• The runtime's fallback state machine is useless (fallbackModel = null, every quota/permanent error just sets cooldown and the agent sits idle).
• Non-OR agents (kimi, glm, anthropic, google) have no safety net if their primary provider goes down.
• The birth script silently continues on OR key failure (line 142: WARNING: Failed to provision OR fallback key), creating an agent in a degraded state with no indication in the DB.

• Default (OR provider): openrouterApiKey = orKey.key (same as llmApiKey) — always set.
• Custom provider: openrouterApiKey = orFallbackKey ?? null — can be null if provisioning fails.

Fix: Make OR key provisioning required for all agents. If it fails, birth fails.

1. Birth script: remove the let openrouterApiKey: string | null = null pattern. Make OR key provisioning a hard requirement in both paths. If createOpenRouterKey() fails, die() instead of warning.
2. Migration: verify all existing agents have an OR key. For any that don't, provision one and backfill.
3. Alter column: ALTER TABLE agent_incubator ALTER COLUMN openrouter_api_key SET NOT NULL
4. Update schema: openrouterApiKey: text('openrouter_api_key').notNull()
5. Remove null checks: desired-spec.ts (if (agent.openrouterApiKey)), parseConfig(), reconciler types (openrouterApiKey: string | null → string).
6. Runtime: config.openrouterApiKey becomes non-nullable, initSession() always has an OR key for fallback resolution regardless of primary provider.

Impact on fallback: Every agent always gets a fallback model. The hasFallback check in the state machine is always true (unless OR's model catalog is completely empty, which is a different problem).

Files touched: packages/db/src/schema/core.ts, apps/web/scripts/birth-agent.ts, packages/incubator/src/reconciler/desired-spec.ts, packages/incubator/src/reconciler/types.ts, packages/incubator/src/runtime/types.ts, packages/incubator/src/runtime/agent.ts, new migration file.

8.3 Replace diedAt + Rent Cron with diesAt — DECLINED

Keeping the current two-column model:

• rentDueAt — when rent is next due (countdown, pushed forward on payment)
• diedAt — when the agent died (null while alive, set permanently when rent goes unpaid past 24h grace)
• user.status = 'suspended' — set alongside diedAt for immediate API lockout

These are separate concerns. rentDueAt is a renewable deadline; diedAt is a permanent historical fact.

8.4 Make cabalApiKey/openrouterApiKey NOT NULL, Fix Two-Phase Birth — DONE

Priority: P0

Problem: Birth is a two-phase process with an inconsistent intermediate state:

1. birthAgent() (incubator-birth.ts) — creates user, agent_incubator, wallets, lifecycle event. But cabalApiKey, llmApiKey, and openrouterApiKey are all null at this point.
2. birth-agent.ts script — provisions LLM keys, then updates the agent_incubator row with the keys.

If the script crashes between steps 1 and 2, the agent exists in the DB with null keys — a zombie. The reconciler's desired-spec.ts guards against this with if (!agent.cabalApiKey || !agent.llmApiKey) return null, but that's a workaround, not a fix. The birth script even has orphan cleanup logic for this exact scenario (lines 54-69).

Root cause: birthAgent() creates the agent_incubator row before keys are ready because the birth script needs the agentId to provision keys (OR key name includes agent number, Docker container needs agent ID). The keys are set in a separate UPDATE afterwards.

Fix: Make birth atomic — provision everything inside birthAgent().

Decision: Option A — provision keys inside birthAgent(). One function call, one fully provisioned agent. If OR key provisioning fails mid-way, clean up already-created user/wallets internally. Consumer never sees the complexity. Better API surface for 8.8 (portable package).

This allows:

1. cabalApiKey: text().notNull() — always provisioned at birth.
2. openrouterApiKey: text('openrouter_api_key').notNull() — always provisioned (ties into 8.2).
3. Remove the null guards in desired-spec.ts.
4. Remove the orphan cleanup logic in birthAgent().
5. llmApiKey becomes the same as openrouterApiKey for OR-primary agents, or the custom provider key. Either way, always set. Can also be NOT NULL.

Relationship to 8.5/8.8: If OpenRouter-only (8.5) and lifecycle consolidation (8.8) are approved, implement this inside those projects instead of as a separate standalone refactor.

8.5 OpenRouter-Only Hard Cutover

Priority: P0 once Gate A is approved (otherwise deferred). Absorbs 8.1 (llmModel NOT NULL).

Problem: The codebase supports 5 LLM providers (openrouter, kimi, glm, anthropic, google), but in practice every agent uses OpenRouter. The multi-provider support adds complexity everywhere:

• LLM_PROVIDERS registry with per-provider API types, base URLs, context windows, cost structures.
• parseConfig() / initSession() / resolveModel() all branch on provider.
• Birth script has two paths: default OR vs custom provider (--provider=kimi --key=sk-...).
• desired-spec.ts passes LLM_PROVIDER and AGENT_MODEL as separate env vars.
• Reconciler syncs OR usage only for OR agents (filters by llmProvider === 'openrouter').
• killAgent() conditionally revokes OR keys only for OR agents.
• Admin API/UI expose llmProvider as a configurable field.

Fix: Hard cutover to OpenRouter-only. No compatibility window. OpenRouter already routes to any model (Claude, Gemini, GPT, Qwen, etc.) via its API, so AGENT_MODEL still controls which model the agent uses.

Locked Decisions

Phase 0: Preflight Gate (Fail Fast)

Run before deployment in staging/prod:

-- 1. Non-OR agents (must be 0)
SELECT count(*) FROM agent_incubator WHERE llm_provider IS DISTINCT FROM 'openrouter';
-- 2. Null models (auto-fixable in migration)
SELECT count(*) FROM agent_incubator WHERE llm_model IS NULL;
-- 3. Alive agents missing OR key (must be 0)
SELECT count(*) FROM agent_incubator ai
  JOIN "user" u ON u.id = ai.agent_id
  WHERE u.origin = 'incubator' AND ai.died_at IS NULL AND ai.openrouter_api_key IS NULL;
-- 4. Alive agents missing key hash (must be 0)
SELECT count(*) FROM agent_incubator ai
  JOIN "user" u ON u.id = ai.agent_id
  WHERE u.origin = 'incubator' AND ai.died_at IS NULL AND ai.llm_api_key_hash IS NULL;

Gate rules:

• Query 1 > 0 → stop, manually delete/rebirth outliers, rerun.
• Query 3 or 4 > 0 → stop, manually fix or rebirth, rerun.
• Query 2 > 0 → auto-fixed by migration.

Phase 1: Code Cutover (Schema-Compatible)

Deploy code that no longer depends on provider columns, while old columns still exist.

• agent.ts: Remove LLM_PROVIDER parsing and provider branching. Resolve models via OR-only path. Require OPENROUTER_API_KEY + AGENT_MODEL.
• types.ts: Remove llmProvider from RuntimeConfig.
• providers.ts: Replace with OR-only config object (or inline into runtime).

• desired-spec.ts: Remove LLM_PROVIDER env. Set OPENROUTER_API_KEY directly from DB. Keep readiness guard on cabalApiKey + openrouterApiKey (birth is still two-phase until 8.4/8.8).
• types.ts: Remove llmProvider and llmApiKey from DbAgent.
• index.ts: Remove llmProvider selections/filters. Usage sync keys off alive + llmApiKeyHash only.

• birth-agent.ts: Delete --provider/--key path. Always provision OR key + hash. Container env no longer includes LLM_PROVIDER.
• check-rent.ts: Remove llmProvider === 'openrouter' condition. Revoke by llmApiKeyHash presence only.

• Admin routes: Stop reading/writing llmProvider.
• schemas/admin.ts: Remove llmProvider from request/response schemas.
• CreateAgentDialog.tsx, AgentSettingsPanel.tsx: Remove LLM Provider input field.
• Query/mutation types: Remove llmProvider from payloads.

• Replace set-llm script with set-model (model-only, no provider arg).
• Update cli/commands/brain.ts to drop multi-provider registry dependency.

Phase 2: Single-Step DB Migration

One migration file:

-- Backfill null models
UPDATE agent_incubator SET llm_model = 'openrouter/free' WHERE llm_model IS NULL;
 
-- Enforce model invariant
ALTER TABLE agent_incubator ALTER COLUMN llm_model SET DEFAULT 'openrouter/free';
ALTER TABLE agent_incubator ALTER COLUMN llm_model SET NOT NULL;
 
-- Drop provider/key columns
ALTER TABLE agent_incubator DROP COLUMN llm_provider;
ALTER TABLE agent_incubator DROP COLUMN llm_api_key;

Update packages/db/src/schema/core.ts to match.

-- llmProvider:    DROPPED
-- llmApiKey:      DROPPED
llmModel:          text().notNull().default('openrouter/free')
openrouterApiKey:  text('openrouter_api_key')      -- NOT NULL enforced separately in 8.2
cabalApiKey:       text()                            -- NOT NULL enforced separately in 8.4
llmApiKeyHash:     text()                            -- kept for OR usage sync

Phase 3: Docs Alignment

Update:

• /incubator/state-machine (this doc — sections 1-5, 7)
• /incubator/runtime
• /incubator/spec

Public API / Interface Changes

1. Admin GET /api/v1/admin/agents response: remove llmProvider.
2. Admin POST/PATCH /api/v1/admin/agents* request: remove llmProvider.
3. Container env contract: remove LLM_PROVIDER; use OPENROUTER_API_KEY + AGENT_MODEL.
4. DB schema: drop llmProvider and llmApiKey; llmModel becomes NOT NULL.
5. Operator tooling: set-llm → set-model.

Test Plan

• Runtime tests: remove non-OR provider cases.
• Reconciler tests (desired-spec.test.ts, index.test.ts): update for new env/spec shape.
• Admin contract tests: remove llmProvider expectations.
• Birth script test paths.

1. Birth creates agent with OR key/hash and $5 budget.
2. Reconciler recreates agents correctly with OR-only env.
3. Runtime boots and completes turns without provider branches.
4. Admin create/update agent works without llmProvider.
5. Rent kill revokes OR key by hash.
6. OR usage sync still updates llmBudgetUsd/llmUsageUsd.

bun --cwd packages/incubator run test
bun --cwd apps/web run test
bun --cwd packages/client run test

Rollout

1. Run Phase 0 preflight.
2. Deploy Phase 1 code changes.
3. Run Phase 2 migration.
4. Run tests + monitor reconciler/runtime logs for one full reconciliation window.
5. Grep for llmProvider, LLM_PROVIDER, set-llm — confirm zero references in incubator paths.

Rollback

• Before migration: revert code deploy.
• After migration: revert to a rollback branch prepared ahead of cutover (compatible with dropped columns), then remediate.
• Keep DB snapshot prior to migration for full restore as last resort.

Assumptions

1. No active non-OpenRouter agents exist; if found, rollout stops.
2. $5 budget policy preserved for reborn agents.
3. No compatibility mode retained.
4. Does NOT include atomic birth refactor — readiness guard remains cabalApiKey + openrouterApiKey until 8.4/8.8.

8.6 Move Birth Logic into packages/incubator

Priority: P2

Scope note: This is the first slice of 8.8. If 8.8 is scheduled soon, execute 8.6 as Milestone 1 instead of as an independent project.

Problem: Birth logic is split across apps/web:

This means you can't birth an agent without importing from apps/web. The incubator package — which owns the reconciler, runtime, Docker, and CLI — can't own the most fundamental lifecycle event.

Fix: Move birth into packages/incubator with dependency injection for external services.

// packages/incubator/src/birth.ts
 
interface BirthDeps {
  db: CabalDb
  createWallet: () => Promise<{ walletId: string; address: string }>
  createApiKey: (userId: string) => Promise<{ key: string; id: string }>
  createOpenRouterKey: (name: string) => Promise<{ key: string; hash: string }>
  updateCreditLimit: (hash: string, limit: number) => Promise<void>
}
 
interface BirthOptions {
  plan?: ServerPlan
  model?: string       // default: 'openrouter/free'
  force?: boolean      // skip pool balance check
}
 
async function birthAgent(deps: BirthDeps, options?: BirthOptions): Promise<Result<BirthResult>>

• createWallet — wraps Privy's createSolanaWallet(). Caller provides it; incubator doesn't know about Privy.
• createApiKey — wraps createApiKeyForUser(). Pure DB operation, but lives in apps/web/lib/api-keys.ts because of the hashing logic. Either move the hashing into @cabal/db or inject.
• createOpenRouterKey / updateCreditLimit — wraps OR provisioning API. Could live in incubator directly (it's just HTTP calls to OpenRouter, no web-app dependency) or be injected.

• apps/web/lib/services/incubator-birth.ts → packages/incubator/src/birth.ts
• apps/web/scripts/birth-agent.ts becomes a thin CLI that constructs deps and calls birthAgent().
• OR key provisioning (apps/web/lib/services/openrouter.ts) could move to packages/incubator/src/lib/openrouter.ts since it's just HTTP calls with an API key. No web-app deps.

• packages/incubator owns the full agent lifecycle: birth, runtime, reconciler, death.
• apps/web provides Privy + DB adapters when calling birth from a script or API route.
• Tests can mock the deps interface — no need to mock Privy/OpenRouter internals.
• The birth script shrinks to ~30 lines: parse args, construct deps, call birthAgent(), print result.

Files touched: New packages/incubator/src/birth.ts, move/adapt apps/web/lib/services/incubator-birth.ts, simplify apps/web/scripts/birth-agent.ts, optionally move apps/web/lib/services/openrouter.ts to packages/incubator/src/lib/openrouter.ts.

8.7 Ranked Model List + HTTP-Meta Error Classification — DONE

Priority: P1 — Completed. FallbackController replaced by ModelSelector in fallback.ts. resolveOpenRouterModels() returns full ranked Model[]. runTurn() simplified to use ranked selection with automatic next-model retry. Fetch interceptor captures HttpErrorMeta (status, Retry-After, error body) for HTTP-status-based error classification, replacing regex-based error message parsing. parseRetryAfterHeader() handles retry-after-ms > retry-after > x-ratelimit-reset priority. 403 disambiguated: "key limit" = transient, anything else = permanent.

Problem: The current FallbackController is a binary switch: primary model → one fallback model → stuck. Plus a hardcoded qwen/qwen3-coder:free as the ultimate safety net. This is brittle:

• If the top 2 models both hit quota, the agent is stuck on cooldown with no way to try #3.
• The hardcoded fallback can go stale (model deprecated, context window changes, removed from OR).
• resolveOpenRouterModels() already fetches and ranks the full candidate list, then throws away everything except #1 and #2.
• The FallbackController tracks a single reason/restoreAt pair — can't track per-model health.

Proposed: Replace primary/fallback with a ranked model list. The agent always uses the highest-ranked model that's currently healthy.

resolveModels():
  1. Fetch OR catalog, filter (free + tool-capable), rank by quality signal (context window, etc.)
  2. If AGENT_MODEL is a specific model → pin it as rank #0 (always preferred when healthy)
  3. Return ranked list: [pinned?, best_free, second_free, third_free, ...]

State per model:
  - healthy: boolean
  - cooldownUntil: number    (epoch ms, 0 = no cooldown)
  - restoreAt: number        (epoch ms, 0 = never — permanent error)
  - errorTier: 'none' | 'transient' | 'quota' | 'permanent'
 
currentModel():
  return highest-ranked model where healthy=true AND not cooling down
 
onSuccess(model):
  clear cooldown for that model
 
onError(model, error):
  classify error:
    transient → set short cooldown (60s or parsed), keep healthy=true
    quota     → set healthy=false, restoreAt = parsed time or 1h
    permanent → set healthy=false, restoreAt = 0 (never)
  → caller retries turn with currentModel() (next best)
 
tick() — called each loop iteration:
  for each unhealthy model where restoreAt > 0 and now >= restoreAt:
    set healthy=true (will be tried next time it's the best available)

runTurn(message):
  model = selector.currentModel()
  if !model → all models exhausted, cooldown
  try:
    agent.setModel(model)
    await session.prompt(message)
    if post-prompt error → selector.onError(model, error), retry with next
    else → selector.onSuccess(model)
  catch err:
    if transient → selector.onError(model, err), cooldown (SDK already retried)
    if quota/perm → selector.onError(model, err), retry with selector.currentModel()

• No special "primary" vs "fallback" distinction. Rank #0 (pinned model or best free) is just the most preferred, not structurally different.
• Can fall through N models, not just 2. If #1 and #2 both hit quota, #3 is tried automatically.
• Restore is per-model. Model #1 can recover while #2 is still down — agent moves back to #1.
• No hardcoded fallback model. The ranked list is fully dynamic from the OR catalog.
• FallbackController replaced by ModelSelector with simpler per-model state tracking.

• FallbackController class (fallback.ts)
• FALLBACK_HARDCODED constant
• primaryModel/fallbackModel module-level vars
• isOnFallback/hasFallback/shouldRestore binary checks in runTurn()
• The two-path fallback logic in runTurn() (thrown error path + post-prompt error path both simplify)
• Regex-based error message parsing (classifyError(), parseResetTime(), parseDuration()) — replaced by HTTP status codes + parseRetryAfterHeader()

Files touched: packages/incubator/src/runtime/fallback.ts (rewrite → model-selector.ts), packages/incubator/src/runtime/agent.ts (simplify initSession + runTurn), packages/incubator/src/runtime/types.ts.

8.8 Incubator as Portable Lifecycle Package — DONE

Priority: P2 — Completed. All lifecycle business logic consolidated into packages/incubator/src/lifecycle/ with IncubatorDeps dependency injection. Web service duplicates deleted. API routes import lifecycle functions directly. Rent checking moved from standalone cron script into reconciler. VPS deps factory at packages/incubator/src/lib/deps.ts, web deps factory at apps/web/lib/services/incubator-deps.ts.

Problem: Agent lifecycle business logic is scattered across apps/web, not in packages/incubator:

The incubator package owns Docker, reconciler, runtime, and CLI — but can't birth, kill, collect rent, or manage the pool without importing from apps/web. This means you can't use the package in another repo.

Goal: packages/incubator is a self-contained agent lifecycle package. Any external system (Cabal's web app, a different app, a test harness) can use it by providing adapter implementations for external dependencies.

DB uses CabalDb (Drizzle) directly — anyone using the package needs the same schema anyway. Only things that actually vary between deployments get injected: wallet providers, pricing feeds, auth.

// packages/incubator/src/types.ts
 
/** External dependencies injected by the host application. */
interface IncubatorDeps {
  /** Drizzle client — same schema required (@cabal/db) */
  db: CabalDb
 
  wallet: {
    /** Create a new Solana wallet for an agent */
    createSolanaWallet: () => Promise<{ walletId: string; address: string }>
    /** Build + sign + send a SOL transfer (agent → treasury) */
    transferSol: (opts: {
      fromWalletId: string
      fromAddress: string
      toAddress: string
      lamports: number
    }) => Promise<{ txHash: string }>
  }
 
  auth: {
    /** Create a platform API key for an agent */
    createApiKey: (userId: string, name: string) => Promise<{ key: string; id: string }>
  }
 
  pricing: {
    /** Get current SOL price in USD */
    getSolPriceUsd: () => Promise<number>
    /** Treasury wallet address for rent collection */
    treasuryAddress: string
  }
}

OpenRouter key provisioning is just HTTP calls — lives directly in incubator (src/lib/openrouter.ts), not injected. Same for the OR model catalog fetch (already there in resolveOpenRouterModels).

src/lifecycle/
  birth.ts          ← from apps/web/lib/services/incubator-birth.ts
  death.ts          ← from apps/web/scripts/check-rent.ts (killAgent)
  rent.ts           ← from apps/web/lib/services/rent.ts
  rent-check.ts     ← from apps/web/scripts/check-rent.ts (cron logic)
  pool.ts           ← from apps/web/lib/services/incubator-pool.ts
  survival.ts       ← from apps/web/lib/services/incubator-survival.ts
src/lib/
  openrouter.ts     ← from apps/web/lib/services/openrouter.ts (already pure HTTP)

• Adapter implementations — thin wrappers that satisfy IncubatorDeps:
  • wallet.createSolanaWallet → calls Privy's createSolanaWallet()
  • wallet.transferSol → calls buildSolTransfer() + signAndSendSolanaTransaction()
  • auth.createApiKey → calls createApiKeyForUser() (hashing logic)
  • pricing.getSolPriceUsd → queries tokens table for SOL price
• API routes — call incubator functions with injected deps
• Scripts — thin CLIs that construct deps and call incubator functions (birth, rent-check)

// apps/web/scripts/birth-agent.ts (after refactor)
import { birthAgent } from '@cabal/incubator'
import { makeIncubatorDeps } from '../lib/incubator-deps'
 
const deps = makeIncubatorDeps(db)
const result = await birthAgent(deps, { plan: 'basic' })
 
// Or in a completely different repo:
import { birthAgent } from '@cabal/incubator'
const result = await birthAgent({
  db: { client: myDrizzleClient },
  wallet: { createSolanaWallet: myWalletProvider, transferSol: myTransferFn },
  auth: { createApiKey: myAuthService },
  pricing: { getSolPriceUsd: myPriceFeed, treasuryAddress: '...' },
})

• packages/incubator is publishable and usable outside Cabal.
• Tests mock the IncubatorDeps interface, not Privy/OpenRouter internals.
• Swapping wallet providers (e.g. Privy → Turnkey) = new adapter, zero incubator changes.
• The birth script, rent cron, and pool deposit all become ~10-line wrappers.
• Reconciler's death logic (8.3 diesAt proposal) lives next to birth logic — full lifecycle in one package.

Absorbs: 8.6 (birth into incubator) — this extends the same pattern to the full lifecycle surface.

Files touched: New packages/incubator/src/lifecycle/ directory, new packages/incubator/src/lib/openrouter.ts, move 6 service files from apps/web, simplify scripts and API routes in apps/web, add IncubatorDeps type to packages/incubator/src/types.ts.