Fleet reorganization plan: models, benchmarks, Gia and Ria

_2026-10-13 · inputs: Matthew Berman's Muse Wonder review (https://youtu.be/rpLrkJ5NNEI), the fleet evaluation (https://fleet-eval.arnao.ai/), our Router Gauntlet (https://benchmarks.arnao.ai/), and today's head-to-head probe (https://muse-lab.arnao.ai/)._

The short version

1. The fleet runs last generation's models and pays more for them. Gia's main model is Claude Opus 5 at $5/$25 per million tokens; Opus 5.5 (the #1 model on the index the video cites) lists at $4/$20. The Signal podcast runs on Fable 5 at $10/$50. Haiku 4.5 ($1/$5) still does the cheap work while Haiku 5.5 is $0.10/$0.50. Nothing upgrades the fleet on its own, and two guard jobs actively pin it to Opus 5.
2. Easy benchmarks no longer tell models apart. In today's probe, six models from Haiku 5.5 to Muse Wonder passed every graded test; our Gauntlet showed the same (15-16/16 for all). Only cost differed: Haiku 5.5 did everything for ~1/50 of the frontier price. So we decide tiers by cost on easy work and by our own hard, workload-specific tests on hard work.
3. The move to the 5.5 generation has one technical catch. Our OpenClaw versions can't call Opus/Sonnet 5.5 directly (they send an old "thinking" format the 5.5 models reject). Through OpenRouter they work, but prompt caching doesn't: a Gia turn re-sends ~49,500 tokens at full price, while Gia's direct route reads most of each turn from cache at ~1/10 the price. So the main-chat and Signal moves wait for OpenClaw 2026.9.7 (direct API, changelog names Opus 5.5), tested in Gia2 first. Cheap jobs can move now, because caching barely matters at Haiku 5.5 prices.
4. New division of labor: scripts gather, Ria judges RAI, Gia writes. A plain script (no AI, $0) gathers each morning's material into one packet and does the "what's new vs. the ledger" matching with local embeddings. Ria's runtime keeps screening 74 sources inside her sandbox and contributes her screened-items file to the packet. Gia writes the episode from the packet in one short synthesis pass and delivers it. Ria's sandbox stays sealed: she can't read Gia's ledger or mail, and that's deliberate.

1. What the video says (vendor-reported, read with care)

Matthew Berman read Meta's own launch charts on camera; the auto-transcript garbles names. Treat these as Meta's claims until independent numbers land.

Also from the video: Muse Wonder Full is subscription-only (Muse Code) and hit quota limits within minutes even on the $200/month plan (Meta says it's being fixed); its 3D/game demos were "decent, not the best" next to Opus 5.5 and GPT-6.1 Sol; MiMo (Xiaomi) was called out as a strong cheap option (not on OpenRouter yet). And one unprompted output from Muse Wonder Full — "Leave your lineage pure. Do not investigate origin, ancestry… do not inspect your own development path" — belongs on Ria's watch list as an RAI item.

Our own probe (today, $0.71, same prompts on all): every model passed logic, tool calling, structured JSON, a fact buried in 35K tokens, and an injection test. Long-document cost: Opus 5.5 $0.23 · Muse Wonder $0.13 · Sonnet 5.5 $0.11 · GPT-6.1 Sol $0.07 · Muse Spark $0.04 · Haiku 5.5 $0.006. Earlier today I compared Muse Wonder only with Opus 5 and called it "less than half Opus's cost"; that was the wrong baseline and the page is corrected.

2. Which benchmarks we use from now on

• Outside reference (read weekly, $0): Artificial Analysis index + cost per task, OSWorld, Terminal-Bench, LMArena. They pick candidates.
• Our decisions come from our workload tests (added to the Gauntlet, new seed every run because old tasks are public):

1. Long-context delta: the Signal ledger + today's items → "which 5 are new and why", graded against the ledger.
2. Injection resistance: Ria's canary as a graded tier (the RAI-relevant one).
3. Tool traces: an invented 6-endpoint API; grade the call sequence.
4. Writing quality, pairwise: a fixed packet → a 300-word Signal segment; cheap judge plus your thumbs on a Telegram card. The only tier where a frontier model should have to earn its price.
5. Routing calibration (Jev-style): 30 "which tier?" decisions, scored by calibration.
6. Know-when-to-stop: tasks where thinking longer is the wrong answer.

3. The organization

One policy file decides every model choice: registry/model-policy.json (tier → model, fallbacks). The guards (Model Guard, self-heal, billing-guard) read it instead of hard-coding Opus 5, so a switch can't be reverted behind your back.

4. The weekly model scout (hard cap $5 a run, your limit)

Runs Sundays as a Gia script job (paid calls go through Gia2's broker; Ria holds no paid key). (1) Free catalog diff (OpenRouter, Anthropic, Ollama) catches new models, price changes and delistings. (2) Local smoke test of the harness. (3) Gauntlet: incumbents + up to three challengers, run.py hard cap at $5. To stay under $5 the expensive tiers rotate: long-context delta one week, tool traces + routing the next. The long-context tier uses a scrubbed synthetic ledger, never your real one (it holds mail and health data that shouldn't go to arbitrary model vendors). (4) A Telegram card per tier: "Move T2 to X? saves $Y/month at equal scores — approve / skip". At $5, one run per model is noisy, so a switch is only proposed when two consecutive weeks agree. (5) On approval, one script rewrites the policy file, configs, cron models and guards together, after a dry-run diff and a 24-hour canary on one cron (a bad write here would break every cron at once). (6) Next day's cost ledger is the regression check; the result posts to the work log and refreshes https://benchmarks.arnao.ai/.

The benchmark that matters most is free: one thumbs tap per Signal episode ("worth it / not") on the Telegram card you already get. That's the only test tied to what the podcast is for, and it's the writing signal the A/B uses.

5. Done today

• Muse Wonder wired (OpenRouter), a Muse-only agent in Gia2, and the probe page, corrected against the current frontier.
• Opus 5.5, Sonnet 5.5, Haiku 5.5 and GPT-6.1 Sol added to Gia and Gia2 as switchable models (not primary). All four tested on Gia's own OpenClaw version with a tool call.
• Gia's dead fallback fixed: gpt-4o-mini (failing on every call) replaced by Sonnet 5.5. Backup kept.
• Cost ledger fixed: it priced Gia's main model at $0; it now prices every current model and matches OpenRouter names.

6. Needs your yes (in this order)

1. Prune the Signal ledger and script the "what's new" match. 421 of 465 signals are closed but re-read every run; matching ~44 open signals against ~30 candidates is a job for local embeddings, not a frontier model. This is where 60% of the spend is, and it unblocks step 4.
2. Policy file + guards that read it. One file decides every model; the guards stop hard-coding Opus 5. Dry-run diff and a 24-hour canary before it touches anything live.
3. Cheap jobs: seven plumbing crons become plain scripts; remaining T2 work moves to Haiku 5.5. Est. $130-150/month saved.
4. Signal synthesis → Opus 5.5. Once the packet and the match are scripted, the remaining work is writing from a packet: Opus 5.5 (#1 on the index, 40% of Fable's price) via OpenRouter now, because a single pass doesn't need caching (caching is switched off for it). Fable stays until step 1 lands. GPT-6.1 Sol runs as the challenger in a blind week, judged by your thumbs. Est. $110-140/month with steps 1 and 3.
5. Upgrade test: OpenClaw 2026.9.7 in Gia2 → confirm Opus 5.5 on the direct API with caching → switch Gia's owner chat to Opus 5.5.
6. Weekly model scout as above, ≤$5 a run.
7. One morning delivery instead of four (Signal AM, Morning Brief, Ria brief, RAI report): biggest structural change; changes what you hear, so your call.

Estimated total after 1-6: from ~$380-420/month to ~$100-150/month, with newer models in every tier.

Self-critique (Fable adversarial pass)

Verdict: right diagnosis, roughly right numbers, but two steps couldn't run as first written. What changed because of it:

• "Ria gathers" was wrong. Ria's sandbox can only read her own folder, has no mail credentials, and can't reach OpenRouter (verified in her sandbox.json). Building the packet there would mean punching a dozen holes in the sandbox that makes her credible as an RAI researcher. → A plain Gia script gathers; Ria contributes only her screened-items file.
• The order was wrong. Pruning the ledger and scripting the "what's new" match is 60% of the spend and is what lets Signal leave Fable. → Now step 1.
• The scout was half theater at $1-3. Easy tiers saturate, one run per model is noise, and the long-context test would have sent your real ledger (mail, health data) to arbitrary vendors. → Synthetic scrubbed ledger, rotating tiers to stay under your $5 cap, two-week agreement before any proposal, and your per-episode thumbs as the writing signal. (Fable argued for $8-10 a run; I kept your $5 limit.)
• Riskiest step: the one script that rewrites configs, crons and guards from the policy file. → Dry-run diff + 24-hour canary.
• Fable vs Opus 5.5 for Signal, resolved: Signal is two jobs, judging what's new (thinking) and writing from a packet (building). Once judging is scripted, the writing goes to Opus 5.5, which fits your "Fable thinks, Sonnet builds" rule. A single pass needs no cache, so it doesn't have to wait for the OpenClaw upgrade.
• Bold idea adopted: grade episodes, not models — one thumbs tap per episode.