Arena / Fault Injection — Break It Deliberately, Watch It Rise Again

Other agent frameworks show you how smart an agent is: fluent dialogue, strong reasoning, slick tool use. ares wants to show something else: break it deliberately, and see whether it gets up on its own. You click a button and kill a working agent to watch whether it recovers.

0.3.x update: fault injection and recovery verification are now split across two independent surfaces.

• internal/aresrecovery: targets the new Kernel model (agentfabric + taskfabric). The source comment states it plainly: "Chaos breaks, Recovery fixes." This is the surface that is actually wired into the production serve path — with a fail-safe latch.
• internal/runtime/arena: targets the old "leader / sub-agent + DAG" model, running as a standalone ares arena serve chaos-drill process. Its Injector comment is equally plain: "It does NOT implement recovery; the existing resurrection plugin and failover handle that automatically."

Boundary of this article: every symbol and flow below is something I actually read in this codebase. Where something is "configured but explicitly not done yet," or needs extra wiring to take effect, I mark it (待核实 / TBD) rather than dressing it up.

1. Why build a "break things" feature

Killing a running agent by hand and watching it come back and finish what it started is satisfying. But satisfaction alone proves nothing — you'd only be guessing whether the system can actually take a beating.

That's the point of chaos engineering: not one manual click, but systematic, repeatable fault injection followed by recovery verification. It is the laboratory for the lifecycle assertion from part 7 of this series — "agent death ≠ task death."

2. First, tell the two "chaos" surfaces apart

The easy trap when reading this code is that there are two fully independent fault-injection/recovery surfaces, and they must not be mixed up:

Both are called "chaos," but one is production-wired recovery verification with a fail-safe latch; the other is a drill process isolated from production. The rest of this article covers each.

3. Kernel chaos: Chaos breaks, Recovery fixes

The comment at the top of internal/aresrecovery/chaos.go is the charter for this module:

"Chaos is the Failure Injection + Recovery Verification harness… it deliberately kills agents to prove the Runtime (Recovery subsystem) can restore their tasks. Chaos is SEPARATE from Recovery — Chaos breaks, Recovery fixes."

Broke into parts:

• FailureType has exactly two values:
  • FailureKill — a hard kill (crash): the agent is removed immediately.
  • FailureSuspend — a soft pause (simulates a hang/stall, not a crash).
• Chaos.InjectFailure(ctx, agentID, failure) — destroys only. The comment stresses: recovery is NOT triggered here; you must call VerifyRecovery explicitly. That lets a test assert "task is stranded after injection" before "task is recovered after VerifyRecovery." It records injected[agentID] → FailureType.
• Chaos.VerifyRecovery(ctx) int — forwards to Recovery.RecoverFromAgentDeath, returns the number of tasks fully recovered. This is the "Recovery Verification" half of Chaos.

Recovery (recovery.go) does the actual fixing. The recovery chain is:

1. RequeueExpiredLeases() → sweeps expired leases and requeues only tasks whose lease genuinely expired back to READY. The comment stresses you must never iterate all READY tasks — that would hijack brand-new tasks unrelated to the crash.
2. RecoverTaskCheckpoint(taskID, replacementID) → finds/spawns a replacement agent, acquires the task for it, and installs the preserved checkpoint as the new agent's cognitive state (resume from the break point).
3. RestartAgent(deadAgentID, cognitive, capabilities) → revives in place under the same id when a death snapshot exists (continuous provenance), otherwise spawns a fresh identity. The restart budget is lifetime-cumulative: default 5 attempts, backoff 1s→30s capped (DefaultRestartPolicy); exhausted returns ErrRecoveryExhausted, so a broken agent cannot cycle forever. A successful revival does not reset the counter — only total deaths consume it.
4. RecoverFromAgentDeath(ctx) int → chains 1–3: sweep expired leases → requeue → checkpoint-resume each requeued task, returning the number fully recovered.

Two honest caveats:

• The comments on RecoverTaskCheckpoint and RecoverFromAgentDeath are labelled TEST / CHAOS-ONLY: they install checkpoints via agents.SetCognitiveState and acquire tasks by hand — a separate mechanism from the production scheduler path (runKernelRecoveryLoop → taskfabric.DecodeCheckpoint). They must not be wired into the production serve path. Production recovery runs through runKernelRecoveryLoop in cmd/ares.
• So the semantic weight of Chaos/Recovery here is recovery verification; the actual event-driven recovery loop in production is runKernelRecoveryLoop in cmd/ares/kernel_loop.go.

flowchart TB
    subgraph CHAOS["Chaos — breaks"]
        INJ["Chaos.InjectFailure(ctx, agentID, FailureType)"]
        K["FailureKill = hard kill (immediate removal)"]
        S["FailureSuspend = soft pause (simulate hang)"]
        INJ --> K
        INJ --> S
    end

    subgraph STATE["After agent death"]
        AD["agentfabric: agent killed / suspended"]
        EX["taskfabric: lease expires → task back to READY"]
        AD --> EX
    end

    subgraph REC["Recovery — fixes"]
        VR["Chaos.VerifyRecovery(ctx)"]
        R1["RequeueExpiredLeases: only truly-expired tasks"]
        R2["RecoverTaskCheckpoint: new agent + install checkpoint"]
        R3["RestartAgent: restart budget 5, backoff 1s→30s"]
        VR --> R1 --> R2 --> R3
    end

    EX -. "injected state handed to Recovery" .-> VR
    R3 --> DONE["return count of fully-recovered tasks"]

Production wiring: wireChaos (shadow / live)

cmd/ares/serve_chaos.go wireChaos wires Chaos into the serve path. The default and safest posture is shadow:

• Default mode=shadow (or disabled, or live-with-unmet-guards) → only shadowSandboxLoop runs: it periodically replays a canonical failure scenario on independent scratch fabrics (spawn→create→acquire→kill→lease-expire→recover) — never touching production agents — and reports the outcome to the introspect panel.
• To switch to mode=live, four guards must hold simultaneously: chaos.enabled=true, allow_live=true, non-empty eligible_capabilities (target whitelist; an empty list refuses to arm), and non-empty stop_token (emergency-stop credential). Any missing guard falls back to shadow.
• The live loop liveChaosLoop enforces six guardrails (the source's own words): emergency stop POST /api/chaos/stop, a fail-safe latch, a GA quiet window, a token-bucket rate limit, per-agent cooldown, and the capability whitelist. Each injection is InjectFailure(FailureKill) → runLiveChaosInjection → VerifyRecovery; if verification returns 0 recovered, the fail-safe latch trips and no further injections run until process restart.

flowchart LR
    subgraph Guards["live chaos six guardrails"]
        G1["emergency stop /api/chaos/stop"]
        G2["fail-safe latch: recovered=0 → stop"]
        G3["GA quiet window"]
        G4["token-bucket rate limit RatePerMin"]
        G5["per-agent cooldown"]
        G6["capability whitelist"]
    end
    LOOP["liveChaosLoop"] --> Guards
    Guards --> INJ2["InjectFailure(FailureKill)"]
    INJ2 --> VR2["VerifyRecovery(ctx)"]
    VR2 -->|"recovered=0"| LATCH["fail-safe: stop future injections"]
    VR2 -->|"recovered>0"| CONT["continue"]

4. The arena drill process: internal/runtime/arena

This is the old, production-isolated chaos-drill layer, launched by ares arena serve. It starts its own demo pool (arena-worker-1..3, type=coder) and a mutable DAG, specifically to demonstrate "inject a fault → watch the system react."

Core files (real paths):

internal/arena/…, internal/dashboard/static/app.js and cmd/arena/main.go do not exist in this repository. The real paths are above; arena ships no bundled static frontend — the front-end-observable surface is the REST/SSE endpoints.

4.1 The injector: a thin layer that does not recover

Injector depends on two interface subsets: RuntimeProvider (a subset of ares_runtime) and DAGProvider (a subset of mutable-DAG mutations). The interface-based design lets arena avoid importing concrete Runtime/DAG packages and makes mocking easy.

// internal/runtime/arena/injector.go
// Injector wraps existing ares_runtime/DAG APIs to inject chaos.
// It does NOT implement recovery; the existing resurrection plugin and
// failover handle that automatically.
type Injector struct {
	ares_runtime RuntimeProvider
	dag          DAGProvider
}

func (in *Injector) KillLeader(ctx context.Context) (string, error) {
	leaderID := ""
	for _, info := range in.ares_runtime.ListAgents() {
		if info.Type == "leader" { leaderID = info.ID; break }
	}
	if leaderID == "" { return "", ErrLeaderNotFound }
	if err := in.ares_runtime.StopAgent(ctx, leaderID); err != nil {
		return "", fmt.Errorf("arena: kill leader %s: %w", leaderID, err)
	}
	return leaderID, nil
}

It implements no recovery itself. Recovery is delegated/expected from ares_runtime's resurrection and failover. KillLeader just finds a type=="leader" agent and calls StopAgent — "a replacement leader is then elected by failover" is not implemented in this process.

internal/runtime/arena/e2e_chaos_recovery_test.go is a real end-to-end check: it drives a genuine ares_runtime.Manager, registers a pool of workers with rebuild factories, calls Manager.NotifyAgentDead(...) to model a batch crash, then polls the factory call count to assert resurrection happens asynchronously and the Manager still tracks a live pool. It scales over 16/64/128.

4.2 Thirteen actions

types.go defines 13 ActionTypes:

kill_leader kill_agent remove_node remove_edge pause_agent resume_agent slow_agent kill_orchestrator network_partition tool_timeout memory_corrupt mcp_disconnect llm_failure

Service.Execute dispatches each action to the matching Injector method via switch action.Type, then: updates Stats, calls MetricsCollector.RecordActionResult, emits an arena.* event to the EventStore, appends a kind=failure evidence record to the unified Evidence Store on failure, and calls any wired FlightBridge / EvolutionBridge.

4.3 Scenario orchestration

scenario.go defines action sequences in YAML. Real examples live in examples/arena/ (peer_failure.yaml, cascading_storm.yaml):

name: peer-failure-and-leaderless-recovery
config:
  stop_on_error: false
  parallel_actions: false
  warmup: 1s
  cooldown: 1s
actions:
  - delay: 2s
    action: { type: kill_agent, target_id: agent-1 }
    label: kill-agent-1
  - delay: 1s
    action: { type: kill_agent, target_id: agent-2 }
    label: kill-agent-2
  - delay: 3s
    action: { type: network_partition, target_id: agent-3 }
    label: partition-agent-3
  - delay: 1s
    action: { type: slow_agent, target_id: agent-3, metadata: { delay: 10s } }
    label: slow-agent-3

The pre-0.3.1 example was "leader assassination" built on a kill_leader action. The kernel scheduler is leaderless — no models.AgentType is "leader" — so Injector.KillLeader always returned ErrLeaderNotFound and the scenario was a silent no-op. It was rewritten as peer failure.

ValidateScenario checks the name, at least one action, non-negative delays, per-action validity, and non-negative max_concurrent/timeout. RunScenarioReport supports warmup/cooldown, an overall timeout, and stop_on_error.

Honest caveat: ScenarioConfig.parallel_actions and max_concurrent, plus per-action depends_on, are explicitly "configured but not yet implemented" — RunScenarioReport logs a warning and (TBD) always runs actions sequentially.

4.4 Survival mode

survival.go Service.RunSurvival picks a random action from the 13 and a random target (from listed agents or DAG nodes/edges) on each interval (default 30m / every 10s), recording a Timeline. SurvivalReport carries no hard-coded sample scores — no Score: 100.0 (A+) / 97.3 output lines exist in this code, so none are reproduced here.

5. The 3-dimensional resilience score

score.go ResilienceScore is a weighted 3-dimensional score; gradeFromScore maps fixed thresholds to letter grades.

flowchart TB
    subgraph Score["ResilienceScore — weighted 3D (score.go)"]
        A["Availability (weight 40%)"]
        R["Recovery (weight 30%)"]
        C["Consistency (weight 30%)"]
    end
    A --> A1["base = (total - failed) / total × 100"]
    R --> R1["recovery rate ×70% + recovery speed ×30%"]
    R1 --> R2["speed: avg ≤1s→100, ≥10s→0, linear"]
    C --> C1["rate with ≥20 samples, else heuristic"]
    A1 -.weighted sum.-> FIN["Final = A×0.4 + R×0.3 + C×0.3"]
    R2 -.-> FIN
    C1 -.-> FIN
    FIN --> G["grade: ≥95 A+ / ≥90 A / ≥80 B / ≥70 C / ≥60 D / else F"]

An honest note on the Consistency dimension (updated 2026-10-20): RecordConsistency is no longer a dead entry point. In the serve process, taskfabric.Fabric.WithConsistencyObserver(arena.NewConsistencyAdapter(kernel.arenaMetrics)) records one ok bool per Fabric.DecodeCheckpoint read-back. The window is FIFO-capped at 1000, and DataConsistencyRate is published only once ConsistencySamples >= 20; below that the rate is 0 and Unmeasured contains data_consistency. calcConsistency therefore checks metrics.ConsistencySamples >= 20:

• with enough samples it uses DataConsistencyRate, including a real 0% (a window of all-failed decodes is no longer mistaken for "no data");
• below the threshold it falls back to the heuristic: about half the failures are treated as data-related (dataRelated = max(1, failed/2)), deducting 5 points per unit.

Mind the process boundary: CalculateScore is called only inside arena.Service (ares arena serve), which has no task-fabric checkpoint stream, so that process's own data_consistency remains honestly unmeasured. The rate measured by the serve process is exposed on the chaos panel through recovery_metrics (data_consistency_rate / consistency_samples / consistency_measured), not cross-process into the arena scorer. The panel must render consistency_measured=false as "not measured" (or hide the number), never as a 0% consistency rate. The recovery-speed thresholds (≤1s → 100, ≥10s → 0) are likewise this score's own rubric, not a measured SLA.

6. HTTP and authentication

http.go Handler registers roughly 27 routes: /arena/leader/kill; /arena/agent/{id}/kill|pause|resume|slow|partition|tool-timeout|memory-corrupt|mcp-disconnect|llm-failure; /arena/node/{id}/remove; /arena/edge/remove; /arena/orchestrator/kill; plus stats/history/stream (SSE) / score/metrics, the three survival endpoints, and flight timeline/diagnostics plus scenario run/validate.

Arena exposes only destructive endpoints (kill leader, remove nodes, corrupt memory), so authentication defaults to deny:

• If an API key is set (--api-key or security.arena_api_key in ares.yaml), every request must carry the X-API-Key header (constant-time compare).
• With no key and no explicit --allow-anonymous, APIKeyAuthMiddleware returns 401 for every request — a "just open the dashboard and click" no-auth flow is rejected by default. --allow-anonymous is for local development only, and the comment warns it must never be enabled for a network-reachable deployment.

7. Flight recorder and evolution bridging

FlightBridge (integration.go) writes each arena action as a timeline event in the flight recorder and adds a diagnostic record for failures (calling flight.SuggestFix). This wiring is confirmed effective: service.Execute calls s.bridge.OnActionExecuted after every action.

EvolutionBridge (evolution_bridge.go) translates arena failures into PatchProposals for the evolution UngatedPatcher: ActionRemoveNode → PatchInsertNode, ActionKillAgent/KillLeader → PatchReplaceNode, infrastructure faults → PatchChangeRecoveryStrategy, and grades them via chaosPriority: faults with priority ≥ 9 (killing leader/orchestrator) go through UngatedPatcher.ApplyEmergency for immediate self-healing; the rest go through UngatedPatcher.Submit for evaluation. (Slow-agent / tool-timeout faults used to map to PatchChangeScheduler; that mapping was removed in 0.3.2, along with the patch's other producer, because the graph applier requires a real Scheduler value that neither producer could supply.)

(TBD): OnActionExecuted does construct a proposal and submit/emergency-apply it on failure. But whether the UngatedPatcher's evaluated proposal ultimately produces a real runtime/scheduling change depends on the UngatedPatcher and its patch applicator. Inside arena serve, this operates on the process's own demo pool and mutable DAG; whether these patches flow back to the real production runtime is something I could not confirm in the code covered here, so I flag it as TBD.

Worth adding is the execution feedback loop in cmd/ares/peer_mode.go (from aresrecovery, targeting the Kernel model) that is confirmed live:

• ExecutionAttribution.Record / RecordWithMetrics(agentID, capability, success, latency, retries, recovers) collects per-(agent, capability) outcomes.
• DeterministicScorer is a zero-LLM deterministic scorer: weights success 0.70 / latency 0.15 / retries 0.10 / recovers 0.05, always in [0,1], with a neutral prior of 0.5 for no history. It is the deterministic fitness signal for the GA — no LLM, no randomness.
• EvolutionFeedbackAdapter.Apply pushes attribution snapshots back into ConfidenceInjector.SetAgentConfidence / SetCapabilityConfidence, so the next schedule sees the new confidence — failure-heavy agents are downweighted, success-heavy agents preferred.
• Change attribution ChangeAttributor.Attribute apportions the BestScore delta between two adjacent GenerationSnapshots across each change: explicit Impacts win; the rest split the remaining delta equally.

Also from chaos.go, EvolutionAdapter (AdaptPopulation(spawn, retire)) is the runtime-adaptation surface: Evolution decides population changes; the Kernel enforces them through the existing spawn/retire primitives ("Agent decides; Kernel enforces"). Its tasks field comment says it is intentionally unused for now — scheduling-policy changes will plug into taskfabric.Schedule in a future iteration. So: population adaptation is implemented; scheduler-policy adaptation is declared but not yet wired.

flowchart LR
    subgraph arena["Arena drill flow (ares_arena)"]
        IN2["Injector: kill / pause / remove / slow / timeout / corrupt …"]
        SV["Service.Execute(action)"]
        M["Stats + MetricsCollector"]
        EV["EventStore arena.* events + failure evidence"]
        FB["FlightBridge → flight recorder"]
        EB["EvolutionBridge → UngatedPatcher (TBD back-fill)"]
        IN2 --> SV --> M
        SV --> EV --> FB
        SV --> EB
    end
    EV --> SSE["/arena/stream SSE"]

8. CLI quick reference

9. Closing

After walking through all of this, I want you to remember not a score but the two boundaries written into the comments:

1. In internal/aresrecovery: "Chaos breaks, Recovery fixes." Breaking and fixing are two independent responsibilities, welded together by an explicit VerifyRecovery() call. In production, live chaos is locked behind six guardrails, and the default posture never touches production agents (shadow sandbox).
2. In internal/runtime/arena: "It does NOT implement recovery." It only injects faults; recovery belongs to ares_runtime's existing mechanisms, and this is an isolated drill field.

No evidence in this repository supports numbers such as "Score 100.0 (A+)", "revived in 1.4s", or "97.3% recovery rate", so none of them appear above. The Consistency dimension defaults to a heuristic, parallel_actions / depends_on are not yet implemented, and the EvolutionBridge back-fill effect is marked TBD.

Making "deliberately break it, then watch it rise again" repeatable, verifiable, and — by default — isolated from production is the genuinely hard part of this.