aresrecovery

The internal/aresrecovery package (package aresrecovery) implements the ARES Kernel Recovery subsystem (P5 of docs/zh/architecture/ares-runtime.md): the independent responsibility of keeping durable tasks alive across agent deaths.

Design invariants (ares-runtime.md §13):

• Agent is disposable, Task is durable — Agent death ≠ Task death.
• Recovery is a distinct responsibility from Chaos (failure injection): Chaos breaks things on purpose; Recovery proves the Runtime survives.

The Recovery subsystem orchestrates three failure paths:

1. Lease expiry → requeue: a dead agent’s lease expires; the task returns to READY and another agent can acquire it (Task Fabric).
2. Checkpoint recovery: a task’s preserved checkpoint is resumed by a new agent (Task Fabric + Agent Fabric CognitiveState).
3. Agent restart: a crashed agent is replaced by a new one that picks up the dead agent’s cognitive checkpoint (Agent Fabric Recover).

Responsibility

• Wire the Task Fabric (durable tasks + lease expiry + checkpoints) to the Agent Fabric (disposable agents + cognitive state) so that an agent death is followed by task requeue + checkpoint resume + agent replacement.
• Sweep the Task Fabric for expired leases via RequeueExpiredLeases and requeue the corresponding tasks to READY (failure path 1).
• Resume a task’s preserved checkpoint with a new agent via RecoverTaskCheckpoint: spawns or reuses a replacement agent, acquires the task for it, and installs the checkpoint as the new agent’s cognitive state (failure path 2).
• Replace a crashed agent with a new one via RestartAgent: spawns a replacement with the dead agent’s capabilities and installs the preserved cognitive state, bounded by the restart budget (failure path 3).
• Enforce the restart budget (RestartPolicy) with exponential backoff: MaxRestarts attempts, Backoff initial delay doubled each attempt (capped at MaxBackoff).
• Provide the EvolutionAwareSpawner integration point (v0.4.0 M2-1): when wired, every replacement spawn is routed through the spawner so the evolution policy (Enabled / MaxConcurrent / PreferredCapabilities) shapes restarts and checkpoint recovery — “Evolution decides; Kernel enforces”. Recovery spawns bypass the population cap (SpawnForRecovery) so a self-healing spawn is never blocked by MaxConcurrent.
• Expose RecoverFromAgentDeath as the full recovery chain: sweep expired leases → requeue tasks → resume each requeued task’s checkpoint with a fresh replacement agent (P5 acceptance path).

Architecture

flowchart TD
    AD["Agent Death"] --> LE["Task Fabric<br/>Lease expiry"]
    LE --> RQ["RequeueExpiredLeases<br/>Task → READY"]
    RQ --> ACQ["Acquire(task, agent)<br/>New lease epoch"]
    RD["Agent Death"] --> CP["Checkpoint preserved"]
    CP --> RC["RecoverTaskCheckpoint<br/>Spawn replacement → Acquire → Install checkpoint"]
    RC --> CS["Agent CognitiveState = checkpoint"]
    RD["Agent Death"] --> CG["CognitiveState preserved"]
    CG --> RA["RestartAgent<br/>Check restart budget → Spawn replacement → Recover cognitive"]
    RA --> RB{"RestartCount &lt; MaxRestarts?"}
    RB -- yes --> SP["spawnAgent<br/>EvolutionAwareSpawner.SpawnForRecovery"]
    RB -- no --> EX["ErrRecoveryExhausted"]
    SP --> RP["Replacement Agent<br/>with dead agent's cognitive state"]
    FC["RecoverFromAgentDeath<br/>Full chain"] --> RQ
    FC --> RC
    CH["Chaos<br/>Failure injection"] --> AD
    EV["Evolution<br/>M2-1 policy"] --> SP

External interfaces

package aresrecovery

// --- Recovery ---

type Recovery struct {
    // tasks *taskfabric.Fabric
    // agents *agentfabric.Fabric
    // spawner *EvolutionAwareSpawner
    // policy RestartPolicy
    // mu sync.Mutex
    // restarts map[string]int
    // now func() time.Time
}

func New(tasks *taskfabric.Fabric, agents *agentfabric.Fabric, policy RestartPolicy) *Recovery
func (r *Recovery) WithClock(now func() time.Time) *Recovery
func (r *Recovery) WithSpawner(s *EvolutionAwareSpawner) *Recovery

func (r *Recovery) RequeueExpiredLeases() []string
func (r *Recovery) RecoverTaskCheckpoint(ctx context.Context, taskID, replacementID string) (string, uint64, error)
func (r *Recovery) RestartAgent(ctx context.Context, deadAgentID string, cognitive agentfabric.CognitiveState, capabilities []string) (*agentfabric.Agent, error)
func (r *Recovery) RestartCount(agentID string) int
func (r *Recovery) RecoverFromAgentDeath(ctx context.Context) int

// --- RestartPolicy ---

type RestartPolicy struct {
    MaxRestarts int
    Backoff     time.Duration
    MaxBackoff  time.Duration
}

func DefaultRestartPolicy() RestartPolicy

// --- EvolutionAwareSpawner (v0.4.0 M2-1) ---

type SpawnPolicy struct {
    Enabled               bool
    MaxConcurrent         int
    PreferredCapabilities []string
}
type SpawnPolicySource interface {
    ActiveSpawnPolicy(ctx context.Context) (SpawnPolicy, error)
}
type EvolutionAwareSpawner struct {
    // agents *agentfabric.Fabric
    // source SpawnPolicySource
}
func NewEvolutionAwareSpawner(agents *agentfabric.Fabric, source SpawnPolicySource) *EvolutionAwareSpawner
func (s *EvolutionAwareSpawner) Spawn(ctx context.Context, spec agentfabric.SpawnSpec) (*agentfabric.Agent, error)
func (s *EvolutionAwareSpawner) SpawnForRecovery(ctx context.Context, spec agentfabric.SpawnSpec) (*agentfabric.Agent, error)

// --- Sentinel errors ---

var ErrRecoveryExhausted = errors.New("aresrecovery: restart budget exhausted")
var ErrSpawnDisabled = errors.New("aresrecovery: evolution disabled spawning")
var ErrSpawnLimitReached = errors.New("aresrecovery: evolution spawn limit reached")

Key types and methods

Module collaboration

• aresrecovery -> internal/fabric/task (via tasks *taskfabric.Fabric; import path internal/fabric/task): sweeps expired leases via CheckExpiredLeases, acquires tasks for replacement agents, reads preserved checkpoints. RequeueExpiredLeases returns the requeued task IDs ([]string).
• aresrecovery -> internal/fabric/agent (via agents *agentfabric.Fabric; import path internal/fabric/agent): spawns replacement agents, installs cognitive state via SetCognitiveState and Recover.
• aresrecovery -> internal/ares_evolution (via SpawnPolicySource, defined in aresrecovery and implemented by the evolution system — aresrecovery never imports the evolution package): evolution policy (SpawnPolicy) shapes restart and recovery spawns; SpawnForRecovery bypasses only the population quota.
• aresrecovery -> internal/kernel (System Runtime control plane): the Recovery subsystem is adopted as a component (Orchestrator.Adopt) and stopped by the Orchestrator; wiring in cmd/ares/kernel.go and internal/ares_bootstrap/system_runtime_wiring.go.

Extension points

1. Wire an evolution-aware spawn gate via Recovery.WithSpawner(spawner) so the evolution policy (Enabled / MaxConcurrent / PreferredCapabilities) shapes restarts and checkpoint recovery. Recovery spawns always use SpawnForRecovery — they bypass the population cap so a self-healing spawn is never blocked by MaxConcurrent.
2. Customise the restart policy by passing a non-default RestartPolicy to New; adjust MaxRestarts, Backoff, and MaxBackoff to match operational requirements.
3. Inject a deterministic clock via Recovery.WithClock(now) for hermetic tests of restart ordering and backoff timing.
4. Verify the full recovery chain by calling RecoverFromAgentDeath — it sweeps expired leases, requeues tasks, and resumes each checkpoint with a fresh replacement agent (P5 acceptance path).
5. Instrument recovery observability by reading RestartCount(agentID) to track per-agent restart attempts and detect exhausted budgets.

Bilingual status

This page is the English reference. A Chinese translation with identical structure and technical content is published as aresrecovery.zh.md. All code identifiers, type names, and signatures are kept in English in both files; only the prose differs.

Maturity

Production. The package is covered by unit tests and implements the Kernel Recovery subsystem (P5) with restart budget enforcement, evolution-aware spawn gates, and the full RecoverFromAgentDeath chain. It integrates with the Task Fabric, Agent Fabric, and Evolution subsystems, and exposes no experimental markers.

Strategy score write-back (zero-LLM feedback loop)

ScoredFeedbackAdapter.Apply computes an aggregate score via DeterministicScorer from execution attribution and writes it to the active strategy through StrategyScoreAdapter.WriteActiveScore → StrategyStore. The write-back requires an active strategy — bootstrap seeds bootstrap-root into an empty store; without that seed every write fails with “no active strategy” and the GA accumulates no fitness. The adapter never changes which strategy is active: it updates the Score field in place.