A close look at the three-tier scoring pipeline in internal/runtime/ares_evolution/scoring: TieredScorer (Cache → LLM → Heuristic), Budget (CAS atomic budget), ScoreCache (LRU + generation-based expiry), plus the optional MemoryAwareScorer overlay. Code samples come from the real source; every constant and default matches the repo.

1. Architecture overview

TieredScorer decides a strategy's fitness score on the principle "use the cheapest tier that gives a good-enough score, and only call the expensive LLM when necessary." File: internal/runtime/ares_evolution/scoring/tiered_scorer.go.

flowchart LR
    S[Score(strategy)] --> H1[StrategyHash]
    H1 --> C{1. cache hit?}
    C -->|hit| R1[return cached value + CacheHit]
    C -->|miss| B{2. llm present and budget allows?}
    B -->|yes| L[tryLLMScore w/ panic recovery]
    L -->|ok| R2[LLM score + write cache]
    L -->|fail| H[3. heuristic fallback]
    B -->|no| H
    H --> R3[heuristic score + write cache]

2. Tier definitions and execution order

2.1 Enum

const (
    TierCache     Tier = iota + 1  // 1
    TierHeuristic                   // 2
    TierLLM                         // 3
)

There's a real mismatch between naming and execution order: the numeric enum order is Cache(1) → Heuristic(2) → LLM(3), but inside Score() the LLM actually runs before the heuristic:

func (ts *TieredScorer) Score(ctx context.Context, s *mutation.Strategy) (float64, Tier, error) {
    hash, err := StrategyHash(s)          // FNV-1a 64-bit hash
    if err != nil { ... }

    // Tier 1: cache hit returns at zero cost
    if entry, ok := ts.cache.Get(hash); ok {
        ts.budget.RecordCacheHit()
        ts.cacheHits.Add(1)
        ts.totalScored.Add(1)
        return entry.Score, TierCache, nil
    }

    // Tier 2: LLM enabled and within budget
    if ts.llm != nil && ts.budget.TryRecordLLMCall() {
        score, scored := ts.tryLLMScore(ctx, s, hash)
        if scored {
            return score, TierLLM, nil
        }
        // LLM failed/panicked -> fall to heuristic
    }

    // Tier 3: heuristic fallback (always available)
    score := ts.heuristic(s)
    entry := MakeEntry(hash, score, ScorerTypeHeuristic, 1, 0.5)
    ts.cache.Put(hash, entry)
    ts.heuristicCalls.Add(1)
    ts.totalScored.Add(1)
    return score, TierHeuristic, nil
}

Rationale: LLM is expensive but more accurate than a heuristic, so the pipeline tries the more accurate tier first and degrades only when the budget is exhausted or the LLM fails.

2.2 LLM panic-safe recovery

func (ts *TieredScorer) tryLLMScore(ctx context.Context, s *mutation.Strategy, hash uint64) (float64, bool) {
    var score float64
    var success bool
    func() {
        defer func() {
            if r := recover(); r != nil {
                log.Warn("tiered_scorer: LLM scorer panicked", "hash", hash, "recovery", r)
                ts.budget.RecordFallback()
                ts.fallbacks.Add(1)
                success = false
            }
        }()
        score = ts.llm(s)
        success = true
    }()
    if !success {
        return 0, false
    }
    entry := MakeEntry(hash, score, ScorerTypeLLM, 1, 1.0)
    ts.cache.Put(hash, entry)
    ts.llmCalls.Add(1)
    ts.totalScored.Add(1)
    return score, true
}

The LLM call is wrapped in a closure with defer recover(): even if a scorer panics, the evolution loop doesn't crash — it degrades gracefully to the heuristic.

Stats() exposes cache_hits / llm_calls / heuristic_calls / fallbacks / total_scored; ResetForGeneration() resets the budget and counters at the start of each generation.

3. Budget: CAS atomic budget

internal/runtime/ares_evolution/scoring/budget.go. Caps LLM calls per generation.

type Budget struct {
    MaxLLMCalls   int64        // immutable
    UsedLLMCalls  atomic.Int64
    CacheHits     atomic.Int64
    FallbackCount atomic.Int64
}

The core CAS spin:

func (b *Budget) TryRecordLLMCall() bool {
    used := b.UsedLLMCalls.Load()
    for used < b.MaxLLMCalls {
        if b.UsedLLMCalls.CompareAndSwap(used, used+1) {
            return true
        }
        used = b.UsedLLMCalls.Load()
    }
    return false
}

Lock-free concurrency: read used → if within limit, try a CAS increment → on contention retry → success returns true → at capacity returns false. Also CanCallLLM(), RecordCacheHit() (doesn't consume budget), RecordFallback(); Reset() zeroes the counters.

4. ScoreCache: LRU + generation-based expiry

internal/runtime/ares_evolution/scoring/cache.go. An LRU keyed by strategy hash, built on container/list.

type CacheEntry struct {
    Hash        uint64
    Score       float64
    ScorerType  string  // "llm" / "heuristic"
    Timestamp   int64   // Unix nanos
    SampleCount int
    Confidence  float64
}

type ScoreCache struct {
    mu          sync.RWMutex
    entries     map[uint64]*list.Element
    lru         list.List   // front = most recently used
    maxSize     int         // 0 = unlimited
    maxCacheAge int         // 0 = unlimited (no expiry)
    generation  uint64
    hits, misses, evictions int64
}

Two real design points:

  • Get() takes the write lock — it calls MoveToFront, mutating LRU order; the RWMutex write path is simpler.
  • Generation expiry is opt-in: maxCacheAge defaults to 0 (unlimited — entries don't expire on their own). To enable it, call SetMaxCacheAge(n) and advance the counter via NewGeneration(); entries older than n generations are treated as misses and evicted.

Put() evicts the least-recently-used entry from the tail when at capacity; an existing hash is updated in place with its generation refreshed.

5. MemoryAwareScorer: an experience-driven overlay

internal/runtime/ares_evolution/scoring/memory_aware_scorer.go layers an experience-memory dimension on top of TieredScorer. Defaults are real constants:

func DefaultMemoryAwareScoringConfig() MemoryAwareScoringConfig {
    return MemoryAwareScoringConfig{
        Enabled:               false,  // off by default
        MemoryWeight:          0.2,
        CostWeight:            0.1,
        LatencyWeight:         0.05,
        RegressionWeight:      0.1,
        MinEvidenceBonus:      0.0,
        MaxEvidenceBonus:      20.0,
        ExperienceLookupLimit: 10,
        SuccessRateBonusScale: 10.0,   // bonus = success_rate * confidence * scale
        LatencyPenaltyScale:   1.0,     // penalty = (latency_p50 / 10000) * scale
        ErrorRatePenaltyScale: 1.0,
    }
}

Its intent is quality + memory_evidence_bonus − cost − latency − regression, with the success-bonus and latency/error-penalty formulas documented on the config fields. The ScoreDetail struct decomposes each component for reporting.

6. Summary

The "cost–quality trade-off" lands in real code as a three-tier pipeline:

  1. Cache absorbs redundant scoring via O(1) hash lookup; Get/Put maintain the LRU.
  2. Budget uses a CAS spin to cap LLM calls per generation under concurrency.
  3. heuristic is the always-on fallback so a score always exists.
  4. MemoryAwareScorer is a switchable evidence overlay (off by default with Enabled:false).

Anywhere a score costs LLM money, this layering is the first and most important cost gate on the scoring side.