Timwood0x10
Tim Wood — memory safety, static analysis, and AI infrastructure.
Zola
2026-05-27T00:00:00+00:00
https://Timwood0x10.github.io/myblog/atom.xml
garbage-code-hunter Deep-Dive Series: Multi-Language Code Quality Analysis in Rust
2026-05-27T00:00:00+00:00
2026-05-27T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/
<h1 id="building-garbage-code-hunter-a-deep-dive-into-multi-language-code-quality-analysis-in-rust">Building garbage-code-hunter: A Deep Dive into Multi-Language Code Quality Analysis in Rust</h1>
<blockquote>
<p><strong>Series: In-Depth Technical Breakdown of garbage-code-hunter</strong></p>
</blockquote>
<h2 id="what-is-this">What Is This?</h2>
<p>garbage-code-hunter is not another bug finder. It is not a security scanner. It does not care about CVEs or null pointer dereferences.</p>
<p>It cares about <strong>taste</strong>.</p>
<p>Does your codebase have 47 <code>TODO</code> comments dating back to 2019? Are there functions spanning 200 lines with 8 levels of nesting? Is every third file a copy-paste variant of the first? These are the things that make a codebase smell — not broken, but <em>wrong</em>. And most tools either ignore them entirely or drown you in false positives.</p>
<p>garbage-code-hunter is a Rust CLI that analyzes code across <strong>11 languages</strong> — Rust, Go, Python, Java, Ruby, C, C++, TypeScript, JavaScript, Swift, and Zig — and tells you, with brutal honesty and occasional humor, how bad your code really is.</p>
<h2 id="why-this-series">Why This Series?</h2>
<p>This project makes several non-obvious design decisions:</p>
<ul>
<li><strong>Why tree-sitter instead of per-language linters?</strong> Because running 11 separate tools is not a strategy — it is a coping mechanism.</li>
<li><strong>Why a StyleIR instead of direct AST rules?</strong> Because coupling every detector to every language's AST is a maintenance nightmare that scales as O(detectors x languages).</li>
<li><strong>Why "fewer rules, stronger signals"?</strong> Because 200 rules with 60% false positives are worse than 10 rules with 90% precision.</li>
<li><strong>Why logarithmic scoring?</strong> Because one god function is bad, but ten god functions is not ten times worse — it is a systemic failure.</li>
<li><strong>Why personality types?</strong> Because making code analysis fun is the only way to make developers actually read the report.</li>
</ul>
<p>Each article in this series takes one of these decisions and breaks it down: the problem it solves, the alternatives considered, and the implementation details in the actual source code.</p>
<h2 id="series-map">Series Map</h2>
<table><thead><tr><th>#</th><th>Article</th><th>Core Question</th></tr></thead><tbody>
<tr><td>01</td><td><a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/./01-why-multi-language-sucks.md">Why Multi-Language Code Quality Analysis Is Hard</a></td><td>What happens when you need to analyze 11 languages with one tool?</td></tr>
<tr><td>02</td><td><a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/./02-architecture-overview.md">Architecture Overview</a></td><td>How do you structure a code analyzer that is both fast and extensible?</td></tr>
<tr><td>03</td><td><a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/./03-treesitter-and-language-adapter.md">Tree-sitter and the LanguageAdapter Pattern</a></td><td>How do you avoid writing 11 copies of the same detection logic?</td></tr>
<tr><td>04</td><td><a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/./04-style-ir-the-neutral-layer.md">StyleIR: The Language-Neutral Intermediate Representation</a></td><td>How do you decouple detection from language-specific AST details?</td></tr>
<tr><td>05</td><td><a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/./05-signal-detection-system.md">The Signal Detection System</a></td><td>Why fewer, stronger detectors beat many weak ones?</td></tr>
<tr><td>06</td><td><a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/./06-duplication-algorithms.md">Duplication Detection Algorithms</a></td><td>How do you find copy-paste code across an entire codebase?</td></tr>
<tr><td>07</td><td><a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/./07-scoring-model.md">The Scoring Model</a></td><td>How do you turn qualitative signals into a single number?</td></tr>
<tr><td>08</td><td><a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/./08-fun-side-roasts-and-personality.md">The Fun Side: Roasts, Personality, and the Tool Belt</a></td><td>Why should code analysis be boring?</td></tr>
</tbody></table>
<h2 id="reading-order">Reading Order</h2>
<p>The articles are ordered by dependency. Each one builds on concepts introduced in the previous:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>01 Problem Space</span></span>
<span class="giallo-l"><span> └─► 02 Architecture</span></span>
<span class="giallo-l"><span> └─► 03 Parsing Layer (tree-sitter + adapters)</span></span>
<span class="giallo-l"><span> └─► 04 Intermediate Representation (StyleIR)</span></span>
<span class="giallo-l"><span> └─► 05 Detection Layer (SignalDetector)</span></span>
<span class="giallo-l"><span> ├─► 06 Duplication (algorithms)</span></span>
<span class="giallo-l"><span> └─► 07 Scoring (math)</span></span>
<span class="giallo-l"><span> └─► 08 Fun Side (personality + tools)</span></span></code></pre>
<p>You can read any single article standalone — each one frames its own problem — but the full picture emerges from reading them in sequence.</p>
<h2 id="get-the-code">Get the Code</h2>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>cargo install garbage-code-hunter</span></span>
<span class="giallo-l"><span>garbage-code-hunter analyze ./your-project</span></span></code></pre>
<p>Or clone and build from source:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>git clone https://github.com/anthropics/garbage-code-hunter.git</span></span>
<span class="giallo-l"><span>cd garbage-code-hunter</span></span>
<span class="giallo-l"><span>cargo build --release</span></span></code></pre>
<p>Let's start with the fundamental question: <a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/00-series-intro/./01-why-multi-language-sucks.md">Why is multi-language code quality analysis so hard?</a></p>
Why Multi-Language Code Quality Analysis Is Hard
2026-05-27T00:00:00+00:00
2026-05-27T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/01-why-multi-language-sucks/
<h1 id="why-multi-language-code-quality-analysis-is-hard">Why Multi-Language Code Quality Analysis Is Hard</h1>
<blockquote>
<p><strong>Problem</strong>: You have a monorepo with Rust services, a Go gateway, Python ML pipelines, a TypeScript frontend, and Java Android code. Your CTO wants "one dashboard for code quality." What do you do?</p>
</blockquote>
<h2 id="the-landscape-of-pain">The Landscape of Pain</h2>
<p>Here is what the real world looks like:</p>
<table><thead><tr><th>Language</th><th>Linter</th><th>Config Format</th><th>AST Library</th><th>Rule Language</th></tr></thead><tbody>
<tr><td>Rust</td><td>Clippy</td><td>TOML</td><td><code>syn</code></td><td>Rust macros</td></tr>
<tr><td>Go</td><td><code>golangci-lint</code></td><td>YAML</td><td><code>go/ast</code></td><td>Go plugins</td></tr>
<tr><td>Python</td><td>Ruff / Pylint</td><td>TOML / INI</td><td><code>libcst</code> / <code>ast</code></td><td>Python</td></tr>
<tr><td>JavaScript</td><td>ESLint</td><td>JS/JSON/YAML</td><td><code>acorn</code> / <code>espree</code></td><td>JavaScript</td></tr>
<tr><td>TypeScript</td><td>ESLint + TSC</td><td>JS/JSON/YAML</td><td><code>typescript</code></td><td>JavaScript</td></tr>
<tr><td>Java</td><td>SpotBugs / PMD</td><td>XML</td><td>Eclipse JDT</td><td>Java / XPath</td></tr>
<tr><td>C</td><td>cppcheck</td><td>CLI flags</td><td>Custom</td><td>Custom</td></tr>
<tr><td>C++</td><td>clang-tidy</td><td>YAML</td><td>Clang AST</td><td>Custom</td></tr>
<tr><td>Ruby</td><td>RuboCop</td><td>YAML</td><td><code>parser</code> gem</td><td>Ruby</td></tr>
<tr><td>Swift</td><td>SwiftLint</td><td>YAML</td><td>SourceKit</td><td>Swift</td></tr>
<tr><td>Zig</td><td><code>zig fmt</code></td><td>—</td><td>Self-hosted</td><td>—</td></tr>
</tbody></table>
<p>Each tool has its own:</p>
<ul>
<li>Installation method</li>
<li>Configuration format</li>
<li>Rule definition language</li>
<li>AST representation</li>
<li>CI integration pattern</li>
<li>Output format</li>
</ul>
<p>If you want to analyze all 11 languages, you are not building a tool. You are building an <strong>integration layer</strong> over 11 tools, each with its own release cycle, breaking changes, and opinionated defaults.</p>
<h2 id="the-three-approaches-and-why-they-fail">The Three Approaches (and Why They Fail)</h2>
<h3 id="approach-1-run-all-linters-aggregate-results">Approach 1: Run All Linters, Aggregate Results</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">cargo</span><span style="color: #A6E3A1;"> clippy --message-format=json</span><span style="color: #94E2D5;"> ></span><span style="color: #A6E3A1;"> rust.json</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">golangci-lint</span><span style="color: #A6E3A1;"> run --out-format=json</span><span style="color: #94E2D5;"> ></span><span style="color: #A6E3A1;"> go.json</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">ruff</span><span style="color: #A6E3A1;"> check --output-format=json</span><span style="color: #94E2D5;"> ></span><span style="color: #A6E3A1;"> python.json</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">eslint</span><span style="color: #A6E3A1;"> --format=json</span><span style="color: #94E2D5;"> ></span><span style="color: #A6E3A1;"> js.json</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># ... 7 more</span></span></code></pre>
<p><strong>Problems</strong>:</p>
<ul>
<li>11 tools to install, configure, and keep updated</li>
<li>Incompatible output schemas — one tool's "warning" is another's "info"</li>
<li>No cross-language signals (e.g., "this Go function and this Rust function are identical")</li>
<li>Each tool has different opinions about what constitutes a "violation"</li>
<li>CI setup becomes a YAML novel</li>
</ul>
<h3 id="approach-2-write-a-custom-parser-per-language">Approach 2: Write a Custom Parser per Language</h3>
<p>Roll your own AST for each language. Full control, unified interface.</p>
<p><strong>Problems</strong>:</p>
<ul>
<li>Each language grammar takes months to implement correctly</li>
<li>Grammars evolve — you are now maintaining 11 parsers</li>
<li>Edge cases in parsing (string interpolation, macros, preprocessor directives) will consume your life</li>
<li>You are essentially rebuilding compiler frontends for fun</li>
</ul>
<h3 id="approach-3-use-a-single-parser-framework">Approach 3: Use a Single Parser Framework</h3>
<p>This is what garbage-code-hunter does. But which framework?</p>
<h2 id="why-tree-sitter">Why Tree-sitter?</h2>
<p>Tree-sitter is an incremental parsing library designed for syntax highlighting in editors. It has compiled grammars for 100+ languages. But more importantly:</p>
<ol>
<li>
<p><strong>One API, many languages.</strong> <code>tree_sitter_rust()</code>, <code>tree_sitter_go()</code>, <code>tree_sitter_python()</code> all return the same <code>Language</code> type with the same query API.</p>
</li>
<li>
<p><strong>Query-based extraction.</strong> Instead of walking the AST manually, you write declarative patterns:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="scheme"><span class="giallo-l"><span style="color: #9399B2;">(</span><span>call_expression</span></span>
<span class="giallo-l"><span> function: </span><span style="color: #9399B2;">(</span><span>identifier</span><span style="color: #9399B2;">)</span><span> @fn</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> (</span><span>#match? @fn </span><span style="color: #A6E3A1;">"^(panic|unwrap|expect)$"</span><span style="color: #9399B2;">))</span></span></code></pre>
<p>This is the same query language for every language.</p>
</li>
<li>
<p><strong>Speed.</strong> Tree-sitter parses most files in under 10ms. It is designed for real-time editing — batch analysis is trivially fast.</p>
</li>
<li>
<p><strong>Incremental.</strong> If you need to re-parse after an edit, only the changed region is re-parsed. This matters for LSP integration.</p>
</li>
</ol>
<p>But tree-sitter alone is not enough. It gives you syntax, not semantics. You still need to answer questions like:</p>
<ul>
<li>"Is this <code>unwrap()</code> call in a test file?"</li>
<li>"Is this function more than 50 lines?"</li>
<li>"Are these two code blocks duplicated?"</li>
</ul>
<p>This is where the architecture gets interesting.</p>
<h2 id="the-real-challenge-language-specific-vs-language-neutral">The Real Challenge: Language-Specific vs. Language-Neutral</h2>
<p>Consider the simple question: "Is this code using debug print statements?"</p>
<table><thead><tr><th>Language</th><th>Debug Patterns</th></tr></thead><tbody>
<tr><td>Rust</td><td><code>println!</code>, <code>dbg!</code>, <code>eprintln!</code></td></tr>
<tr><td>Go</td><td><code>fmt.Println</code>, <code>log.Println</code>, <code>println</code></td></tr>
<tr><td>Python</td><td><code>print()</code>, <code>pprint()</code></td></tr>
<tr><td>Java</td><td><code>System.out.println</code>, <code>System.err.println</code></td></tr>
<tr><td>JavaScript</td><td><code>console.log</code>, <code>console.warn</code>, <code>console.error</code></td></tr>
<tr><td>Ruby</td><td><code>puts</code>, <code>p </code>, <code>pp </code></td></tr>
<tr><td>Swift</td><td><code>print()</code>, <code>debugPrint()</code>, <code>dump()</code></td></tr>
<tr><td>Zig</td><td><code>std.debug.print</code></td></tr>
<tr><td>C</td><td><code>printf</code>, <code>fprintf(stderr, ...)</code></td></tr>
<tr><td>C++</td><td><code>std::cout</code>, <code>std::cerr</code>, <code>printf</code></td></tr>
</tbody></table>
<p>Each language has its own set of patterns. But the <strong>concept</strong> — "debug output that should not be in production code" — is language-neutral.</p>
<p>This is the fundamental tension:</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph LR
A[Language-Specific<br/>AST Details] --> B[???]
B --> C[Language-Neutral<br/>Quality Signals]
</div>
<p>How do you bridge this gap? Two options:</p>
<p><strong>Option A: Each detector handles all languages.</strong> Your <code>DebugPrintDetector</code> has a match statement for 11 languages. When you add language #12, you update every detector. This is O(detectors x languages).</p>
<p><strong>Option B: Each language adapter produces a common output.</strong> Your <code>RustAdapter</code> knows that <code>println!</code> is a debug call. Your <code>GoAdapter</code> knows that <code>fmt.Println</code> is a debug call. Both emit the same counter: <code>debug_call_count</code>. Detectors never see the language. This is O(detectors + languages).</p>
<p>garbage-code-hunter chose Option B.</p>
<h2 id="the-architecture-that-emerges">The Architecture That Emerges</h2>
<p>Once you commit to Option B, the architecture becomes clear:</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph TB
subgraph "Per-Language (O(languages))"
RA[RustAdapter] --> |debug_call_count| IR
GA[GoAdapter] --> |debug_call_count| IR
PA[PythonAdapter] --> |debug_call_count| IR
VA[...11 adapters] --> |debug_call_count| IR
end
subgraph "Language-Neutral (O(1))"
IR[StyleIR] --> DD[DebugPrintDetector]
IR --> ND[NamingDetector]
IR --> PD[PanicDetector]
IR --> SD[...10 detectors]
end
</div>
<p>The key insight: <strong>adapters are the complexity sink.</strong> They absorb all language-specific knowledge so that detectors can be simple.</p>
<p>This is the pattern that the rest of this series explores in depth:</p>
<ul>
<li><strong>Article 03</strong> dives into the <code>LanguageAdapter</code> trait and how tree-sitter queries are batched</li>
<li><strong>Article 04</strong> explains <code>StyleIR</code> — the language-neutral fact layer</li>
<li><strong>Article 05</strong> shows how <code>SignalDetector</code> implementations consume StyleIR without knowing the language</li>
</ul>
<h2 id="what-this-buys-you">What This Buys You</h2>
<p>The O(detectors + languages) scaling is not just theoretical. When garbage-code-hunter added Zig support, the changes were:</p>
<ol>
<li>Add <code>ZigAdapter</code> implementing <code>LanguageAdapter</code> (~200 lines)</li>
<li>Add <code>Language::Zig</code> variant and extension mapping</li>
<li>Add <code>tree_sitter_zig</code> to dependencies</li>
</ol>
<p>Zero detectors were modified. Zero scoring logic changed. Zero configuration updates needed.</p>
<p>When a new detector is added (say, <code>MagicNumberDetector</code>), it works across all 11 languages immediately — because it reads <code>StyleIr.magic_number_count</code>, which every adapter already computes.</p>
<p>This is the payoff of the adapter pattern: <strong>decoupling that actually scales.</strong></p>
<hr />
<p><em>Next: <a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/01-why-multi-language-sucks/./02-architecture-overview.md">Architecture Overview</a> — How the four-phase pipeline works, and why the module boundaries are drawn where they are.</em></p>
Architecture Overview: How to Structure a Multi-Language Code Analyzer
2026-05-27T00:00:00+00:00
2026-05-27T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/02-architecture-overview/
<h1 id="architecture-overview">Architecture Overview</h1>
<blockquote>
<p><strong>Question</strong>: You have a code analyzer that needs to parse files, detect issues, compute scores, and generate reports. It must support 11 languages, run in CI, and finish in seconds. How do you structure it?</p>
</blockquote>
<h2 id="the-four-phase-pipeline">The Four-Phase Pipeline</h2>
<p>Most code analysis tools follow a linear pipeline: parse, analyze, report. garbage-code-hunter adds a twist — it separates <strong>duplication detection</strong> from <strong>signal detection</strong> because they operate at different granularities:</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph LR
P1[Phase 1<br/>Parse] --> P2[Phase 2<br/>Cross-File<br/>Duplication]
P2 --> P3[Phase 3<br/>Intra-File<br/>Duplication]
P3 --> P4[Phase 4<br/>Signal<br/>Detection]
</div>
<p>Here is how it works in the actual source code (<code>src/analyzer.rs:174-275</code>):</p>
<h3 id="phase-1-parse">Phase 1: Parse</h3>
<p>Walk the directory tree, filter by supported extensions, parse each file with tree-sitter:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/analyzer.rs:183-200</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span> file_path</span><span style="color: #CBA6F7;"> in</span><span style="color: #94E2D5;"> &</span><span>files</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span style="color: #F38BA8;"> Self</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">is_generated_file</span><span style="color: #9399B2;">(</span><span>file_path</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> continue</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> content</span><span style="color: #94E2D5;"> =</span><span style="color: #CBA6F7;"> match</span><span style="color: #F9E2AF;"> fs</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">read_to_string</span><span style="color: #9399B2;">(</span><span>file_path</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Ok</span><span style="color: #9399B2;">(</span><span>c</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =></span><span> c</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Err</span><span style="color: #9399B2;">(</span><span>_</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =></span><span style="color: #CBA6F7;"> continue</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> };</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> lang</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Language</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">from_path</span><span style="color: #9399B2;">(</span><span>file_path</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> lang</span><span style="color: #94E2D5;"> ==</span><span style="color: #F9E2AF;font-style: italic;"> Language</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Unknown</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> continue</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>parsed</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>ts_engine</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">parse_file</span><span style="color: #9399B2;">(</span><span>file_path</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>content</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span> parsed_files</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">((</span><span>parsed</span><span style="color: #9399B2;">,</span><span> file_path</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">(),</span><span> is_test_file</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Key design decision: <strong>parsed files are cached</strong> in a <code>Vec<(ParsedFile, PathBuf, bool)></code> so phases 2-4 do not re-parse. The <code>bool</code> tracks whether the file is a test file.</p>
<h3 id="phase-2-cross-file-duplication">Phase 2: Cross-File Duplication</h3>
<p>Fingerprint every function across the entire codebase. Find exact duplicates by hash grouping, near-duplicates by Jaccard similarity:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/analyzer.rs:202-212</span></span>
<span class="giallo-l"><span style="color: #94E2D5;">*</span><span style="color: #F38BA8;">self</span><span style="color: #94E2D5;">.</span><span>cross_detector</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">borrow_mut</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> CrossFileDupDetector</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #9399B2;"> (</span><span>parsed</span><span style="color: #9399B2;">,</span><span> _</span><span style="color: #9399B2;">,</span><span> is_test</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span style="color: #94E2D5;"> &</span><span>parsed_files</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>cross_detector</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">borrow_mut</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">process_file</span><span style="color: #9399B2;">(</span><span>parsed</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"><span>issues</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">extend</span><span style="color: #9399B2;">(</span><span style="color: #F38BA8;">self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">cross_detector</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">borrow</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">find_duplicates</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span>issues</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">extend</span><span style="color: #9399B2;">(</span><span style="color: #F38BA8;">self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">cross_detector</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">borrow</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">find_near_duplicates</span><span style="color: #9399B2;">());</span></span></code></pre>
<p>This phase needs to see <strong>all files at once</strong> — which is why it runs before per-file signal detection.</p>
<h3 id="phase-3-intra-file-duplication">Phase 3: Intra-File Duplication</h3>
<p>Find repeated code blocks within a single file using 5-line chunk hashing:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/analyzer.rs:215-220</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #9399B2;"> (</span><span>parsed</span><span style="color: #9399B2;">,</span><span> _</span><span style="color: #9399B2;">,</span><span> is_test</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span style="color: #94E2D5;"> &</span><span>parsed_files</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> issues</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">extend</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">IntraFileDupDetector</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">check</span><span style="color: #9399B2;">(</span><span>parsed</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre><h3 id="phase-4-signal-detection">Phase 4: Signal Detection</h3>
<p>Run all 10 detectors against each file. If a pre-computed <code>StyleIR</code> exists, use it to avoid redundant computation:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/analyzer.rs:239-271</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #9399B2;"> (</span><span>parsed</span><span style="color: #9399B2;">,</span><span> file_path</span><span style="color: #9399B2;">,</span><span> is_test_file</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span style="color: #94E2D5;"> &</span><span>parsed_files</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> ir</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> StyleIr</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">from_parsed</span><span style="color: #9399B2;">(</span><span>parsed</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> detector</span><span style="color: #CBA6F7;"> in</span><span style="color: #94E2D5;"> &</span><span style="color: #F38BA8;">self</span><span style="color: #94E2D5;">.</span><span>detectors </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span style="color: #94E2D5;"> !</span><span>detector</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">supported_languages</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">contains</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>lang</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> continue</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> findings_iter</span><span style="color: #94E2D5;"> =</span><span style="color: #CBA6F7;"> if let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span style="color: #CBA6F7;">ref</span><span> ir</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span> ir</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> detector</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">detect_findings_with_ir</span><span style="color: #9399B2;">(</span><span>ir</span><span style="color: #9399B2;">,</span><span> parsed</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> *</span><span>is_test_file</span><span style="color: #9399B2;">,</span><span> skip_tests_config</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span><span style="color: #CBA6F7;"> else</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> detector</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">detect_findings</span><span style="color: #9399B2;">(</span><span>parsed</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> *</span><span>is_test_file</span><span style="color: #9399B2;">,</span><span> skip_tests_config</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> };</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span style="color: #9399B2;"> (</span><span>signal</span><span style="color: #9399B2;">,</span><span> count</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span> findings_iter</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // Test files get 20% weight</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> count</span><span style="color: #94E2D5;"> =</span><span style="color: #CBA6F7;"> if</span><span style="color: #94E2D5;"> *</span><span>is_test_file</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> (</span><span>count</span><span style="color: #CBA6F7;"> as f64</span><span style="color: #94E2D5;"> *</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">2</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">round</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> as usize</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span><span style="color: #CBA6F7;"> else</span><span style="color: #9399B2;"> {</span><span> count</span><span style="color: #9399B2;"> };</span></span>
<span class="giallo-l"><span> findings</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">StyleFinding</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">for_signal</span><span style="color: #9399B2;">(</span><span>signal</span><span style="color: #9399B2;">,</span><span> count</span><span style="color: #9399B2;">,</span><span> file_path</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">()));</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Notice the test file downweighting: violations in test code count as 20% of their actual value. This prevents test helpers from inflating scores.</p>
<h2 id="module-map">Module Map</h2>
<p>The project is organized as a library + binary:</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph TB
subgraph "Binary (main.rs)"
CLI[CLI Parser<br/>clap] --> SCAN[scan<br/>14 tools in parallel]
CLI --> ANALYZE[analyze<br/>4-phase pipeline]
end
subgraph "Library (lib.rs)"
ANALYZE --> A[analyzer]
A --> TS[treesitter]
A --> LANG[language]
A --> SIG[signals]
A --> IR[style_ir]
A --> DET[detectors]
A --> DUP[duplication]
A --> SCORE[scoring]
A --> FIND[finding]
SCAN --> REPORT[reporter]
REPORT --> CONFIG[config]
REPORT --> I18N[i18n]
REPORT --> LLM[llm]
end
subgraph "Language Adapters"
LANG --> RA[rust.rs]
LANG --> GA[go.rs]
LANG --> PA[python.rs]
LANG --> TA[ts.rs]
LANG --> JA[java.rs]
LANG --> CA[c.rs]
LANG --> CPA[cpp.rs]
LANG --> SA[swift.rs]
LANG --> ZA[zig.rs]
LANG --> RBA[ruby.rs]
LANG --> JSA[js.rs]
end
</div><h3 id="key-modules">Key Modules</h3>
<table><thead><tr><th>Module</th><th>Responsibility</th><th>Key Type</th></tr></thead><tbody>
<tr><td><code>analyzer</code></td><td>Orchestrates the 4-phase pipeline</td><td><code>CodeAnalyzer</code></td></tr>
<tr><td><code>treesitter</code></td><td>Parser management, query execution, duplication</td><td><code>TreeSitterEngine</code></td></tr>
<tr><td><code>language</code></td><td>Language enum + per-language adapters</td><td><code>LanguageAdapter</code> trait</td></tr>
<tr><td><code>style_ir</code></td><td>Language-neutral fact extraction</td><td><code>StyleIr</code></td></tr>
<tr><td><code>signals</code></td><td>Signal definitions, scoring helpers, personality</td><td><code>SignalDetector</code> trait</td></tr>
<tr><td><code>detectors</code></td><td>10 concrete detector implementations</td><td><code>PanicAddictionDetector</code>, etc.</td></tr>
<tr><td><code>scoring</code></td><td>Two-tier log scoring model</td><td><code>CodeScorer</code></td></tr>
<tr><td><code>finding</code></td><td>Structured finding model</td><td><code>StyleFinding</code></td></tr>
<tr><td><code>reporter</code></td><td>Terminal/Markdown/JSON output</td><td><code>Reporter</code></td></tr>
<tr><td><code>config</code></td><td>App + project configuration</td><td><code>AppConfig</code>, <code>ProjectConfig</code></td></tr>
<tr><td><code>llm</code></td><td>LLM-powered roast generation</td><td><code>RoastProvider</code></td></tr>
</tbody></table>
<h2 id="the-dual-command-model">The Dual Command Model</h2>
<p>garbage-code-hunter has two main execution modes:</p>
<h3 id="analyze-deep-code-analysis"><code>analyze</code> — Deep Code Analysis</h3>
<p>Runs the 4-phase pipeline, produces detailed findings with line numbers and evidence. This is the primary mode.</p>
<h3 id="scan-full-project-health-check"><code>scan</code> — Full Project Health Check</h3>
<p>Runs <strong>all 14 tools in parallel</strong> using <code>std::thread::scope</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/main.rs:153-436 (conceptual)</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;">std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">thread</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">scope</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>s</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> s</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">spawn</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">||</span><span style="color: #F9E2AF;"> commit_roaster</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">run</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span> s</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">spawn</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">||</span><span style="color: #F9E2AF;"> deps_shamer</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">run</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span> s</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">spawn</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">||</span><span style="color: #F9E2AF;"> pr_title_hunter</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">run</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span> s</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">spawn</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">||</span><span style="color: #F9E2AF;"> debt_invoice</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">run</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ... 10 more tools</span></span>
<span class="giallo-l"><span style="color: #9399B2;">});</span></span></code></pre>
<p>Each tool runs independently and produces its own score. The final report aggregates all scores into a combined project health metric.</p>
<h2 id="data-flow-from-source-file-to-report">Data Flow: From Source File to Report</h2>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
sequenceDiagram
participant CLI
participant Analyzer
participant TreeSitter
participant Adapter
participant StyleIR
participant Detector
participant Scorer
participant Reporter
CLI->>Analyzer: analyze(path)
Analyzer->>TreeSitter: parse_file(path, content)
TreeSitter-->>Analyzer: ParsedFile (AST + source)
Note over Analyzer: Phase 2-3: Duplication detection
Analyzer->>StyleIR: from_parsed(parsed)
StyleIR->>Adapter: compute_all(file)
Adapter->>Adapter: batch_captures() — single AST traversal
Adapter-->>StyleIR: AdapterCounts
StyleIR-->>Analyzer: StyleIr { counts, functions }
Analyzer->>Detector: detect_findings_with_ir(ir, file)
Detector->>Detector: read ir.panic_call_count (etc.)
Detector-->>Analyzer: Vec<(signal, count)>
Analyzer->>Scorer: calculate_score(findings)
Scorer-->>Analyzer: CodeQualityScore
Analyzer->>Reporter: report(score, findings)
Reporter-->>CLI: Formatted output
</div>
<p>The key insight in this flow: <strong>StyleIR is computed once, consumed by many detectors.</strong> This is the O(detectors + languages) scaling from Article 01 in action.</p>
<h2 id="test-file-handling">Test File Handling</h2>
<p>One detail that deserves attention: test file awareness is baked into the architecture at multiple levels:</p>
<ol>
<li><strong>File identification</strong>: <code>is_test_file()</code> checks path patterns like <code>test/</code>, <code>tests/</code>, <code>_test.go</code>, <code>Test.java</code></li>
<li><strong>Rust-specific</strong>: <code>#[cfg(test)]</code> module byte ranges are detected and panics inside them are excluded (<code>src/language/adapter/rust.rs:16-46</code>)</li>
<li><strong>Detector level</strong>: <code>SignalDetector::skips_test_files()</code> returns <code>true</code> by default — detectors opt-in to test file analysis</li>
<li><strong>Scoring level</strong>: Test file violations are multiplied by 0.2 (<code>src/analyzer.rs:257-260</code>)</li>
<li><strong>Config level</strong>: Users can set <code>signals.skip_tests = true</code> in <code>.garbage-code-hunter.toml</code></li>
</ol>
<p>This layered approach means test code is neither fully ignored nor fully counted — it is proportionally reduced.</p>
<h2 id="configuration-discovery">Configuration Discovery</h2>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph TD
A[Working Directory] --> B{.garbage-code-hunter.toml<br/>exists?}
B -->|Yes| C[Load project config]
B -->|No| D[Walk up to parent]
D --> B
E[Config File] --> F{./config.toml<br/>exists?}
F -->|Yes| G[Load app config]
F -->|No| H{~/.config/garbage-code-hunter/<br/>config.toml exists?}
H -->|Yes| G
H -->|No| I[Use defaults]
</div>
<p>Two independent config systems:</p>
<ul>
<li><strong>ProjectConfig</strong> (<code>.garbage-code-hunter.toml</code>): Per-project rules, whitelists, overrides. Lives in the repo.</li>
<li><strong>AppConfig</strong> (<code>config.toml</code>): Global settings — LLM mode, language preferences. Lives in user's home or project root.</li>
</ul>
<hr />
<p><em>Next: <a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/02-architecture-overview/./03-treesitter-and-language-adapter.md">Tree-sitter and the LanguageAdapter Pattern</a> — How 11 language adapters share a single tree-sitter query engine, and why <code>compute_all()</code> matters.</em></p>
Tree-sitter and the LanguageAdapter Pattern
2026-05-27T00:00:00+00:00
2026-05-27T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/03-treesitter-and-language-adapter/
<h1 id="tree-sitter-and-the-languageadapter-pattern">Tree-sitter and the LanguageAdapter Pattern</h1>
<blockquote>
<p><strong>Question</strong>: You need to detect "debug print statements" in 11 languages. Each language has different syntax, different function names, and different conventions. Do you write 11 separate detection functions, or is there a better way?</p>
</blockquote>
<h2 id="the-naive-approach-11-detectors">The Naive Approach: 11 Detectors</h2>
<p>The most direct solution is to write a <code>DebugPrintDetector</code> that has a match statement for each language:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> count_debug_calls</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> match</span><span> file</span><span style="color: #94E2D5;">.</span><span>language </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Language</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Rust</span><span style="color: #94E2D5;"> =></span><span style="color: #89B4FA;font-style: italic;"> count_rust_debug</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">),</span><span style="color: #9399B2;font-style: italic;"> // println!, dbg!</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Language</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Go</span><span style="color: #94E2D5;"> =></span><span style="color: #89B4FA;font-style: italic;"> count_go_debug</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">),</span><span style="color: #9399B2;font-style: italic;"> // fmt.Println</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Language</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Python</span><span style="color: #94E2D5;"> =></span><span style="color: #89B4FA;font-style: italic;"> count_python_debug</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">),</span><span style="color: #9399B2;font-style: italic;"> // print()</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Language</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Java</span><span style="color: #94E2D5;"> =></span><span style="color: #89B4FA;font-style: italic;"> count_java_debug</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">),</span><span style="color: #9399B2;font-style: italic;"> // System.out.println</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ... 7 more</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This works, but it has a scaling problem. When you add <code>MagicNumberDetector</code>, <code>NamingDetector</code>, and 7 more detectors, each one needs the same 11-way match. You now have O(detectors x languages) code paths, and each one is a potential bug.</p>
<h2 id="the-adapter-pattern-absorb-language-complexity">The Adapter Pattern: Absorb Language Complexity</h2>
<p>garbage-code-hunter inverts the dependency. Instead of detectors knowing about languages, <strong>language adapters know about detections</strong>.</p>
<p>The <code>LanguageAdapter</code> trait (<code>src/language/adapter/mod.rs:78-137</code>) defines a contract: every language must report its own counts for each quality dimension:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub trait</span><span style="color: #F9E2AF;font-style: italic;"> LanguageAdapter</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Send</span><span style="color: #94E2D5;"> +</span><span style="color: #F9E2AF;font-style: italic;"> Sync</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> language</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Language</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_panic_calls</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> extract_functions</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">FunctionNode</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_naming_violations</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_deeply_nested_blocks</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_debug_calls</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_excessive_params</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> threshold</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_unsafe_blocks</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_magic_numbers</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_goroutine_spawns</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;"> }</span><span style="color: #9399B2;font-style: italic;"> // Go-specific</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_defer_in_loop</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;"> }</span><span style="color: #9399B2;font-style: italic;"> // Go-specific</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_go_convention_violations</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_python_issues</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_java_issues</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ... language-specific methods with default no-op implementations</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Notice the pattern: <strong>common methods</strong> (<code>count_panic_calls</code>, <code>count_debug_calls</code>) are required. <strong>Language-specific methods</strong> (<code>count_goroutine_spawns</code>, <code>count_python_issues</code>) have default no-op implementations that only the relevant adapter overrides.</p>
<h2 id="how-adapters-use-tree-sitter-queries">How Adapters Use Tree-sitter Queries</h2>
<p>Each adapter defines a set of tree-sitter query patterns. Here is the GoAdapter (<code>src/language/adapter/go.rs:14-40</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">const</span><span style="color: #FAB387;"> GO_PATTERNS</span><span style="color: #94E2D5;">: &</span><span style="color: #9399B2;">[</span><span style="color: #94E2D5;">&</span><span style="color: #F9E2AF;font-style: italic;">str</span><span style="color: #9399B2;">]</span><span style="color: #94E2D5;"> = &</span><span style="color: #9399B2;">[</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // pc_ — panic calls</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "(call_expression function: (identifier) @pc_fn (#eq? @pc_fn </span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">panic</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">))"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ex_ — extract functions</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "[(function_declaration name: (identifier) @ex_name)</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> (method_declaration name: (field_identifier) @ex_name)] @ex_fn"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // nv_ — naming violations</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "[(short_var_declaration left: (expression_list (identifier) @nv_var))</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> (var_spec name: (identifier) @nv_var)]"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // dp_ — debug calls</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> r#"(call_expression</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> function: (selector_expression</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> operand: (identifier) @dp_pkg</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> field: (field_identifier) @dp_method)</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> (#match? @dp_pkg "^(fmt|log)$")</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> (#match? @dp_method "^(Print|Println|Printf|Fprint|Fprintln|Fprintf)$"))"#</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ep_ — excessive params</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "[(function_declaration parameters: (parameter_list) @ep_params)</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> (method_declaration parameters: (parameter_list) @ep_params)]"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // mn_ — magic numbers</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "[(int_literal) @mn_num (float_literal) @mn_num]"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // gs_ — goroutine spawns</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "(go_statement) @gs_go"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // cv_ — convention violations</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> r#"(call_expression function: (selector_expression</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> operand: (identifier) @cv_pkg field: (field_identifier) @cv_method)</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> (#eq? @cv_pkg "fmt")</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> (#match? @cv_method "^(Errorf|New)$"))"#</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ui_ — unsafe operations</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> r#"(selector_expression operand: (identifier) @ui_pkg (#eq? @ui_pkg "unsafe"))"#</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">];</span></span></code></pre>
<p>The naming convention is critical: each capture name is prefixed with a <strong>2-letter code</strong> (<code>pc_</code>, <code>ex_</code>, <code>dp_</code>, <code>mn_</code>) that identifies which counting function should process it. This allows all patterns to be merged into a single query string.</p>
<h2 id="the-batch-query-optimization">The Batch Query Optimization</h2>
<p>Here is where the architecture gets clever. Instead of running each pattern separately (10 AST traversals per file), <code>batch_captures()</code> merges all patterns and runs them in <strong>one pass</strong>:</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph LR
subgraph "Naive: 10 traversals"
T1[Pattern 1] --> AST1[AST Walk 1]
T2[Pattern 2] --> AST2[AST Walk 2]
T3[...] --> AST3[AST Walk ...]
T10[Pattern 10] --> AST10[AST Walk 10]
end
subgraph "Batch: 1 traversal"
BM[All Patterns<br/>Merged] --> BAST[Single<br/>AST Walk]
BAST --> BC[Vec of<br/>Capture Groups]
end
</div>
<p>The implementation (<code>src/language/adapter/mod.rs:265-271</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> batch_captures</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">a</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">a</span><span style="color: #F9E2AF;font-style: italic;"> ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">QueryCapture</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">a</span><span style="color: #89DCEB;">>>></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> patterns</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">query_patterns</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> patterns</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">is_empty</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> collect_captures_multi</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span> patterns</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">unwrap_or_default</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>And <code>compute_all()</code> (<code>src/language/adapter/mod.rs:278-304</code>) is the single entry point that calls <code>batch_captures()</code> once and distributes results:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> compute_all</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> AdapterCounts</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> batch</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">batch_captures</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> AdapterCounts</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> functions</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">extract_functions_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> panic_calls</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_panic_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> naming_violations</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_naming_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> deeply_nested_blocks</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_deeply_nested_blocks</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> debug_calls</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_debug_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> excessive_params</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_excessive_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> unsafe_blocks</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_unsafe_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> magic_numbers</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_magic_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> goroutine_spawns</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_goroutine_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> defer_in_loop</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_defer_in_loop</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> go_conventions</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_go_convention_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> python_issues</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_python_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> java_issues</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_java_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> ruby_issues</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_ruby_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> c_issues</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_c_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> ts_issues</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_ts_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> js_issues</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_js_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> swift_issues</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_swift_from_batch</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>batch</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> dead_code</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_dead_code</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> duplicate_imports</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count_duplicate_imports</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ...</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The result is an <code>AdapterCounts</code> struct — a flat bag of numbers that knows nothing about which language produced them.</p>
<h2 id="thread-local-query-cache">Thread-Local Query Cache</h2>
<p>Tree-sitter queries must be compiled before execution. Compilation is not free — it involves regex compilation and pattern analysis. To avoid recompiling the same query on every file, garbage-code-hunter uses a thread-local cache (<code>src/treesitter/query.rs:64-67</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">thread_local!</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> static</span><span style="color: #FAB387;"> QUERY_CACHE</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> RefCell</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">HashMap</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">Language</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">),</span><span> tree_sitter</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Query</span><span style="color: #89DCEB;">>></span><span style="color: #94E2D5;"> =</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> RefCell</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">HashMap</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The key is <code>(Language, String)</code> — the language and the query pattern string. The first time a pattern is used for a language, it is compiled and cached. Subsequent calls reuse the compiled query.</p>
<p>This is thread-local rather than shared because tree-sitter <code>Query</code> is not <code>Send</code>. Each thread gets its own cache, which avoids lock contention in the parallel <code>scan</code> mode.</p>
<h2 id="the-full-call-flow">The Full Call Flow</h2>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
sequenceDiagram
participant IR as StyleIR
participant Adapter as LanguageAdapter
participant Cache as Thread-Local Cache
participant TS as Tree-sitter
IR->>Adapter: compute_all(file)
Adapter->>Adapter: query_patterns() → all patterns
Adapter->>Adapter: batch_captures(file)
loop For each pattern
Adapter->>Cache: get(language, pattern)
Cache-->>Adapter: compiled Query (or compile + cache)
end
Adapter->>TS: run merged query on AST
TS-->>Adapter: Vec of capture groups
Adapter->>Adapter: count_*_from_batch() for each signal
Adapter-->>IR: AdapterCounts { all counts }
</div><h2 id="adding-a-new-language-the-checklist">Adding a New Language: The Checklist</h2>
<p>When someone adds Zig support to garbage-code-hunter, here is exactly what they do:</p>
<ol>
<li>
<p><strong>Add the grammar dependency</strong> in <code>Cargo.toml</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="toml"><span class="giallo-l"><span>tree-sitter-zig</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "0.7"</span></span></code></pre></li>
<li>
<p><strong>Register the parser</strong> in <code>src/treesitter/parsers.rs</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;">Language</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Zig</span><span style="color: #94E2D5;"> =></span><span style="color: #F9E2AF;"> tree_sitter_zig</span><span style="color: #94E2D5;">::</span><span style="color: #FAB387;">LANGUAGE</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">into</span><span style="color: #9399B2;">(),</span></span></code></pre></li>
<li>
<p><strong>Add the language variant</strong> in <code>src/language/mod.rs</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;">Zig</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "zig"</span><span style="color: #9399B2;">,</span></span></code></pre></li>
<li>
<p><strong>Implement the adapter</strong> in <code>src/language/adapter/zig.rs</code> (~200 lines):</p>
<ul>
<li>Define <code>ZIG_PATTERNS</code> with prefixed capture names</li>
<li>Implement <code>LanguageAdapter</code> for <code>ZigAdapter</code></li>
<li>Override the counting methods to process captures</li>
</ul>
</li>
<li>
<p><strong>Register the adapter</strong> in <code>src/language/adapter/mod.rs</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;">Language</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Zig</span><span style="color: #94E2D5;"> =></span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F9E2AF;font-style: italic;">ZigAdapter</span><span style="color: #9399B2;">),</span></span></code></pre></li>
</ol>
<p><strong>Zero detectors modified. Zero scoring logic changed.</strong> The new language immediately works with all 10 detectors because they read from <code>StyleIr</code>, not from language-specific ASTs.</p>
<h2 id="design-tradeoffs">Design Tradeoffs</h2>
<p>This adapter pattern has clear advantages, but also tradeoffs:</p>
<p><strong>Advantage</strong>: O(detectors + languages) scaling. Adding a detector works for all languages. Adding a language works with all detectors.</p>
<p><strong>Tradeoff</strong>: The adapter must pre-compute everything a detector might need. If a new detector needs a signal the adapter does not compute, you must update the adapter trait AND all 11 implementations. In practice, this happens rarely — the current trait covers all signals used by the 10 detectors.</p>
<p><strong>Tradeoff</strong>: Language-specific features (Go goroutines, Rust unsafe blocks, Python decorators) require dedicated methods on the trait. The trait grows as language diversity increases. The default no-op implementations mitigate this — only the relevant adapter overrides each method.</p>
<hr />
<p><em>Next: <a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/03-treesitter-and-language-adapter/./04-style-ir-the-neutral-layer.md">StyleIR: The Language-Neutral Intermediate Representation</a> — How adapter counts become a stable fact layer that detectors consume without knowing the language.</em></p>
StyleIR: The Language-Neutral Intermediate Representation
2026-05-27T00:00:00+00:00
2026-05-27T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/04-style-ir-the-neutral-layer/
<h1 id="styleir-the-language-neutral-intermediate-representation">StyleIR: The Language-Neutral Intermediate Representation</h1>
<blockquote>
<p><strong>Question</strong>: You have 11 language adapters that each produce raw counts. You have 10 detectors that each need those counts. How do you connect them without coupling detectors to language-specific data structures?</p>
</blockquote>
<h2 id="the-coupling-problem">The Coupling Problem</h2>
<p>Consider the <code>PanicAddictionDetector</code>. It needs to know how many panic-like calls exist in a file. In Rust, that is <code>unwrap()</code> and <code>expect()</code>. In Go, that is <code>panic()</code>. In Python, that is bare <code>raise</code> without a specific exception type.</p>
<p>Without an intermediate layer, the detector would need to either:</p>
<ol>
<li>Query each adapter directly (coupling detector to adapter)</li>
<li>Maintain its own tree-sitter queries (duplicating adapter logic)</li>
<li>Read raw AST nodes (coupling detector to language grammar)</li>
</ol>
<p>All three options create tight coupling that breaks when you add language #12.</p>
<h2 id="styleir-facts-not-judgments">StyleIR: Facts, Not Judgments</h2>
<p>StyleIR (<code>src/style_ir/mod.rs</code>) is a language-neutral fact layer. It stores <strong>counts</strong> — objective measurements extracted by adapters — not <strong>judgments</strong> about whether those counts are problematic.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/style_ir/mod.rs:55-133</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> StyleIr</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> language</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Language</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> line_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> functions</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">FunctionNode</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> panic_call_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> naming_violation_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> deeply_nested_block_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> debug_call_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> excessive_param_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> unsafe_block_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> magic_number_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> commented_out_lines</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> todo_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> goroutine_spawn_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> defer_in_loop_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> go_convention_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> python_issue_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> java_issue_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> ruby_issue_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> c_issue_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> ts_issue_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> js_issue_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> swift_issue_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> dead_code_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> duplicate_import_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The key insight: <strong>every field is a <code>usize</code></strong> (or <code>Vec<FunctionNode></code>). There are no language-specific types. A detector reading <code>ir.panic_call_count</code> does not know or care whether the panics came from Rust's <code>unwrap()</code> or Go's <code>panic()</code>.</p>
<h2 id="construction-from-parsedfile-to-styleir">Construction: From ParsedFile to StyleIR</h2>
<p>The data flow is a three-step pipeline:</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph LR
PF[ParsedFile<br/>AST + source] --> AD[adapter.compute_all<br/>single traversal]
AD --> AC[AdapterCounts<br/>raw counts]
AC --> IR[StyleIr<br/>language-neutral]
</div>
<p>The construction happens in <code>StyleIr::from_parsed()</code> (<code>src/style_ir/mod.rs:143-172</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> from_parsed</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;">file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #F38BA8;">Self</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> adapter</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> adapter_for</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">language</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span><span style="color: #9399B2;font-style: italic;"> // 1. Get the right adapter</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> counts</span><span style="color: #94E2D5;"> =</span><span> adapter</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">compute_all</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">);</span><span style="color: #9399B2;font-style: italic;"> // 2. Run batch query</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span style="color: #F38BA8;">Self</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> language</span><span style="color: #94E2D5;">:</span><span> file</span><span style="color: #94E2D5;">.</span><span>language</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> line_count</span><span style="color: #94E2D5;">:</span><span> file</span><span style="color: #94E2D5;">.</span><span>content</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">lines</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> functions</span><span style="color: #94E2D5;">:</span><span> counts</span><span style="color: #94E2D5;">.</span><span>functions</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> panic_call_count</span><span style="color: #94E2D5;">:</span><span> counts</span><span style="color: #94E2D5;">.</span><span>panic_calls</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> naming_violation_count</span><span style="color: #94E2D5;">:</span><span> counts</span><span style="color: #94E2D5;">.</span><span>naming_violations</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> deeply_nested_block_count</span><span style="color: #94E2D5;">:</span><span> counts</span><span style="color: #94E2D5;">.</span><span>deeply_nested_blocks</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> debug_call_count</span><span style="color: #94E2D5;">:</span><span> counts</span><span style="color: #94E2D5;">.</span><span>debug_calls</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> excessive_param_count</span><span style="color: #94E2D5;">:</span><span> counts</span><span style="color: #94E2D5;">.</span><span>excessive_params</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ... all counts mapped 1:1</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Three important details:</p>
<ol>
<li>
<p><strong><code>adapter_for()</code> returns <code>Option</code></strong> — if a language has no adapter, <code>from_parsed()</code> returns <code>None</code>. Callers fall back to legacy rule logic.</p>
</li>
<li>
<p><strong><code>compute_all()</code> runs once</strong> — the batch query optimization from Article 03 means all counts are extracted in a single AST traversal.</p>
</li>
<li>
<p><strong>Counts are mapped 1:1</strong> — <code>AdapterCounts</code> fields map directly to <code>StyleIr</code> fields. No transformation, no filtering. The adapter already did the semantic work.</p>
</li>
</ol>
<h2 id="derived-signals-composite-counts">Derived Signals: Composite Counts</h2>
<p>Some quality signals are not directly measured — they are <strong>derived</strong> from multiple base counts. StyleIR computes these composites:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/style_ir/mod.rs:174-204</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;">/// Functions exceeding the god-function threshold (50 lines).</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> god_function_count</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>functions</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">filter</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>f</span><span style="color: #94E2D5;">|</span><span> f</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">end_line</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">saturating_sub</span><span style="color: #9399B2;">(</span><span>f</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">start_line</span><span style="color: #9399B2;">)</span><span style="color: #89DCEB;"> ></span><span style="color: #F38BA8;"> Self</span><span style="color: #94E2D5;">::</span><span style="color: #FAB387;">GOD_FUNCTION_LINE_THRESHOLD</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">count</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;">/// Combined over-engineering signal.</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> over_engineering_count</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">god_function_count</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>excessive_param_count </span><span style="color: #94E2D5;">+</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>goroutine_spawn_count</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;">/// Combined code-smell signal.</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> code_smell_count</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>unsafe_block_count </span><span style="color: #94E2D5;">*</span><span style="color: #FAB387;"> 2</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>magic_number_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>go_convention_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>python_issue_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>java_issue_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>ruby_issue_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>c_issue_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>ts_issue_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>js_issue_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>swift_issue_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>dead_code_count</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>duplicate_import_count</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is where the "facts, not judgments" philosophy pays off. The <code>god_function_count()</code> method is a <strong>judgment</strong> — it decides that 50 lines is the threshold. But the underlying data (function start/end lines) is a <strong>fact</strong> extracted by the adapter. If you want to change the threshold, you change one constant, not 11 adapters.</p>
<h2 id="functionnode-the-richer-data">FunctionNode: The Richer Data</h2>
<p>Not everything fits in a <code>usize</code>. Function metadata needs more structure:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/language/adapter/mod.rs:38-44</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> FunctionNode</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> start_line</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> end_line</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> nesting_depth</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This gives detectors the ability to ask questions like:</p>
<ul>
<li>"Is there a function longer than 50 lines?" (<code>god_function_count()</code>)</li>
<li>"Is there a function with more than 5 parameters?" (via <code>excessive_param_count</code>)</li>
<li>"Is there a deeply nested function?" (via <code>nesting_depth</code>)</li>
</ul>
<h2 id="the-summary-json-ready-output">The Summary: JSON-Ready Output</h2>
<p>For downstream consumers (CI bots, dashboards, the <code>analyze --json</code> command), StyleIR produces a stable summary:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/style_ir/mod.rs:207-250</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> summary</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> StyleIrSummary</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> StyleIrSummary</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> language</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>language</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">display_name</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> line_count</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>line_count</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> function_count</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>functions</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">len</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> god_function_count</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">god_function_count</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> panic_call_count</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>panic_call_count</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> naming_violation_count</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>naming_violation_count</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ... all fields</span></span>
<span class="giallo-l"><span> is_clean_signal_baseline</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">is_clean_signal_baseline</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> thresholds</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> StyleIrThresholdSummary</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> excessive_param_threshold</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> Self</span><span style="color: #94E2D5;">::</span><span style="color: #FAB387;">EXCESSIVE_PARAM_THRESHOLD</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> god_function_line_threshold</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> Self</span><span style="color: #94E2D5;">::</span><span style="color: #FAB387;">GOD_FUNCTION_LINE_THRESHOLD</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The <code>is_clean_signal_baseline</code> field is interesting — it returns <code>true</code> when a file has zero violations across all signals. This allows CI pipelines to quickly identify "clean" files without inspecting every count.</p>
<h2 id="why-not-use-a-general-ast">Why Not Use a General AST?</h2>
<p>A natural question: why not give detectors access to the full AST and let them extract what they need?</p>
<p>The answer is <strong>separation of concerns</strong>:</p>
<table><thead><tr><th>Approach</th><th>Pros</th><th>Cons</th></tr></thead><tbody>
<tr><td>Direct AST access</td><td>Maximum flexibility</td><td>Detectors must understand 11 grammars</td></tr>
<tr><td>StyleIR counts</td><td>Language-agnostic, fast</td><td>Must pre-compute everything detectors need</td></tr>
<tr><td>Hybrid (counts + AST)</td><td>Best of both</td><td>Complex, easy to bypass the abstraction</td></tr>
</tbody></table>
<p>garbage-code-hunter chose the pure StyleIR approach because:</p>
<ol>
<li>
<p><strong>Detectors are simple.</strong> A detector is 10-30 lines of code — it reads a count and decides if it is above threshold. No AST walking, no language matching.</p>
</li>
<li>
<p><strong>Testing is easy.</strong> You can construct a <code>StyleIr</code> with known counts and verify detector behavior without parsing real code.</p>
</li>
<li>
<p><strong>Performance is predictable.</strong> <code>from_parsed()</code> does one batch query. No lazy evaluation, no surprise traversals.</p>
</li>
<li>
<p><strong>Adding languages is free.</strong> New adapters produce the same <code>AdapterCounts</code>. Detectors do not change.</p>
</li>
</ol>
<p>The cost is that adding a new quality signal requires touching the adapter trait. In practice, this happens infrequently — the current set of 20+ counts covers all signals used by the 10 detectors.</p>
<h2 id="how-detectors-consume-styleir">How Detectors Consume StyleIR</h2>
<p>Here is the <code>PanicAddictionDetector</code> (<code>src/detectors.rs:44-62</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">impl</span><span style="color: #F9E2AF;font-style: italic;"> SignalDetector</span><span style="color: #CBA6F7;"> for</span><span style="color: #F9E2AF;font-style: italic;"> PanicAddictionDetector</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> signal</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> StyleSignal</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> StyleSignal</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">PanicAddiction</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> supported_languages</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> &</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #9399B2;"> [</span><span style="color: #F9E2AF;font-style: italic;">Language</span><span style="color: #9399B2;">] {</span></span>
<span class="giallo-l"><span style="color: #FAB387;"> ADAPTER_LANGUAGES</span><span style="color: #9399B2;font-style: italic;"> // all 11 languages</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_violations</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> StyleIr</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">from_parsed</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>ir</span><span style="color: #94E2D5;">|</span><span> ir</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">panic_call_count</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">unwrap_or</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_violations_with_ir</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ir</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">StyleIr</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> _file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> ir</span><span style="color: #94E2D5;">.</span><span>panic_call_count</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Notice two paths:</p>
<ul>
<li><code>count_violations()</code> — constructs a fresh StyleIR (used when no pre-computed IR is available)</li>
<li><code>count_violations_with_ir()</code> — reads from an existing StyleIR (used in the batch pipeline to avoid redundant computation)</li>
</ul>
<p>Both return <code>ir.panic_call_count</code>. The detector has zero knowledge of Rust, Go, or any language.</p>
<h2 id="the-architecture-in-one-diagram">The Architecture in One Diagram</h2>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph TB
subgraph "Language Layer"
R[Rust<br/>Adapter] --> AC1[AdapterCounts]
G[Go<br/>Adapter] --> AC2[AdapterCounts]
P[Python<br/>Adapter] --> AC3[AdapterCounts]
Z[...11] --> ACN[AdapterCounts]
end
subgraph "Fact Layer"
AC1 --> IR[StyleIr]
AC2 --> IR
AC3 --> IR
ACN --> IR
end
subgraph "Judgment Layer"
IR --> D1[PanicAddiction<br/>Detector]
IR --> D2[NamingChaos<br/>Detector]
IR --> D3[NestedHell<br/>Detector]
IR --> DN[...10<br/>Detectors]
end
D1 --> F[StyleFinding]
D2 --> F
D3 --> F
DN --> F
</div>
<p>Three layers, each with a clear responsibility:</p>
<ul>
<li><strong>Language Layer</strong>: Adapters absorb grammar-specific knowledge</li>
<li><strong>Fact Layer</strong>: StyleIR stores objective counts, language-agnostic</li>
<li><strong>Judgment Layer</strong>: Detectors decide what constitutes a problem</li>
</ul>
<hr />
<p><em>Next: <a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/04-style-ir-the-neutral-layer/./05-signal-detection-system.md">The Signal Detection System</a> — How 10 detectors cover the full spectrum of code smells, and why "fewer rules, stronger signals" beats the alternative.</em></p>
The Signal Detection System
2026-05-27T00:00:00+00:00
2026-05-27T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/05-signal-detection-system/
<h1 id="the-signal-detection-system">The Signal Detection System</h1>
<blockquote>
<p><strong>Question</strong>: A codebase has 200 different "rules" that each flag a specific pattern. 120 of them fire on every file. Developers ignore the output. Sound familiar?</p>
</blockquote>
<h2 id="the-rule-explosion-problem">The Rule Explosion Problem</h2>
<p>Traditional linters follow a "more rules = more coverage" philosophy. ESLint has 250+ rules. Pylint has 200+. RuboCop has 500+. The problem is not finding issues — it is <strong>drowning in them</strong>.</p>
<p>When every line triggers a warning, developers stop reading warnings. This is the linter equivalent of car alarms — everyone ignores them because they go off constantly.</p>
<p>garbage-code-hunter takes the opposite approach: <strong>fewer rules, stronger signals</strong>.</p>
<h2 id="10-detectors-not-200-rules">10 Detectors, Not 200 Rules</h2>
<p>Instead of hundreds of granular rules, garbage-code-hunter defines 10 signal detectors (<code>src/detectors.rs</code>). Each detector covers a <strong>behavioral dimension</strong> — a category of code smell that indicates a specific kind of developer behavior:</p>
<table><thead><tr><th>#</th><th>Detector</th><th>Signal</th><th>What It Detects</th><th>Why It Matters</th></tr></thead><tbody>
<tr><td>1</td><td><code>PanicAddictionDetector</code></td><td>PanicAddiction</td><td><code>.unwrap()</code>, <code>panic!()</code>, <code>expect()</code></td><td>Error handling laziness</td></tr>
<tr><td>2</td><td><code>NamingChaosDetector</code></td><td>NamingChaos</td><td>Single-letter vars, meaningless names</td><td>Communication failure</td></tr>
<tr><td>3</td><td><code>NestedHellDetector</code></td><td>NestedHell</td><td>Blocks nested >= 5 levels</td><td>Cognitive complexity</td></tr>
<tr><td>4</td><td><code>HotfixCultureDetector</code></td><td>HotfixCulture</td><td><code>println!</code>, <code>dbg!</code>, <code>todo!</code>, <code>unimplemented!</code></td><td>Debug leftovers</td></tr>
<tr><td>5</td><td><code>OverEngineeringDetector</code></td><td>OverEngineering</td><td>God functions (>50 lines), >5 params</td><td>Over-abstraction</td></tr>
<tr><td>6</td><td><code>CodeSmellsDetector</code></td><td>CodeSmells</td><td>Unsafe blocks, magic numbers, dup imports</td><td>General hygiene</td></tr>
<tr><td>7</td><td><code>DuplicationDetector</code></td><td>Duplication</td><td>Repeated code blocks</td><td>Copy-paste culture</td></tr>
<tr><td>8</td><td><code>LegacyCodeDetector</code></td><td>LegacyCode</td><td>Commented-out code (3+ lines)</td><td>Dead weight</td></tr>
<tr><td>9</td><td><code>TodoMountainDetector</code></td><td>TodoMountain</td><td>TODO/FIXME/BUG/HACK markers</td><td>Deferred debt</td></tr>
<tr><td>10</td><td><code>LineCountSmellDetector</code></td><td>LineCountSmell</td><td>Files >1000 lines</td><td>Monolith tendency</td></tr>
</tbody></table>
<p>Each detector is <strong>language-agnostic</strong> — it reads from <code>StyleIr</code> and does not know which language the code is written in.</p>
<h2 id="the-signaldetector-trait">The SignalDetector Trait</h2>
<p>The trait (<code>src/signals.rs:17-79</code>) is deliberately minimal:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub trait</span><span style="color: #F9E2AF;font-style: italic;"> SignalDetector</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Send</span><span style="color: #94E2D5;"> +</span><span style="color: #F9E2AF;font-style: italic;"> Sync</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> signal</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> StyleSignal</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> supported_languages</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> &</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #9399B2;"> [</span><span style="color: #F9E2AF;font-style: italic;">Language</span><span style="color: #9399B2;">];</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_violations</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_violations_with_ir</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ir</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">StyleIr</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> skips_test_files</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> bool</span><span style="color: #9399B2;"> {</span><span style="color: #F38BA8;"> true</span><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> detect_findings</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">...</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">StyleSignal</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> detect_findings_with_ir</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">...</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">StyleSignal</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The key methods:</p>
<ul>
<li><strong><code>signal()</code></strong> — returns the <code>StyleSignal</code> variant this detector produces. This is the detector's identity.</li>
<li><strong><code>supported_languages()</code></strong> — which languages this detector applies to. Most return <code>ADAPTER_LANGUAGES</code> (all 11).</li>
<li><strong><code>count_violations()</code></strong> — the core detection logic. Returns the raw violation count.</li>
<li><strong><code>count_violations_with_ir()</code></strong> — optimized path using pre-computed StyleIR.</li>
<li><strong><code>skips_test_files()</code></strong> — whether test files should be excluded (default: <code>true</code>).</li>
</ul>
<h2 id="how-a-detector-works-panicaddiction">How a Detector Works: PanicAddiction</h2>
<p>Here is the complete implementation (<code>src/detectors.rs:44-62</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">impl</span><span style="color: #F9E2AF;font-style: italic;"> SignalDetector</span><span style="color: #CBA6F7;"> for</span><span style="color: #F9E2AF;font-style: italic;"> PanicAddictionDetector</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> signal</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> StyleSignal</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> StyleSignal</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">PanicAddiction</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> supported_languages</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> &</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #9399B2;"> [</span><span style="color: #F9E2AF;font-style: italic;">Language</span><span style="color: #9399B2;">] {</span></span>
<span class="giallo-l"><span style="color: #FAB387;"> ADAPTER_LANGUAGES</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_violations</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> StyleIr</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">from_parsed</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>ir</span><span style="color: #94E2D5;">|</span><span> ir</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">panic_call_count</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">unwrap_or</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_violations_with_ir</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ir</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">StyleIr</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> _file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> ir</span><span style="color: #94E2D5;">.</span><span>panic_call_count</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>That is it. The entire detector is 18 lines. It reads <code>ir.panic_call_count</code> — a number that the language adapter already computed. The detector does not know about <code>unwrap()</code>, <code>panic!()</code>, or any language-specific syntax.</p>
<h2 id="test-file-handling-the-20-rule">Test File Handling: The 20% Rule</h2>
<p>Test code is different from production code. <code>unwrap()</code> in a test is acceptable — in production, it is not. But completely ignoring test code would miss real issues in test helpers and utilities.</p>
<p>garbage-code-hunter applies a <strong>20% weight</strong> to test file violations (<code>src/analyzer.rs:257-260</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> count</span><span style="color: #94E2D5;"> =</span><span style="color: #CBA6F7;"> if</span><span style="color: #94E2D5;"> *</span><span>is_test_file</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> (</span><span>count</span><span style="color: #CBA6F7;"> as f64</span><span style="color: #94E2D5;"> *</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">2</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">round</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> as usize</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span><span style="color: #CBA6F7;"> else</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> count</span></span>
<span class="giallo-l"><span style="color: #9399B2;">};</span></span></code></pre>
<p>This means:</p>
<ul>
<li>10 <code>unwrap()</code> calls in production code = 10 violations</li>
<li>10 <code>unwrap()</code> calls in test code = 2 violations</li>
</ul>
<p>The 20% is not arbitrary — it is low enough to prevent test code from dominating scores, but high enough to flag genuinely problematic patterns in test helpers.</p>
<h2 id="rust-specific-cfg-test-awareness">Rust-Specific: <code>#[cfg(test)]</code> Awareness</h2>
<p>For Rust, the adapter goes further. It detects <code>#[cfg(test)]</code> module byte ranges and <strong>excludes panics inside them</strong> from counting entirely (<code>src/language/adapter/rust.rs:16-46</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// Find #[cfg(test)] module byte ranges</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> cfg_test_ranges</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">(</span><span style="color: #CBA6F7;">usize</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)</span><span style="color: #89DCEB;">></span><span style="color: #94E2D5;"> = ...</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// When counting panic calls, skip those inside #[cfg(test)] modules</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> is_in_cfg_test_module</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;">node</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Node</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ranges</span><span style="color: #94E2D5;">: &</span><span style="color: #9399B2;">[(</span><span style="color: #CBA6F7;">usize</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)])</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> bool</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> start</span><span style="color: #94E2D5;"> =</span><span> node</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">start_byte</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span> ranges</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">any</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|&</span><span style="color: #9399B2;">(</span><span>lo</span><span style="color: #9399B2;">,</span><span> hi</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">|</span><span> start</span><span style="color: #94E2D5;"> >=</span><span> lo</span><span style="color: #94E2D5;"> &&</span><span> start</span><span style="color: #94E2D5;"> <</span><span> hi</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is more precise than the 20% rule — it identifies exactly which code is test-only at the language level, not just at the file-path level.</p>
<h2 id="detection-flow">Detection Flow</h2>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
sequenceDiagram
participant A as Analyzer
participant IR as StyleIR
participant D as Detector
participant S as Scorer
A->>IR: from_parsed(file)
IR-->>A: StyleIr { counts }
loop For each detector
A->>D: detect_findings_with_ir(ir, file, is_test, skip_tests)
alt is_test && detector.skips_test_files()
D-->>A: [] (empty)
else
D->>D: count_violations_with_ir(ir)
D-->>A: [(signal, count)]
end
alt is_test_file
A->>A: count = count * 0.2
end
A->>A: push StyleFinding
end
A->>S: calculate_score(findings)
</div><h2 id="why-fewer-rules-works">Why "Fewer Rules" Works</h2>
<p>The 10-detector approach works because of a key insight: <strong>developers do not have 200 bad habits — they have 10.</strong></p>
<p>The <code>PanicAddiction</code> signal does not care whether you called <code>.unwrap()</code> on a <code>Result</code> or an <code>Option</code>. It does not care whether the call is in a <code>match</code> arm or a chain. It cares about one thing: <strong>how many times did you skip error handling?</strong></p>
<p>This coarse-grained approach has advantages:</p>
<ol>
<li>
<p><strong>Low false positive rate.</strong> A detector that counts <code>unwrap()</code> calls has near-zero false positives — every <code>unwrap()</code> is a conscious choice to skip error handling.</p>
</li>
<li>
<p><strong>Actionable feedback.</strong> "You have 47 panic calls" is more actionable than "Line 42: consider using <code>match</code> instead of <code>unwrap()</code>" x 47.</p>
</li>
<li>
<p><strong>Cross-language consistency.</strong> The same 10 signals apply to all 11 languages. A Python project and a Rust project are scored on the same scale.</p>
</li>
<li>
<p><strong>Personality inference.</strong> With 10 signals, you can map patterns to archetypes (Article 08). With 200 rules, the signal is lost in noise.</p>
</li>
</ol>
<h2 id="the-signal-score">The Signal Score</h2>
<p>Each detector produces a raw violation count. The scoring system converts this to a normalized score using density-based logarithmic scaling (<code>src/signals.rs:82-86</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> violations_to_score</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;">count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> total_lines</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> f64</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> k_lines</span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> (</span><span>total_lines</span><span style="color: #CBA6F7;"> as f64</span><span style="color: #94E2D5;"> /</span><span style="color: #FAB387;"> 1000</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">max</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">001</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> density</span><span style="color: #94E2D5;"> =</span><span> count</span><span style="color: #CBA6F7;"> as f64</span><span style="color: #94E2D5;"> /</span><span> k_lines</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> ((</span><span>density</span><span style="color: #94E2D5;"> +</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">log2</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #FAB387;"> 6</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">min</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">25</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The formula:</p>
<ul>
<li><strong>Density</strong>: violations per 1000 lines (fair across project sizes)</li>
<li><strong>Log2</strong>: diminishing returns (10 violations is bad, 100 is not 10x worse)</li>
<li><strong>Cap at 25</strong>: prevents any single signal from dominating</li>
</ul>
<p>The scoring model is covered in depth in Article 07.</p>
<h2 id="adding-a-new-detector">Adding a New Detector</h2>
<p>To add a <code>MagicNumberDetector</code>:</p>
<ol>
<li>Add <code>StyleSignal::MagicNumber</code> to the signal enum</li>
<li>Create <code>MagicNumberDetector</code> implementing <code>SignalDetector</code></li>
<li><code>count_violations_with_ir()</code> returns <code>ir.magic_number_count</code></li>
<li>Register it in the detector list</li>
</ol>
<p><strong>Zero adapter changes.</strong> The <code>magic_number_count</code> field already exists in <code>StyleIr</code> because every adapter already computes it. The detector just reads it.</p>
<hr />
<p><em>Next: <a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/05-signal-detection-system/./06-duplication-algorithms.md">Duplication Detection Algorithms</a> — How to find copy-paste code across an entire codebase, and why Jaccard similarity is not enough.</em></p>
Duplication Detection Algorithms
2026-05-27T00:00:00+00:00
2026-05-27T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/06-duplication-algorithms/
<h1 id="duplication-detection-algorithms">Duplication Detection Algorithms</h1>
<blockquote>
<p><strong>Question</strong>: You have a 10,000-line codebase. Somewhere in it, two functions are 95% identical — differing only in variable names. How do you find them without comparing every pair of functions?</p>
</blockquote>
<h2 id="why-duplication-is-hard">Why Duplication Is Hard</h2>
<p>Code duplication is the most pervasive code smell and the hardest to detect automatically. The challenge is not finding exact copies — <code>diff</code> can do that. The challenge is finding <strong>near-duplicates</strong>: code blocks that do the same thing with different variable names, different string literals, or trivially different control flow.</p>
<p>garbage-code-hunter uses two complementary approaches:</p>
<ul>
<li><strong>Cross-file duplication</strong>: finds duplicate functions across the entire codebase</li>
<li><strong>Intra-file duplication</strong>: finds repeated code blocks within a single file</li>
</ul>
<h2 id="cross-file-duplication-ast-token-fingerprinting">Cross-File Duplication: AST Token Fingerprinting</h2>
<p>The cross-file detector (<code>src/treesitter/duplication.rs</code>) works in three steps:</p>
<h3 id="step-1-extract-and-normalize-functions">Step 1: Extract and Normalize Functions</h3>
<p>For each file, the detector extracts all functions and normalizes their AST tokens:</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph LR
SRC[Source Code] --> AST[Parse to AST]
AST --> TOKENS[Extract Tokens]
TOKENS --> NORM[Normalize]
subgraph "Normalization"
NORM --> ID[Identifiers → Hash]
NORM --> STR[Strings → STRING_LIT]
NORM --> NUM[Numbers → NUMBER_LIT]
NORM --> KEY[Keywords → Keep]
end
NORM --> HASH[Token Hash]
</div>
<p>The normalization rules:</p>
<ul>
<li><strong>Identifiers</strong> are replaced with a hash of their name. Two functions that differ only in variable names will produce the same token sequence.</li>
<li><strong>String literals</strong> are replaced with <code>STRING_LIT</code>. Two functions that differ only in hardcoded strings will match.</li>
<li><strong>Numeric literals</strong> are replaced with <code>NUMBER_LIT</code>.</li>
<li><strong>Keywords and operators</strong> are kept as-is — they represent structure, not naming.</li>
</ul>
<h3 id="step-2-exact-duplicate-detection">Step 2: Exact Duplicate Detection</h3>
<p>After normalization, each function is represented as a hash of its token sequence. Functions with identical hashes are exact duplicates (after normalization):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// Conceptual: group functions by their normalized hash</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> hash_groups</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> HashMap</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">u64</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #94E2D5;">&</span><span style="color: #F9E2AF;font-style: italic;">FunctionInfo</span><span style="color: #89DCEB;">>></span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> HashMap</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span> func</span><span style="color: #CBA6F7;"> in</span><span style="color: #94E2D5;"> &</span><span>functions</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> hash_groups</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">entry</span><span style="color: #9399B2;">(</span><span>func</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">token_hash</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">or_default</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span>func</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// Groups with >1 member are exact duplicates</span></span></code></pre>
<p>This is O(n) — linear in the number of functions. Hash collisions are possible but extremely rare with 64-bit hashes.</p>
<h3 id="step-3-near-duplicate-detection-with-jaccard-similarity">Step 3: Near-Duplicate Detection with Jaccard Similarity</h3>
<p>Exact duplicates are easy. Near-duplicates require a similarity metric. garbage-code-hunter uses <strong>Jaccard similarity on token bigrams</strong> (<code>src/treesitter/duplication.rs:275-343</code>):</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph LR
F1[Function A<br/>token bigrams] --> JACC[Jaccard<br/>Similarity]
F2[Function B<br/>token bigrams] --> JACC
JACC --> |">= 0.95"| MATCH[Near Duplicate]
JACC --> |"< 0.95"| SKIP[Different]
</div>
<p>The algorithm:</p>
<ol>
<li>
<p><strong>Extract bigrams</strong> from the normalized token sequence. For tokens <code>[A, B, C, D]</code>, the bigrams are <code>{AB, BC, CD}</code>.</p>
</li>
<li>
<p><strong>Compute Jaccard similarity</strong>: <code>|A ∩ B| / |A ∪ B|</code>. This measures how much the two token sequences overlap.</p>
</li>
<li>
<p><strong>Apply thresholds</strong>:</p>
<ul>
<li>Jaccard similarity >= <strong>0.95</strong> (95% overlap)</li>
<li>Minimum <strong>20 bigrams</strong> (prevents tiny functions from matching)</li>
<li>Type-set overlap >= <strong>0.85</strong> (85% of token types must be shared)</li>
</ul>
</li>
</ol>
<p>Why three thresholds? Each one catches a different failure mode:</p>
<ul>
<li>Jaccard alone: two 3-line functions with 2 shared tokens score 0.67 — false positive</li>
<li>Bigram count alone: a 2-token function matches everything — noise</li>
<li>Type-set overlap: catches cases where Jaccard is high only because of common tokens like <code>return</code> and <code>if</code></li>
</ul>
<h2 id="intra-file-duplication-chunk-hashing">Intra-File Duplication: Chunk Hashing</h2>
<p>Within a single file, duplication takes a different form: repeated code blocks that are typically 5-20 lines long. The intra-file detector uses a <strong>sliding window approach</strong>:</p>
<h3 id="the-algorithm">The Algorithm</h3>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph TB
FILE[Source File] --> SPLIT[Split into<br/>5-line chunks]
SPLIT --> NORM[Normalize each chunk<br/>strip comments, structure]
NORM --> HASH[Hash each chunk]
HASH --> GROUP[Group by hash]
GROUP --> FILTER[Filter: appears 2+ times<br/>with spacing]
FILTER --> DUPS[Duplicated Blocks]
</div>
<ol>
<li><strong>Split</strong> the file into overlapping 5-line chunks</li>
<li><strong>Normalize</strong> each chunk:
<ul>
<li>Strip comment lines</li>
<li>Strip structural-only lines (braces, blank lines)</li>
<li>Keep only "content" lines</li>
</ul>
</li>
<li><strong>Hash</strong> each normalized chunk</li>
<li><strong>Group</strong> chunks by hash</li>
<li><strong>Filter</strong>: keep only chunks that appear 2+ times with at least one non-matching line between them</li>
</ol>
<h3 id="why-5-lines">Why 5 Lines?</h3>
<p>The chunk size is a tradeoff:</p>
<ul>
<li><strong>3 lines</strong>: too many false positives — <code>if err != nil { return err }</code> appears everywhere in Go</li>
<li><strong>10 lines</strong>: too few detections — small duplicated blocks are missed</li>
<li><strong>5 lines</strong>: captures the most common duplication patterns (error handling, boilerplate, repeated logic)</li>
</ul>
<h3 id="normalization-matters">Normalization Matters</h3>
<p>Without normalization, these two chunks would not match:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// Chunk A</span></span>
<span class="giallo-l"><span>userName</span><span style="color: #94E2D5;"> :=</span><span style="color: #89B4FA;font-style: italic;"> getUser</span><span style="color: #9399B2;">(</span><span>id</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">if</span><span> userName</span><span style="color: #94E2D5;"> ==</span><span style="color: #A6E3A1;"> ""</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span> errors</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">New</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"not found"</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">processUser</span><span style="color: #9399B2;">(</span><span>userName</span><span style="color: #9399B2;">)</span></span></code></pre><pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// Chunk B</span></span>
<span class="giallo-l"><span>orderID</span><span style="color: #94E2D5;"> :=</span><span style="color: #89B4FA;font-style: italic;"> getOrder</span><span style="color: #9399B2;">(</span><span>id</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">if</span><span> orderID</span><span style="color: #94E2D5;"> ==</span><span style="color: #A6E3A1;"> ""</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span> errors</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">New</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"not found"</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">processOrder</span><span style="color: #9399B2;">(</span><span>orderID</span><span style="color: #9399B2;">)</span></span></code></pre>
<p>After normalization (identifiers hashed, strings replaced):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>IDENT == ""</span></span>
<span class="giallo-l"><span>return STRING_LIT</span></span>
<span class="giallo-l"><span>processIDENT(IDENT)</span></span></code></pre>
<p>They match — because the <strong>structure</strong> is identical, even though the <strong>names</strong> are different.</p>
<h2 id="the-duplicationdetector">The DuplicationDetector</h2>
<p>Both approaches feed into the <code>DuplicationDetector</code> (<code>src/detectors.rs</code>), which implements <code>SignalDetector</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">impl</span><span style="color: #F9E2AF;font-style: italic;"> SignalDetector</span><span style="color: #CBA6F7;"> for</span><span style="color: #F9E2AF;font-style: italic;"> DuplicationDetector</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> signal</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> StyleSignal</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> StyleSignal</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Duplication</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> count_violations</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> file</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">ParsedFile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // Cross-file: counted globally by CrossFileDupDetector</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // Intra-file: counted per file by IntraFileDupDetector</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> IntraFileDupDetector</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">check</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">len</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The cross-file detector runs in Phase 2 (before per-file signal detection) because it needs to see all files at once. The intra-file detector runs in Phase 3.</p>
<h2 id="performance-considerations">Performance Considerations</h2>
<p>Comparing every pair of functions is O(n^2) — unacceptable for large codebases. garbage-code-hunter reduces this to near-linear:</p>
<ol>
<li><strong>Hash grouping</strong> for exact duplicates: O(n)</li>
<li><strong>Bigram indexing</strong> for near-duplicates: only functions with similar bigram-set sizes are compared</li>
<li><strong>Early termination</strong>: if the type-set overlap is below 85%, skip the Jaccard computation</li>
</ol>
<p>For a codebase with 10,000 functions, the typical comparison count is under 100,000 — not 50,000,000.</p>
<h2 id="threshold-tuning">Threshold Tuning</h2>
<p>The thresholds (95% Jaccard, 20 bigrams, 85% type-set) were chosen to minimize false positives while catching real duplication:</p>
<table><thead><tr><th>Threshold</th><th>Too Low</th><th>Too High</th></tr></thead><tbody>
<tr><td>Jaccard (0.95)</td><td>Catches unrelated functions with shared tokens</td><td>Misses functions that differ in 10% of tokens</td></tr>
<tr><td>Bigrams (20)</td><td>Matches trivially small functions</td><td>Misses short but meaningful duplicates</td></tr>
<tr><td>Type-set (0.85)</td><td>Matches functions with same keywords but different logic</td><td>Misses functions with different structure</td></tr>
</tbody></table>
<p>These are configurable in <code>.garbage-code-hunter.toml</code> under the <code>[rules.duplication]</code> section.</p>
<hr />
<p><em>Next: <a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/06-duplication-algorithms/./07-scoring-model.md">The Scoring Model</a> — How to turn 10 signal scores into a single number between 0 and 100, and why logarithmic scaling is the right choice.</em></p>
The Scoring Model: From Qualitative Signals to a Single Number
2026-05-27T00:00:00+00:00
2026-05-27T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/07-scoring-model/
<h1 id="the-scoring-model">The Scoring Model</h1>
<blockquote>
<p><strong>Question</strong>: You have 10 signal detectors, each producing a violation count. Some signals are severe (deep nesting), some are noisy (magic numbers). How do you combine them into a single, fair score?</p>
</blockquote>
<h2 id="the-problem-with-linear-scoring">The Problem with Linear Scoring</h2>
<p>The simplest approach is linear addition: <code>score = sum(all violations)</code>. This fails because:</p>
<ol>
<li>
<p><strong>Scale mismatch.</strong> 100 magic numbers is common in large codebases. 100 deeply nested blocks is catastrophic. Linear scoring treats them equally.</p>
</li>
<li>
<p><strong>Project size bias.</strong> A 100,000-line project will have more violations than a 100-line project, even if the code quality is identical. Raw counts are not comparable.</p>
</li>
<li>
<p><strong>Diminishing severity.</strong> The 1st god function is alarming. The 100th is just more of the same systemic issue. Linear scoring says the second is 100x worse — but it is not.</p>
</li>
</ol>
<p>garbage-code-hunter solves all three problems with a <strong>two-tier logarithmic scoring model</strong>.</p>
<h2 id="the-two-tier-model">The Two-Tier Model</h2>
<p>The total score is the sum of two independent tiers (<code>src/scoring.rs:141-166</code>):</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph TB
ISSUES[All Issues] --> SEV{Severity?}
SEV --> |"Nuclear"| T1[Tier 1: Nuclear Score]
SEV --> |"Spicy"| T2[Tier 2: Noisy Density Score]
SEV --> |"Mild"| T2
T1 --> |"log2(1+n) * 8<br/>cap: 40"| TOTAL[Total Score<br/>0-100]
T2 --> |"log2(1+d) * 6<br/>cap: 60"| TOTAL
TOTAL --> QL[Quality Level]
</div><h3 id="tier-1-nuclear-score-high-confidence">Tier 1: Nuclear Score (High Confidence)</h3>
<p>Nuclear issues are <strong>high-confidence problems</strong> — when they appear, they are almost always real:</p>
<ul>
<li>Deeply nested blocks (>= 5 levels)</li>
<li>God functions (> 50 lines)</li>
<li>Bare <code>except</code> in Python</li>
<li><code>unsafe</code> blocks in Rust</li>
</ul>
<p>The formula:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/scoring.rs:155-156</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> n_score</span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> (</span><span>severity_distribution</span><span style="color: #94E2D5;">.</span><span>nuclear </span><span style="color: #CBA6F7;">as f64</span><span style="color: #94E2D5;"> +</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">log2</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #FAB387;"> 8</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> n_score</span><span style="color: #94E2D5;"> =</span><span> n_score</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">min</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">40</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">);</span></span></code></pre><table><thead><tr><th>Nuclear Count</th><th>Score</th><th>Interpretation</th></tr></thead><tbody>
<tr><td>0</td><td>0.0</td><td>No high-confidence issues</td></tr>
<tr><td>1</td><td>8.0</td><td>One nuclear issue detected</td></tr>
<tr><td>2</td><td>12.7</td><td>Two issues</td></tr>
<tr><td>5</td><td>20.7</td><td>Pattern forming</td></tr>
<tr><td>10</td><td>27.7</td><td>Systemic problem</td></tr>
<tr><td>30</td><td>39.6</td><td>Cap reached</td></tr>
</tbody></table>
<p>The logarithm ensures that going from 0 to 1 nuclear issue is a bigger jump than going from 30 to 31. This matches human intuition: the first deeply nested function is alarming; the 31st is just confirmation.</p>
<h3 id="tier-2-noisy-density-score">Tier 2: Noisy Density Score</h3>
<p>Spicy and Mild issues are <strong>noisy</strong> — magic numbers, naming violations, and <code>println</code> calls often have legitimate reasons:</p>
<ul>
<li>Magic numbers in UI layouts</li>
<li>Short variable names in mathematical formulas</li>
<li><code>println</code> in main functions</li>
</ul>
<p>The formula uses <strong>density</strong> (per 1000 lines) to normalize across project sizes:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/scoring.rs:158-164</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> k_lines</span><span style="color: #94E2D5;"> =</span><span> total_lines</span><span style="color: #CBA6F7;"> as f64</span><span style="color: #94E2D5;"> /</span><span style="color: #FAB387;"> 1000</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> noisy_density</span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> (</span><span>severity_distribution</span><span style="color: #94E2D5;">.</span><span>spicy </span><span style="color: #CBA6F7;">as f64</span><span style="color: #94E2D5;"> *</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">5</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> +</span><span> severity_distribution</span><span style="color: #94E2D5;">.</span><span>mild </span><span style="color: #CBA6F7;">as f64</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> /</span><span> k_lines</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> d_score</span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> (</span><span>noisy_density</span><span style="color: #94E2D5;"> +</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">log2</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #FAB387;"> 6</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> d_score</span><span style="color: #94E2D5;"> =</span><span> d_score</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">min</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">60</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">);</span></span></code></pre>
<p>Key details:</p>
<ul>
<li><strong>Spicy counts 1.5x</strong> vs Mild — slightly more reliable, but still noisy</li>
<li><strong>Density normalization</strong>: a 100-line file with 5 magic numbers scores the same as a 10,000-line file with 500</li>
<li><strong>Cap at 60</strong>: prevents noisy signals from overwhelming the score</li>
</ul>
<h3 id="the-final-score">The Final Score</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/scoring.rs:166</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> total_score</span><span style="color: #94E2D5;"> =</span><span> n_score</span><span style="color: #94E2D5;"> +</span><span> d_score</span><span style="color: #9399B2;">;</span></span></code></pre>
<p>Range: 0-100. But note the asymmetry:</p>
<ul>
<li>Nuclear cap: 40 (high confidence)</li>
<li>Noisy cap: 60 (volume matters more)</li>
<li>Nuclear issues can never exceed 40% of the total score</li>
</ul>
<p>This means a codebase with 0 nuclear issues but lots of mild noise can still score poorly — because high-volume noise is still a quality problem.</p>
<h2 id="quality-levels">Quality Levels</h2>
<p>The score maps to five quality levels (<code>src/scoring.rs:33-39</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> QualityLevel</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Excellent</span><span style="color: #9399B2;">,</span><span style="color: #9399B2;font-style: italic;"> // 0-20</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Good</span><span style="color: #9399B2;">,</span><span style="color: #9399B2;font-style: italic;"> // 21-40</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Average</span><span style="color: #9399B2;">,</span><span style="color: #9399B2;font-style: italic;"> // 41-60</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Poor</span><span style="color: #9399B2;">,</span><span style="color: #9399B2;font-style: italic;"> // 61-80</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Terrible</span><span style="color: #9399B2;">,</span><span style="color: #9399B2;font-style: italic;"> // 81+</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>These are not arbitrary. Each level corresponds to a different response:</p>
<table><thead><tr><th>Level</th><th>Score</th><th>Meaning</th><th>Action</th></tr></thead><tbody>
<tr><td>Excellent</td><td>0-20</td><td>Clean code</td><td>Ship it</td></tr>
<tr><td>Good</td><td>21-40</td><td>Minor issues</td><td>Fix when convenient</td></tr>
<tr><td>Average</td><td>41-60</td><td>Noticeable smells</td><td>Schedule cleanup</td></tr>
<tr><td>Poor</td><td>61-80</td><td>Significant problems</td><td>Prioritize fixes</td></tr>
<tr><td>Terrible</td><td>81+</td><td>Systemic failure</td><td>Halt and refactor</td></tr>
</tbody></table>
<h2 id="why-logarithmic">Why Logarithmic?</h2>
<p>The logarithm serves three purposes:</p>
<h3 id="1-diminishing-returns">1. Diminishing Returns</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>log2(1 + n):</span></span>
<span class="giallo-l"><span> n=0 → 0.0</span></span>
<span class="giallo-l"><span> n=1 → 1.0</span></span>
<span class="giallo-l"><span> n=10 → 3.5</span></span>
<span class="giallo-l"><span> n=100 → 6.7</span></span>
<span class="giallo-l"><span> n=1000 → 10.0</span></span></code></pre>
<p>Going from 0 to 1 issue is a 1.0 jump. Going from 100 to 101 is a 0.01 jump. This matches the "first one matters most" intuition.</p>
<h3 id="2-scale-invariance">2. Scale Invariance</h3>
<p>Logarithmic scoring makes the score comparable across project sizes. A 100-line file with 5 issues and a 10,000-line file with 500 issues get similar density scores.</p>
<h3 id="3-outlier-resistance">3. Outlier Resistance</h3>
<p>Without log, a single file with 1000 <code>println</code> calls would dominate the entire project score. With log, it contributes about 10 points — significant but not catastrophic.</p>
<h2 id="signal-level-scoring">Signal-Level Scoring</h2>
<p>Each individual signal also gets a normalized score (<code>src/signals.rs:82-86</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> violations_to_score</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;">count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> total_lines</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> f64</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> k_lines</span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> (</span><span>total_lines</span><span style="color: #CBA6F7;"> as f64</span><span style="color: #94E2D5;"> /</span><span style="color: #FAB387;"> 1000</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">max</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">001</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> density</span><span style="color: #94E2D5;"> =</span><span> count</span><span style="color: #CBA6F7;"> as f64</span><span style="color: #94E2D5;"> /</span><span> k_lines</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> ((</span><span>density</span><span style="color: #94E2D5;"> +</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">log2</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #FAB387;"> 6</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">min</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">25</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Each signal's score ranges from 0 to 25. This is used for:</p>
<ul>
<li><strong>Personality inference</strong> (Article 08): which signal dominates determines the archetype</li>
<li><strong>Category breakdown</strong>: naming, complexity, duplication, code-smells, student-code</li>
<li><strong>Trend tracking</strong>: signal scores over time show whether a specific smell is getting better or worse</li>
</ul>
<h2 id="category-scoring">Category Scoring</h2>
<p>Beyond the two-tier model, the scorer also computes per-category scores (<code>src/scoring.rs:117-139</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// Five categories, each log-scaled and capped at 20</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #94E2D5;"> &</span><span>cat_name</span><span style="color: #CBA6F7;"> in</span><span style="color: #94E2D5;"> &</span><span style="color: #9399B2;">[</span><span style="color: #A6E3A1;">"naming"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "complexity"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "duplication"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "code-smells"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "student-code"</span><span style="color: #9399B2;">] {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> cat_count</span><span style="color: #94E2D5;"> =</span><span> category_counts</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">get</span><span style="color: #9399B2;">(</span><span>cat_name</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">copied</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">unwrap_or</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> cat_density</span><span style="color: #94E2D5;"> =</span><span> cat_count</span><span style="color: #CBA6F7;"> as f64</span><span style="color: #94E2D5;"> /</span><span> k_lines</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> cat_score</span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> ((</span><span>cat_density</span><span style="color: #94E2D5;"> +</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">log2</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #FAB387;"> 6</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">min</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">20</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span> category_scores</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">insert</span><span style="color: #9399B2;">(</span><span>cat_name</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">(),</span><span> cat_score</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>These are informational — they do not affect the total score, but they help developers understand <strong>which dimension</strong> of code quality needs attention.</p>
<h2 id="severity-classification">Severity Classification</h2>
<p>Not all issues are equal. garbage-code-hunter classifies issues into three severity levels (<code>src/analyzer.rs:39-44</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> Severity</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Mild</span><span style="color: #9399B2;">,</span><span style="color: #9399B2;font-style: italic;"> // Minor issues (naming, magic numbers)</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Spicy</span><span style="color: #9399B2;">,</span><span style="color: #9399B2;font-style: italic;"> // Medium issues (debug calls, shallow nesting)</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Nuclear</span><span style="color: #9399B2;">,</span><span style="color: #9399B2;font-style: italic;"> // Serious issues (deep nesting, god functions)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The classification is not per-rule — it is per-signal. A <code>NestedHell</code> violation is always Nuclear because deep nesting is always a structural problem. A <code>NamingChaos</code> violation is always Mild because bad names are annoying but not dangerous.</p>
<h2 id="putting-it-all-together">Putting It All Together</h2>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph TB
subgraph "Per-File"
F1[File A] --> IR1[StyleIR]
F2[File B] --> IR2[StyleIR]
F3[File N] --> IRN[StyleIR]
end
subgraph "Detection"
IR1 --> DET[10 Detectors]
IR2 --> DET
IRN --> DET
DET --> FIND[StyleFindings]
end
subgraph "Scoring"
FIND --> SEV[Classify Severity]
SEV --> T1[Tier 1: Nuclear<br/>log2(1+n)*8]
SEV --> T2[Tier 2: Noisy<br/>log2(1+d)*6]
T1 --> TOTAL[Total: 0-100]
T2 --> TOTAL
TOTAL --> QL[Quality Level]
end
</div>
<hr />
<p><em>Next: <a href="https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/07-scoring-model/./08-fun-side-roasts-and-personality.md">The Fun Side: Roasts, Personality, and the Tool Belt</a> — Why code analysis should be entertaining, and how garbage-code-hunter makes developers actually read their reports.</em></p>
The Fun Side: Roasts, Personality, and the Tool Belt
2026-05-27T00:00:00+00:00
2026-05-27T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/garbage-code-hunter/08-fun-side-roasts-and-personality/
<h1 id="the-fun-side-roasts-personality-and-the-tool-belt">The Fun Side: Roasts, Personality, and the Tool Belt</h1>
<blockquote>
<p><strong>Question</strong>: Developers ignore code quality reports. They close linter warnings without reading them. They treat CI checks as gatekeepers, not advisors. How do you make them actually care?</p>
</blockquote>
<h2 id="the-engagement-problem">The Engagement Problem</h2>
<p>Every code quality tool faces the same problem: <strong>developers do not read the output.</strong></p>
<p>ESLint output scrolls past in terminal. SonarQube dashboards collect dust. Code review comments about "consider using const instead of let" are resolved without thought.</p>
<p>The problem is not that the tools are wrong. The problem is that they are <strong>boring</strong>. They speak in the language of compliance, not in the language of developers.</p>
<p>garbage-code-hunter takes a different approach: <strong>make it entertaining.</strong></p>
<h2 id="14-tools-not-just-code-analysis">14 Tools, Not Just Code Analysis</h2>
<p>garbage-code-hunter is not a single tool — it is a <strong>tool belt</strong> of 14 independent analyzers, each targeting a different aspect of project health:</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph TB
subgraph "Code Quality"
A[analyze<br/>10 signal detectors]
D[danger_zone<br/>security patterns]
end
subgraph "Git & Collaboration"
C[commit_roaster<br/>commit message quality]
P[pr_title_hunter<br/>PR title quality]
T[team_roast<br/>team patterns]
end
subgraph "Dependencies & Debt"
DE[deps_shamer<br/>dependency shame]
INV[debt_invoice<br/>technical debt invoice]
DEC[decay<br/>code decay rate]
end
subgraph "Personality & Fun"
PER[personality<br/>coder archetype]
RAD[radar<br/>skill radar chart]
BADGE[badge<br/>achievement badges]
end
subgraph "Reporting"
AUT[autopsy<br/>post-mortem report]
LW[last_words<br/>commit message analysis]
CI[ci_bot<br/>CI integration]
TREND[trend<br/>quality over time]
end
</div>
<p>Each tool runs independently and produces its own score. The <code>scan</code> command runs all 14 in parallel:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/main.rs:153-436 (conceptual)</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;">std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">thread</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">scope</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>s</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> s</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">spawn</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">||</span><span style="color: #F9E2AF;"> commit_roaster</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">run</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span> s</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">spawn</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">||</span><span style="color: #F9E2AF;"> deps_shamer</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">run</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span> s</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">spawn</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">||</span><span style="color: #F9E2AF;"> pr_title_hunter</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">run</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ... 11 more tools</span></span>
<span class="giallo-l"><span style="color: #9399B2;">});</span></span></code></pre><h2 id="the-commit-roaster">The Commit Roaster</h2>
<p>The <code>commit_roaster</code> analyzes git commit messages and rates their quality. It checks for:</p>
<ul>
<li><strong>Vague messages</strong>: "fix stuff", "update", "changes"</li>
<li><strong>Missing context</strong>: no issue reference, no explanation of why</li>
<li><strong>Too long</strong>: commit messages should be a tweet, not an essay</li>
<li><strong>Too short</strong>: "fix" is not a commit message</li>
</ul>
<p>The output is not a score — it is a <strong>roast</strong>:</p>
<blockquote>
<p>"Your last 50 commits have the literary quality of a shopping list. 'fix bug' is not a commit message — it is a cry for help."</p>
</blockquote>
<h2 id="the-personality-system">The Personality System</h2>
<p>The <code>personality</code> tool maps signal score patterns to developer archetypes (<code>src/signals.rs:344-382</code>):</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> infer_personality_type</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> &</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #F9E2AF;font-style: italic;"> str</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> dup</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">score</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">StyleSignal</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Duplication</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> panic</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">score</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">StyleSignal</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">PanicAddiction</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> naming</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">score</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">StyleSignal</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">NamingChaos</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> nested</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">score</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">StyleSignal</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">NestedHell</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> hotfix</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">score</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">StyleSignal</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">HotfixCulture</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> over_eng</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">score</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">StyleSignal</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">OverEngineering</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> dup</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 12</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #94E2D5;"> &&</span><span> dup</span><span style="color: #94E2D5;"> >=</span><span> panic</span><span style="color: #94E2D5;"> &&</span><span> dup</span><span style="color: #94E2D5;"> >=</span><span> naming</span><span style="color: #94E2D5;"> &&</span><span> dup</span><span style="color: #94E2D5;"> >=</span><span> nested</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> "The Copy-Paste Artist"</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> panic</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 12</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #94E2D5;"> &&</span><span> panic</span><span style="color: #94E2D5;"> >=</span><span> dup</span><span style="color: #94E2D5;"> &&</span><span> panic</span><span style="color: #94E2D5;"> >=</span><span> naming</span><span style="color: #94E2D5;"> &&</span><span> panic</span><span style="color: #94E2D5;"> >=</span><span> nested</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> "The YOLO Engineer"</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> nested</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 12</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #94E2D5;"> &&</span><span> nested</span><span style="color: #94E2D5;"> >=</span><span> naming</span><span style="color: #94E2D5;"> &&</span><span> nested</span><span style="color: #94E2D5;"> >=</span><span> hotfix</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> "The Trait Wizard"</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> naming</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 12</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #94E2D5;"> &&</span><span> naming</span><span style="color: #94E2D5;"> >=</span><span> nested</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> "The Legacy Necromancer"</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> hotfix</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 12</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> "The Hotfix Mercenary"</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> dup</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 6</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #94E2D5;"> &&</span><span> panic</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 6</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> "The Startup Survivor"</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span style="color: #9399B2;"> (</span><span>naming</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 6</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #94E2D5;"> &&</span><span> nested</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 6</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> ||</span><span> over_eng</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 12</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> "The Academic Wizard"</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "The Enterprise Bureaucrat"</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The archetypes:</p>
<table><thead><tr><th>Archetype</th><th>Dominant Signal</th><th>Description</th></tr></thead><tbody>
<tr><td>The Copy-Paste Artist</td><td>Duplication</td><td>Ctrl+C, Ctrl+V is their IDE</td></tr>
<tr><td>The YOLO Engineer</td><td>PanicAddiction</td><td><code>.unwrap()</code> everywhere, hope for the best</td></tr>
<tr><td>The Trait Wizard</td><td>NestedHell</td><td>Callbacks within callbacks within callbacks</td></tr>
<tr><td>The Legacy Necromancer</td><td>NamingChaos</td><td><code>x</code>, <code>temp</code>, <code>data2</code>, <code>final_final_v3</code></td></tr>
<tr><td>The Hotfix Mercenary</td><td>HotfixCulture</td><td><code>println!</code> in production, <code>// TODO: remove this</code></td></tr>
<tr><td>The Startup Survivor</td><td>Duplication + Panic</td><td>Move fast, break things, copy-paste the fixes</td></tr>
<tr><td>The Academic Wizard</td><td>Naming + Nested</td><td>Over-engineered abstractions with cryptic names</td></tr>
<tr><td>The Enterprise Bureaucrat</td><td>(default)</td><td>The code works. It is not pretty. It ships.</td></tr>
</tbody></table>
<p>This is not just entertainment — it is a <strong>communication tool</strong>. "You are a Copy-Paste Artist" is more memorable than "Your Duplication score is 14.2."</p>
<h2 id="llm-powered-roasts">LLM-Powered Roasts</h2>
<p>For users who want more creative feedback, garbage-code-hunter supports <strong>LLM-generated roasts</strong> via Ollama or any OpenAI-compatible API:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// src/config.rs</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> AppConfig</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Local</span><span style="color: #9399B2;"> {</span><span style="color: #9399B2;font-style: italic;"> /* hardcoded roasts */</span><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Llm</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> provider</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> LlmProvider</span><span style="color: #9399B2;">,</span><span style="color: #9399B2;font-style: italic;"> // Ollama or OpenAI-compatible</span></span>
<span class="giallo-l"><span> model</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> endpoint</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>When LLM mode is enabled, the tool sends the analysis results to the LLM and asks it to generate a personalized, humorous critique. The LLM sees:</p>
<ul>
<li>The quality score and level</li>
<li>The top 3 worst signals</li>
<li>The personality archetype</li>
<li>Specific code examples from the analysis</li>
</ul>
<p>The result is a roast that is specific to YOUR code, not a generic template.</p>
<h2 id="the-badge-system">The Badge System</h2>
<p>The <code>badge</code> tool generates achievement badges for project milestones:</p>
<ul>
<li>"Zero Nuclear" — no high-confidence issues</li>
<li>"Naming Conformist" — naming score below 5</li>
<li>"Copy-Paste Free" — zero duplication</li>
<li>"The Refactorer" — improved score by 20+ points</li>
</ul>
<p>These can be displayed in README badges, CI comments, or team dashboards.</p>
<h2 id="the-debt-invoice">The Debt Invoice</h2>
<p>The <code>debt_invoice</code> tool calculates the <strong>estimated cost</strong> of technical debt:</p>
<ul>
<li>Each issue is assigned an estimated fix time</li>
<li>Fix time is multiplied by an average developer hourly rate</li>
<li>The output is an "invoice" showing how much the technical debt would cost to fix</li>
</ul>
<blockquote>
<p>"Your technical debt invoice: $47,200. This includes 23 god functions ($18,400), 156 magic numbers ($7,800), and 89 TODO markers ($21,000 at $250/hour for the emotional labor of reading them)."</p>
</blockquote>
<h2 id="extensibility-adding-new-tools">Extensibility: Adding New Tools</h2>
<p>Adding a new tool to the belt is straightforward:</p>
<ol>
<li><strong>Create a module</strong> with a <code>run()</code> function</li>
<li><strong>Add CLI args</strong> in <code>src/args.rs</code></li>
<li><strong>Add dispatch</strong> in <code>src/main.rs</code></li>
<li><strong>Register in scan</strong> if it should run with <code>garbage-code-hunter scan</code></li>
</ol>
<p>The tool can use any of the existing infrastructure:</p>
<ul>
<li><code>TreeSitterEngine</code> for parsing</li>
<li><code>LanguageAdapter</code> for language-aware analysis</li>
<li><code>StyleIr</code> for signal data</li>
<li><code>Reporter</code> for output formatting</li>
<li><code>RoastProvider</code> for LLM integration</li>
</ul>
<h2 id="adding-a-new-language">Adding a New Language</h2>
<p>Adding a new language (e.g., Kotlin) requires:</p>
<ol>
<li>Add <code>tree-sitter-kotlin</code> dependency</li>
<li>Register parser in <code>parsers.rs</code></li>
<li>Add <code>Language::Kotlin</code> variant</li>
<li>Implement <code>KotlinAdapter</code> with <code>query_patterns()</code></li>
<li>Register in <code>adapter_for()</code></li>
</ol>
<p><strong>Zero changes to detectors, scoring, or tools.</strong> The new language automatically works with all 14 tools.</p>
<h2 id="adding-a-new-detector">Adding a New Detector</h2>
<p>Adding a new detector (e.g., <code>MagicNumberDetector</code>) requires:</p>
<ol>
<li>Add <code>StyleSignal::MagicNumber</code> to the enum</li>
<li>Implement <code>SignalDetector</code> trait</li>
<li>Register in the detector list</li>
</ol>
<p><strong>Zero changes to adapters.</strong> The detector reads from <code>StyleIr</code>, which already contains <code>magic_number_count</code>.</p>
<h2 id="the-philosophy-analysis-should-be-fun">The Philosophy: Analysis Should Be Fun</h2>
<p>garbage-code-hunter is built on a belief: <strong>developers will only care about code quality if the feedback is engaging.</strong></p>
<p>A dry report saying "47 violations found" does not change behavior. A roast saying "Your code reads like a horror novel — chapter 1: The God Function" makes developers laugh, then think, then fix.</p>
<p>The humor is not decoration — it is the <strong>delivery mechanism</strong> for genuine quality insights. Behind every joke is a real signal. Behind every personality archetype is a real pattern in the code.</p>
<div class="mermaid-direct" data-series="garbage-code-hunter" data-name="graph.mmd">
graph LR
DRY[Traditional<br/>Linter Output] --> IGNORE[Ignored]
FUN[garbage-code-hunter<br/>Roast + Signal] --> LAUGH[Laugh]
LAUGH --> THINK[Think]
THINK --> FIX[Fix]
</div>
<p>The 10 signal detectors provide the substance. The roasts, personalities, and badges provide the engagement. Together, they make code quality analysis something developers actually want to run.</p>
<hr />
<p><em>This concludes the series. To get started:</em></p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">cargo</span><span style="color: #A6E3A1;"> install garbage-code-hunter</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">garbage-code-hunter</span><span style="color: #A6E3A1;"> scan ./your-project</span></span></code></pre>
<p><em>Find out what kind of developer you are.</em></p>
OmniScope Deep-Dive Series: From LLVM IR to Cross-Language Ownership Auditing
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/00-series-index/
<h1 id="omniscope-deep-dive-series-from-llvm-ir-to-cross-language-ownership-auditing">OmniScope Deep-Dive Series: From LLVM IR to Cross-Language Ownership Auditing</h1>
<p>This series explains what OmniScope does, how data moves through the project, and which modules make which decisions. It is written for readers interested in LLVM IR, static analysis, FFI safety, and cross-language boundaries across Rust, C, Zig, Go, and C++.</p>
<p>OmniScope focuses on one concrete question: <strong>after ownership, lifetime, and deallocation protocols cross a language boundary, can a static analyzer recover enough semantics to produce reviewable findings?</strong></p>
<p>In one sentence: OmniScope reads LLVM IR (<code>.ll</code> / <code>.bc</code>), turns calls, pointer flows, allocation/free events, language boundaries, and unsafe regions into queryable facts, then reports high-risk paths related to FFI, ownership, and lifetime through text, JSON, or SARIF.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[LLVM IR input] --> B[Module loading]
B --> C[PassContext]
C --> D[CallGraph / CrossLangEdge]
C --> E[MemoryGraph]
C --> F[Zone / Registry]
D --> G[DangerSurface]
E --> G
F --> G
G --> H[Ownership / Lifetime / FFI rules]
H --> I[Issue / JSON / SARIF]
</div><h2 id="articles">Articles</h2>
<ol>
<li><a href="https://Timwood0x10.github.io/myblog/log/OmniScope/01-what-is-omniscope/">Why OmniScope analyzes cross-language safety at the LLVM IR layer</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/OmniScope/02-cli-to-pipeline/">Lifecycle of an analysis run: CLI, IRLoader, Pipeline, and output</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/OmniScope/03-pass-system/">Inside the Pass system: dependency ordering, shared context, and graceful degradation</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/OmniScope/04-zone-and-registry/">Zone Classification and Semantic Registry: avoiding blacklist-style reporting</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/OmniScope/05-memory-graph-danger-surface/">MemoryGraph and DangerSurface: from pointer facts to risk paths</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/OmniScope/06-rust-ffi-auditor/">Rust FFI Auditor: reconstructing and checking cross-language ownership protocols</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/OmniScope/07-output-and-integration/">Reporting pipeline: Issue objects, JSON, SARIF, and engineering integration</a></li>
</ol>
<h2 id="source-reading-map">Source Reading Map</h2>
<ul>
<li>Entry and output: <code>src/main.zig:73</code>, <code>src/main.zig:153</code>, <code>src/main.zig:171</code>, <code>src/main.zig:207</code></li>
<li>Pipeline: <code>src/pipeline/pipeline.zig:27</code>, <code>src/pipeline/pipeline.zig:66</code>, <code>src/pipeline/pipeline.zig:223</code></li>
<li>Pass system: <code>src/pass/manager.zig:61</code>, <code>src/pass/manager.zig:193</code>, <code>src/pass/pass.zig:192</code></li>
<li>Semantic layer: <code>src/semantics/zone_classifier.zig:24</code>, <code>src/registry/semantic_registry.zig:90</code></li>
<li>Risk paths: <code>src/pass/pass.zig:866</code>, <code>src/semantics/memory_graph.zig:892</code>, <code>src/pass/analysis/danger_surface.zig:37</code></li>
<li>Rust FFI: <code>src/pass/analysis/rust_ffi_auditor.zig:63</code>, <code>src/pass/analysis/rust_ffi_auditor.zig:180</code></li>
</ul>
<h2 id="how-to-read">How to Read</h2>
<p>Read this as a problem chain: first identify where cross-language security auditing gets difficult, then look at why common approaches fall short, and finally see how OmniScope turns those constraints into LLVM IR input, passes, MemoryGraph, Zone, Registry, Rust FFI auditing, and structured output.</p>
<p>Each article follows one implementation path: how input becomes analysis context, how passes exchange facts, how Zone and Registry reduce noise, how MemoryGraph models pointer flow, and how Rust FFI ownership rules are checked.</p>
<p>This series is written as technical explainer content. It starts from implementation constraints, data flow, and module boundaries instead of abstract definitions. Each article follows the same pattern:</p>
<ol>
<li>State the engineering problem first.</li>
<li>Show the source entry points, not just the abstraction.</li>
<li>Break down the data structures that hold analysis facts.</li>
<li>Explain why the design exists and where it breaks down.</li>
</ol>
<p>If you want to read the code first, start here:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>main.zig</span></span>
<span class="giallo-l"><span> └─ runSingleFileAnalysis / runModulePipeline / emitOutput</span></span>
<span class="giallo-l"><span>pipeline.zig</span></span>
<span class="giallo-l"><span> └─ Pipeline.init / Pipeline.run</span></span>
<span class="giallo-l"><span>pass.zig</span></span>
<span class="giallo-l"><span> └─ PassContext / addIssue / isOnDangerPathFull</span></span>
<span class="giallo-l"><span>manager.zig</span></span>
<span class="giallo-l"><span> └─ resolveDependencies / run</span></span>
<span class="giallo-l"><span>semantics/memory_graph.zig</span></span>
<span class="giallo-l"><span> └─ MemoryGraph / isOnDangerPath</span></span>
<span class="giallo-l"><span>pass/analysis/danger_surface.zig</span></span>
<span class="giallo-l"><span> └─ DangerSurfacePass.run</span></span>
<span class="giallo-l"><span>pass/analysis/rust_ffi_auditor.zig</span></span>
<span class="giallo-l"><span> └─ auditFunction / Rust-specific rules / universal FFI rules</span></span></code></pre>
<p>The real architecture is not just "uses LLVM IR". It is the way OmniScope turns low-level IR facts into three reusable semantic layers: <code>PassContext</code>, <code>MemoryGraph</code>, and <code>DangerSurface</code>.</p>
Why OmniScope Analyzes Cross-Language Safety at the LLVM IR Layer
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/01-what-is-omniscope/
<h1 id="why-omniscope-analyzes-cross-language-safety-at-the-llvm-ir-layer">Why OmniScope Analyzes Cross-Language Safety at the LLVM IR Layer</h1>
<p>Rust ownership, Zig allocators, Go GC, and C++ RAII are language-level safety models. Across FFI, those models are lowered into ABI-level facts: functions, pointers, integers, layouts, and calling conventions. OmniScope works at this layer because many cross-language issues are difficult to cover from a single language AST alone.</p>
<h2 id="start-with-the-problem-language-guarantees-stop-at-the-boundary">Start with the problem: language guarantees stop at the boundary</h2>
<p>Inside one language, many guarantees come from the compiler or runtime: Rust tracks ownership, Go runs a GC, Zig exposes allocators explicitly, and C++ relies on RAII. Across FFI, those guarantees do not travel as-is. What remains at the boundary is usually a symbol, a few raw pointers, and a calling convention.</p>
<p>That creates the audit problem: the risky part is often a broken protocol, not a single dangerous API. After Rust turns a <code>Box</code> into a raw pointer, does C store it? Who releases it? Is the same allocator family used? Looking at only one side often cannot answer the full question.</p>
<h2 id="where-common-approaches-fall-short">Where common approaches fall short</h2>
<p>Single-language AST tools understand their own syntax but cannot see the full protocol on the other side. Dangerous-function lists can find <code>free</code> or <code>memcpy</code>, but they do not know where a pointer came from or where it flows. A call graph can show an edge, but not whether that edge carries ownership, a borrow, or ordinary data.</p>
<p>OmniScope’s choice follows from that gap: move down to LLVM IR, the layer where many languages meet, then reconstruct part of the missing semantics from symbols, calls, pointer flow, allocation/free events, and FFI boundaries.</p>
<h2 id="the-issue-is-not-free-it-is-the-lost-deallocation-protocol">The issue is not <code>free</code>; it is the lost deallocation protocol</h2>
<p>A common Rust/C boundary pattern is: Rust exposes a pointer through <code>Box::into_raw</code>, then C stores or releases that pointer. In Rust, <code>into_raw</code> is an explicit ownership transfer. In C, <code>free(ptr)</code> is an ordinary deallocation. The problem is that no single compiler verifies the full protocol across both sides.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
sequenceDiagram
participant R as Rust ownership model
participant ABI as C ABI / LLVM IR
participant C as C manual memory
R->>ABI: Box::into_raw becomes a raw pointer
ABI->>C: extern C argument passing
C->>C: store / free / callback use
C-->>R: Rust cannot verify the C-side protocol
</div>
<p>At the IR layer, source syntax is gone, but external declarations, call/invoke instructions, allocas, loads/stores, bitcasts, symbol names, and some debug information may remain. The analyzer tries to recover enough ownership and lifetime semantics from these facts.</p>
<h2 id="what-omniscope-actually-analyzes">What OmniScope actually analyzes</h2>
<p>OmniScope’s entry point consumes LLVM IR files such as <code>.ll</code> and <code>.bc</code>, not source directories. Argument parsing starts at <code>src/main.zig:73</code>, the main entry is <code>src/main.zig:567</code>, and single-module analysis is driven by <code>runModulePipeline</code> at <code>src/main.zig:171</code>.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[Source languages: Rust / C / Zig / Go / C++] --> B[Compiler emits LLVM IR]
B --> C[IRLoader obtains ModuleRef]
C --> D[Pipeline.setModule]
D --> E[PassContext.module]
E --> F[Passes iterate functions, blocks, instructions]
</div>
<p>This also defines the limits. OmniScope can inspect facts that remain in IR and can use symbols and debug information when available. Heavy optimization, missing symbols, or wrapper-heavy code may reduce the amount of recoverable semantics.</p>
<h2 id="it-is-not-a-dangerous-function-blacklist">It is not a dangerous-function blacklist</h2>
<p><code>src/registry/semantic_registry.zig:3</code> describes the registry as a function-semantics knowledge base for FFI boundary analysis, not a simple blacklist. <code>src/registry/semantic_registry.zig:8</code> also notes that the same function may carry different risk depending on context.</p>
<p>For example:</p>
<ul>
<li>A local C <code>free</code> may be a normal lifetime endpoint.</li>
<li>A Rust <code>Box</code> pointer released by C may indicate allocator mismatch or ownership protocol breakage.</li>
<li>Rust <code>as_ptr</code> used locally may be benign, while passing it to FFI and storing it may create a dangling pointer.</li>
</ul>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[Function call] --> B{Crosses language boundary?}
B -->|No| C[Local semantics]
B -->|Yes| D[FFI semantics]
C --> E[May be lower priority]
D --> F[Check ownership / lifetime / allocator context]
</div><h2 id="main-source-level-pillars">Main source-level pillars</h2>
<p>The implementation is organized around shared analysis structures:</p>
<ul>
<li><code>PassContext</code>: shared state for passes, defined at <code>src/pass/pass.zig:192</code>.</li>
<li><code>cross_lang_edges</code>: cross-language call edges.</li>
<li><code>MemoryGraph</code>: memory objects, frees, call arguments, returns, and alias relations.</li>
<li><code>ZoneKind</code>: <code>safe</code>, <code>unsafe</code>, <code>ffi</code>, <code>runtime_internal</code>, and <code>unknown</code>, defined at <code>src/semantics/zone_classifier.zig:24</code>.</li>
<li><code>SemanticRegistry</code>: layered function semantics, looked up through <code>src/registry/semantic_registry.zig:90</code>.</li>
</ul>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TB
A[LLVM IR facts] --> B[PassContext]
B --> C[CrossLangEdge]
B --> D[MemoryGraph]
B --> E[Zone cache]
B --> F[Registry cache]
C --> G[DangerSurface]
D --> G
E --> G
F --> G
G --> H[Issue]
</div><h2 id="practical-limits">Practical limits</h2>
<p>OmniScope performs static recovery and risk classification; it is not a runtime proof system. It depends on:</p>
<ul>
<li>Enough call and symbol information remaining in IR.</li>
<li>Recognizable cross-language declarations and call sites.</li>
<li>MemoryGraph coverage for relevant pointer flows.</li>
<li>Zone and Registry rules that cover the project’s FFI patterns.</li>
</ul>
<p>A careful description should avoid absolute detection claims. A more accurate framing is: OmniScope reconstructs queryable facts about ownership, lifetime, and allocator protocols at language boundaries, then uses risk-path filtering to prioritize findings.</p>
<h2 id="source-level-view-omniscope-analyzes-facts-not-text">Source-level view: OmniScope analyzes facts, not text</h2>
<p>From the code, OmniScope does not treat the source language as the primary abstraction. <code>Pipeline.run</code> builds a <code>PassContext</code> that holds the module, fact store, query engine, data flow graph, memory graph, cross-language edges, registry cache, and zone cache. Source code is first reduced to LLVM IR, then lifted back into analysis facts.</p>
<p>The core initialization lives around <code>src/pipeline/pipeline.zig:66</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">var</span><span> ctx</span><span style="color: #94E2D5;"> =</span><span> PassContext{</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>allocator</span><span style="color: #94E2D5;"> =</span><span> self</span><span style="color: #94E2D5;">.</span><span>allocator</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>module</span><span style="color: #94E2D5;"> =</span><span> self</span><span style="color: #94E2D5;">.</span><span>module</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>fact_store</span><span style="color: #94E2D5;"> =</span><span> self</span><span style="color: #94E2D5;">.</span><span>fact_store</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>query_engine</span><span style="color: #94E2D5;"> =</span><span> self</span><span style="color: #94E2D5;">.</span><span>query_engine</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>data_flow_graph</span><span style="color: #94E2D5;"> = &</span><span>self</span><span style="color: #94E2D5;">.</span><span>data_flow_graph</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>cross_lang_edges</span><span style="color: #94E2D5;"> =</span><span> std</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">ArrayList</span><span>(CrossLangEdge)</span><span style="color: #94E2D5;">.</span><span>empty</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>global_alloc_tracker</span><span style="color: #94E2D5;"> =</span><span> GlobalAllocTracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">init</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>memory_graph</span><span style="color: #94E2D5;"> =</span><span style="color: #CBA6F7;"> try</span><span> MemoryGraph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">init</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>danger_surface_relevant</span><span style="color: #94E2D5;"> =</span><span> std</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">AutoHashMap</span><span>(</span><span style="color: #CBA6F7;">u64</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> void</span><span>)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">init</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>ffi_auto_relevant</span><span style="color: #94E2D5;"> =</span><span> std</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">AutoHashMap</span><span>(</span><span style="color: #CBA6F7;">u64</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> void</span><span>)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">init</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>relevant_functions</span><span style="color: #94E2D5;"> =</span><span> std</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">AutoHashMap</span><span>(</span><span style="color: #CBA6F7;">u64</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> void</span><span>)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">init</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>CallSiteIndex</span><span style="color: #94E2D5;"> =</span><span> CallSiteIndex</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">init</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span>}</span><span style="color: #9399B2;">;</span></span></code></pre>
<p>That snippet shows the real design center: OmniScope does not let each pass rescan IR and rediscover the same facts. It establishes a shared fact space first. <code>cross_lang_edges</code> models language boundaries, <code>memory_graph</code> models pointer facts, <code>danger_surface_relevant</code> narrows the analysis to risk paths, and <code>CallSiteIndex</code> turns repeated module scans into indexed lookups.</p>
<h2 id="how-it-works-recover-enough-semantics-after-the-language-model-is-gone">How it works: recover enough semantics after the language model is gone</h2>
<p>LLVM IR does not preserve Rust’s borrow checker, Zig’s allocator types, or C++ RAII as language constructs. OmniScope does not pretend otherwise. Instead, it uses a layered recovery model:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>IR facts</span></span>
<span class="giallo-l"><span> -> function / call / load / store / alloca / return / bitcast</span></span>
<span class="giallo-l"><span>name and debug hints</span></span>
<span class="giallo-l"><span> -> Rust mangling / allocator names / extern patterns / registry patterns</span></span>
<span class="giallo-l"><span>semantic facts</span></span>
<span class="giallo-l"><span> -> zone / function semantics / cross-lang edge / memory graph node</span></span>
<span class="giallo-l"><span>audit facts</span></span>
<span class="giallo-l"><span> -> danger path / ownership mismatch / borrow escape / issue</span></span></code></pre>
<p>This is a pragmatic static analysis design. It does not aim for formal proof; it aims to extract high-value signals reliably from real-world IR. That is why the code keeps <code>confidence</code>, <code>reason</code>, and <code>classification</code> fields. They are not UI decoration. They acknowledge that analysis facts have different strengths.</p>
<h2 id="the-line-between-omniscope-and-blacklist-scanners">The line between OmniScope and blacklist scanners</h2>
<p>A blacklist scanner can only say "this dangerous API appears". OmniScope asks a narrower question: does this pointer cross an FFI boundary, come from unsafe code, violate ownership across languages, or flow through alias paths to a boundary?</p>
<p>That is why <code>MemoryGraph.isOnDangerPath</code> becomes the core question in later articles. It changes the unit of analysis from "dangerous function present" to "pointer on a dangerous path". That is the fundamental difference between OmniScope and a rule list.</p>
Lifecycle of an Analysis Run: CLI, IRLoader, Pipeline, and Output
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/02-cli-to-pipeline/
<h1 id="lifecycle-of-an-analysis-run-cli-irloader-pipeline-and-output">Lifecycle of an Analysis Run: CLI, IRLoader, Pipeline, and Output</h1>
<p>This article follows one OmniScope run from command-line arguments to structured findings. The important part is how the code turns user input into a <code>PassContext</code>, runs analysis passes, and emits reports.</p>
<h2 id="start-with-the-problem-an-analyzer-run-is-not-one-rule-execution">Start with the problem: an analyzer run is not one rule execution</h2>
<p>A cross-language audit begins at the CLI, but the real job is a sequence: load <code>.ll</code> or <code>.bc</code>, create an LLVM module, identify languages and boundaries, run multiple passes, collect shared facts, and emit results for a terminal, CI system, or security platform.</p>
<p>If those steps are mixed together, each rule ends up parsing input, scanning calls, and printing reports on its own. OmniScope separates them into CLI parsing, IR loading, Pipeline orchestration, PassManager execution, and output formatting so the whole run becomes traceable.</p>
<h2 id="omniscope-s-entry-point-pipeline-owns-the-analysis-lifecycle">OmniScope’s entry point: Pipeline owns the analysis lifecycle</h2>
<p>The CLI describes how the user wants to run. IRLoader turns input into a module. Pipeline is where the module, shared context, passes, indexes, and output path are assembled. Later structures such as <code>PassContext</code>, <code>CallSiteIndex</code>, <code>MemoryGraph</code>, and <code>GlobalAllocTracker</code> appear naturally from this lifecycle.</p>
<h2 id="config-defines-the-external-shape-of-the-run">Config defines the external shape of the run</h2>
<p><code>Config</code> is defined at <code>src/main.zig:24</code>. It stores input files, output format, output path, visualization flag, user-code focus, FFI-only mode, and stdlib inclusion. Argument parsing starts at <code>src/main.zig:73</code>.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[argv] --> B[parseArgs]
B --> C[Config]
C --> D[input_files]
C --> E[output_format: text/json/sarif]
C --> F[visualize]
C --> G[focus_user_code / ffi_only / include_stdlib]
</div>
<p>One implementation detail matters: the CLI parses analysis intent, but the exact effect depends on individual pass implementations. For example, some noise-reduction options are instantiated inside specific passes.</p>
<h2 id="runmodulepipeline-is-the-analysis-loop"><code>runModulePipeline</code> is the analysis loop</h2>
<p><code>runModulePipeline</code> is located at <code>src/main.zig:171</code>. It initializes the Pipeline, attaches the LLVM module, registers passes, runs static analysis, and collects issues.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
sequenceDiagram
participant Main as main.zig
participant Loader as IRLoader
participant Pipe as Pipeline
participant PM as PassManager
Main->>Pipe: Pipeline.init(allocator)
Loader-->>Main: getModule()
Main->>Pipe: setModule(module_ref)
Main->>Pipe: registerAllPasses()
Main->>Pipe: runStaticAnalysis()
Pipe->>PM: run(ctx, diag)
PM-->>Pipe: shared facts and issues
Pipe-->>Main: PipelineResult
</div>
<p>Source anchors:</p>
<ul>
<li><code>src/main.zig:171</code> initializes <code>Pipeline</code>.</li>
<li><code>src/main.zig:173</code> obtains the module from <code>IRLoader</code>.</li>
<li><code>src/main.zig:177</code> registers passes.</li>
<li><code>src/main.zig:180</code> runs static analysis.</li>
<li><code>src/main.zig:184</code> reads issues from the pipeline.</li>
</ul>
<h2 id="registerallpasses-reveals-the-analysis-sequence"><code>registerAllPasses</code> reveals the analysis sequence</h2>
<p><code>registerAllPasses</code> is at <code>src/main.zig:153</code>. It registers CallGraph, TaintPropagation, FFI Boundary, FFI Type Mismatch, FFI Body Check, FFI Unsafe, PtrLifetime, DangerSurface, PointerOwnership, CallbackEscape, RustFfiAuditor, ReturnCheck, MemorySafety, FreeValidation, and BufferOverflow.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[CallGraph] --> B[FFI Boundary]
B --> C[Type / Body / Unsafe]
C --> D[PtrLifetime]
D --> E[DangerSurface]
E --> F[PointerOwnership]
F --> G[CallbackEscape / RustFfiAuditor]
G --> H[MemorySafety / FreeValidation / BufferOverflow]
</div>
<p>The registration order is not necessarily the final execution order. The final order is resolved by <code>PassManager</code>, with execution starting at <code>src/pass/manager.zig:193</code>.</p>
<h2 id="pipeline-run-builds-the-analysis-context"><code>Pipeline.run</code> builds the analysis context</h2>
<p><code>Pipeline</code> is defined at <code>src/pipeline/pipeline.zig:27</code>. It stores <code>FactStore</code>, <code>QueryEngine</code>, <code>DataFlowGraph</code>, <code>PassManager</code>, and the current module. <code>Pipeline.run</code> creates the <code>PassContext</code> at <code>src/pipeline/pipeline.zig:66</code>.</p>
<p><code>PassContext</code> contains shared state such as:</p>
<ul>
<li>facts and query engine;</li>
<li>data-flow graph;</li>
<li>value-id map;</li>
<li>registry and zone caches;</li>
<li><code>cross_lang_edges</code>;</li>
<li><code>global_alloc_tracker</code>;</li>
<li><code>memory_graph</code>;</li>
<li><code>danger_surface_relevant</code>, <code>ffi_auto_relevant</code>, and <code>relevant_functions</code>.</li>
</ul>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TB
subgraph Pipeline.run
A[Clear DataFlowGraph] --> B[Create PassContext]
B --> C[Attach module]
B --> D[Initialize caches]
B --> E[Initialize MemoryGraph]
B --> F[Initialize CrossLangEdges]
F --> G[PassManager.run]
end
</div><h2 id="output-is-part-of-the-design">Output is part of the design</h2>
<p><code>emitOutput</code> is at <code>src/main.zig:207</code>. It branches into JSON, SARIF, or text output. JSON is produced by <code>formatIssuesAsJson</code>; SARIF is produced by <code>SarifOutput</code>.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[Issue list] --> B[emitOutput]
B --> C[JSON: formatIssuesAsJson]
B --> D[SARIF: SarifOutput]
B --> E[Text]
C --> F[stdout / file]
D --> F
E --> G[terminal]
</div><h2 id="summary">Summary</h2>
<p>An OmniScope run follows a clear data path: CLI creates configuration, IRLoader supplies a module, Pipeline creates shared analysis context, PassManager executes passes, and <code>emitOutput</code> formats results.</p>
<h2 id="source-breakdown-why-pipeline-builds-callsiteindex-early">Source breakdown: why Pipeline builds CallSiteIndex early</h2>
<p><code>Pipeline.run</code> performs one important pre-pass optimization: before any analysis pass runs, it walks the LLVM module and indexes callee names to call sites. The code lives around <code>src/pipeline/pipeline.zig:130</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">var</span><span> func</span><span style="color: #94E2D5;"> =</span><span> c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMGetFirstFunction</span><span>(raw_mod)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">while</span><span> (</span><span style="color: #F38BA8;font-style: italic;">@intFromPtr</span><span>(func) </span><span style="color: #94E2D5;">!=</span><span style="color: #FAB387;"> 0</span><span>) </span><span style="color: #94E2D5;">:</span><span> (func</span><span style="color: #94E2D5;"> =</span><span> c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMGetNextFunction</span><span>(func)) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMIsDeclaration</span><span>(func) </span><span style="color: #94E2D5;">!=</span><span style="color: #FAB387;"> 0</span><span>) </span><span style="color: #CBA6F7;">continue</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> func_ptr</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;font-style: italic;"> @as</span><span>(</span><span style="color: #CBA6F7;">u64</span><span style="color: #9399B2;">,</span><span style="color: #F38BA8;font-style: italic;"> @intFromPtr</span><span>(func))</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> var</span><span> bb</span><span style="color: #94E2D5;"> =</span><span> c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMGetFirstBasicBlock</span><span>(func)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> while</span><span> (</span><span style="color: #F38BA8;font-style: italic;">@intFromPtr</span><span>(bb) </span><span style="color: #94E2D5;">!=</span><span style="color: #FAB387;"> 0</span><span>) </span><span style="color: #94E2D5;">:</span><span> (bb</span><span style="color: #94E2D5;"> =</span><span> c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMGetNextBasicBlock</span><span>(bb)) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> var</span><span> inst</span><span style="color: #94E2D5;"> =</span><span> c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMGetFirstInstruction</span><span>(bb)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> while</span><span> (</span><span style="color: #F38BA8;font-style: italic;">@intFromPtr</span><span>(inst) </span><span style="color: #94E2D5;">!=</span><span style="color: #FAB387;"> 0</span><span>) </span><span style="color: #94E2D5;">:</span><span> (inst</span><span style="color: #94E2D5;"> =</span><span> c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMGetNextInstruction</span><span>(inst)) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (</span><span style="color: #F38BA8;font-style: italic;">@intFromPtr</span><span>(c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMIsACallInst</span><span>(inst)) </span><span style="color: #94E2D5;">==</span><span style="color: #FAB387;"> 0</span><span>) </span><span style="color: #CBA6F7;">continue</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> called_val</span><span style="color: #94E2D5;"> =</span><span> c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMGetCalledValue</span><span>(inst)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> called_name</span><span style="color: #94E2D5;"> =</span><span> std</span><span style="color: #94E2D5;">.</span><span>mem</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">span</span><span>(c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMGetValueName</span><span>(called_val))</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> inst_ptr</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;font-style: italic;"> @as</span><span>(</span><span style="color: #CBA6F7;">u64</span><span style="color: #9399B2;">,</span><span style="color: #F38BA8;font-style: italic;"> @intFromPtr</span><span>(inst))</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> ctx</span><span style="color: #94E2D5;">.</span><span>CallSiteIndex</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">addCall</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator</span><span style="color: #9399B2;">,</span><span> called_name</span><span style="color: #9399B2;">,</span><span> func_ptr</span><span style="color: #9399B2;">,</span><span> inst_ptr) </span><span style="color: #CBA6F7;">catch</span><span style="color: #94E2D5;"> |</span><span>err</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span> std</span><span style="color: #94E2D5;">.</span><span>log</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">warn</span><span>(</span><span style="color: #A6E3A1;">"[WARN] Failed to add call site for '{s}': {}"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{ called_name</span><span style="color: #9399B2;">,</span><span> err })</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>This is more than a micro-optimization. Without this index, every pass that asks "where is this external function called?" would need to rescan functions, basic blocks, and instructions. With <code>CallSiteIndex</code>, call-site lookup becomes a shared context query used by FFI boundary detection, danger-surface tracing, and report evidence.</p>
<h2 id="pipeline-also-owns-post-pass-reduction">Pipeline also owns post-pass reduction</h2>
<p>After <code>PassManager.run</code>, the pipeline scans <code>GlobalAllocTracker</code> for unfreed allocations and calls <code>ctx.isOnDangerPathFull(rec.ptr_id)</code>. If a leaked pointer reaches an FFI path, severity and confidence are promoted.</p>
<p>This shows that Pipeline is not just a container for passes. It also performs final reduction over facts that only become meaningful after all passes have contributed:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>pass phase</span></span>
<span class="giallo-l"><span> -> collect allocations / frees / aliases / ffi edges</span></span>
<span class="giallo-l"><span>post-pass phase</span></span>
<span class="giallo-l"><span> -> inspect unfreed allocations</span></span>
<span class="giallo-l"><span> -> query danger path</span></span>
<span class="giallo-l"><span> -> convert high-value candidates into issues</span></span></code></pre>
<p>The design choice is deliberate: a single pass sees local events, while Pipeline can inspect globally accumulated state.</p>
<h2 id="engineering-trade-off-lifecycle-management-is-part-of-the-architecture">Engineering trade-off: lifecycle management is part of the architecture</h2>
<p>The <code>Pipeline.run</code> path contains many <code>defer</code> and <code>errdefer</code> blocks for <code>PassContext</code>, hash maps, <code>MemoryGraph</code>, and semantic call graph cleanup. That is not incidental. A Zig-based analyzer processes large IR modules and allocates many maps, lists, and strings. If context ownership is unclear, the analyzer itself becomes unreliable in batch scans and CI.</p>
<p>So the Pipeline has three concrete responsibilities: assemble the shared context, turn expensive repeated scans into indexes, and run post-pass reduction over global facts.</p>
Inside the Pass System: Dependency Ordering, Shared Context, and Graceful Degradation
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/03-pass-system/
<h1 id="inside-the-pass-system-dependency-ordering-shared-context-and-graceful-degradation">Inside the Pass System: Dependency Ordering, Shared Context, and Graceful Degradation</h1>
<p>OmniScope organizes analysis logic as passes rather than one large procedure. The relevant implementation questions are how passes share facts, how execution order is resolved, and how the system behaves when one pass fails.</p>
<h2 id="start-with-the-problem-static-analysis-is-not-linear">Start with the problem: static analysis is not linear</h2>
<p>Cross-language findings rarely come from one rule alone. A call graph pass finds boundaries, a lifetime pass contributes pointer facts, DangerSurfacePass narrows the relevant objects, and ownership or memory passes then decide what is worth reporting. The key is not just having many passes; it is the dependencies and fact exchange between them.</p>
<p>Without orchestration, a pass can run before its inputs exist. Without shared context, every pass rescans the IR. Without graceful degradation, one unfamiliar IR pattern can stop an entire audit run. OmniScope’s pass system addresses those three problems.</p>
<h2 id="omniscope-s-entry-point-passmanager-orders-work-passcontext-carries-facts">OmniScope’s entry point: PassManager orders work, PassContext carries facts</h2>
<p><code>PassManager</code> registers, sorts, and executes passes. <code>PassContext</code> carries facts across them. This lets call relationships, MemoryGraph state, danger surfaces, and issues move through one analysis context instead of being trapped inside individual rules.</p>
<h2 id="what-passmanager-owns">What PassManager owns</h2>
<p><code>PassManager</code> is defined at <code>src/pass/manager.zig:23</code>. It stores registered passes, a name-to-index map, resolved execution order, and cached execution names. Registration starts at <code>src/pass/manager.zig:61</code>.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[registerPass] --> B[passes ArrayList]
A --> C[pass_map]
B --> D[resolveDependencies]
C --> D
D --> E[resolved_order]
E --> F[run]
</div>
<p>This keeps scheduling separate from the CLI and from each individual pass.</p>
<h2 id="dependency-resolution-uses-kahn-topological-sorting">Dependency resolution uses Kahn topological sorting</h2>
<p>The dependency resolver builds an adjacency list and in-degree table. Passes with zero in-degree enter the queue. Each popped node reduces the in-degree of its successors. If the final result contains fewer nodes than the registered pass count, a cycle is reported.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[Passes + dependencies] --> B[Build adjacency / in_degree]
B --> C[Queue zero in-degree nodes]
C --> D[Pop node]
D --> E[Reduce successor in-degree]
E --> F{Successor reaches zero?}
F -->|Yes| C
F -->|No| G[Continue]
G --> H{All passes resolved?}
H -->|Yes| I[resolved_order]
H -->|No| J[CycleDetected]
</div>
<p>This matters because later passes may consume facts produced by earlier passes: risk-path checks need cross-language edges and memory facts; ownership checks need allocation and call information.</p>
<h2 id="graceful-degradation-is-deliberate">Graceful degradation is deliberate</h2>
<p>The run loop starts at <code>src/pass/manager.zig:193</code>. If a pass fails, the manager logs a warning, increments a failure counter, and continues executing later passes.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[Run pass] --> B{run_fn succeeds?}
B -->|Yes| C[Record timing]
B -->|No| D[warn + pass_failures++]
C --> E{More passes?}
D --> E
E -->|Yes| A
E -->|No| F[Finish analysis]
</div>
<p>This is useful for real-world IR, which may come from different compilers, optimization levels, or link configurations. A failure in one analysis stage should not necessarily discard unrelated findings.</p>
<h2 id="passcontext-is-the-fact-bus">PassContext is the fact bus</h2>
<p><code>PassContext</code> is defined at <code>src/pass/pass.zig:192</code>. It is the shared state through which passes exchange information. CallGraphPass may write <code>cross_lang_edges</code>; PtrLifetimePass may write <code>memory_graph</code>; DangerSurfacePass may write relevance sets; issue-producing passes read those facts.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[CallGraphPass] -->|writes| X[PassContext.cross_lang_edges]
B[PtrLifetimePass] -->|writes| Y[PassContext.memory_graph]
C[DangerSurfacePass] -->|reads X/Y, writes| Z[relevant sets]
D[FFI / Ownership / Memory passes] -->|read Z/Y| E[Issue]
</div><h2 id="diagnosticwriter-and-structured-issues">DiagnosticWriter and structured issues</h2>
<p>Passes receive both <code>PassContext</code> and <code>DiagnosticWriter</code>. <code>DiagnosticWriter</code> handles logs and diagnostics. Structured findings are added through <code>ctx.addIssue</code>, whose entry point is at <code>src/pass/pass.zig:458</code>.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[Pass detects finding] --> B{Diagnostic only?}
B -->|Yes| C[DiagnosticWriter]
B -->|No| D[Issue.init / initWithReason]
D --> E[ctx.addIssue]
E --> F[DataFlowGraph issues]
F --> G[Pipeline.getIssues]
</div><h2 id="summary">Summary</h2>
<p>The pass system provides registration, dependency ordering, shared state, graceful degradation, and structured issue collection. The later MemoryGraph, Zone, and Registry mechanisms rely on this shared execution model.</p>
<h2 id="source-breakdown-dependency-ordering-is-a-runtime-constraint">Source breakdown: dependency ordering is a runtime constraint</h2>
<p><code>PassManager.resolveDependencies</code> uses Kahn’s algorithm to topologically sort passes. The implementation in <code>src/pass/manager.zig:86</code> to <code>src/pass/manager.zig:180</code> builds the graph from <code>pass.deps</code>, tracks in-degrees, and computes <code>resolved_order</code>.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">for</span><span> (</span><span style="color: #FAB387;">0</span><span style="color: #94E2D5;">..</span><span>num_passes) </span><span style="color: #94E2D5;">|</span><span>i</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> pass</span><span style="color: #94E2D5;"> =</span><span> self</span><span style="color: #94E2D5;">.</span><span>passes</span><span style="color: #94E2D5;">.</span><span>items[i]</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> (pass</span><span style="color: #94E2D5;">.</span><span>deps) </span><span style="color: #94E2D5;">|</span><span>dep_name</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> dep_idx</span><span style="color: #94E2D5;"> =</span><span> self</span><span style="color: #94E2D5;">.</span><span>pass_map</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">get</span><span>(dep_name) </span><span style="color: #CBA6F7;">orelse</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return error</span><span style="color: #94E2D5;">.</span><span>MissingDependency</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> adjacency</span><span style="color: #94E2D5;">.</span><span>items[dep_idx]</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">append</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator</span><span style="color: #9399B2;">,</span><span> i)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> in_degree[i] </span><span style="color: #94E2D5;">+=</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">while</span><span> (queue</span><span style="color: #94E2D5;">.</span><span>items</span><span style="color: #94E2D5;">.</span><span>len</span><span style="color: #94E2D5;"> ></span><span style="color: #FAB387;"> 0</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> node</span><span style="color: #94E2D5;"> =</span><span> queue</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">swapRemove</span><span>(</span><span style="color: #FAB387;">0</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> result</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">append</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator</span><span style="color: #9399B2;">,</span><span> node)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> (adjacency</span><span style="color: #94E2D5;">.</span><span>items[node]</span><span style="color: #94E2D5;">.</span><span>items) </span><span style="color: #94E2D5;">|</span><span>neighbor</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span> in_degree[neighbor] </span><span style="color: #94E2D5;">-=</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (in_degree[neighbor] </span><span style="color: #94E2D5;">==</span><span style="color: #FAB387;"> 0</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> queue</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">append</span><span>(self</span><span style="color: #94E2D5;">.</span><span>allocator</span><span style="color: #9399B2;">,</span><span> neighbor)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>Two design signals stand out. First, a missing dependency returns <code>MissingDependency</code> immediately, because a wrong pass order does not just break configuration — it can produce wrong facts. Second, the queue uses <code>swapRemove(0)</code> instead of <code>orderedRemove(0)</code> to avoid O(N) cost. The exact order of equal-ready passes does not matter as long as dependency constraints are satisfied.</p>
<h2 id="graceful-degradation-one-failed-pass-must-not-blind-the-tool">Graceful degradation: one failed pass must not blind the tool</h2>
<p><code>PassManager.run</code> executes each pass independently, logs failures, and keeps going.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">for</span><span> (self</span><span style="color: #94E2D5;">.</span><span>resolved_order</span><span style="color: #94E2D5;">.?</span><span>) </span><span style="color: #94E2D5;">|</span><span>idx</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> pass_name</span><span style="color: #94E2D5;"> =</span><span> self</span><span style="color: #94E2D5;">.</span><span>passes</span><span style="color: #94E2D5;">.</span><span>items[idx]</span><span style="color: #94E2D5;">.</span><span>name</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> t0</span><span style="color: #94E2D5;"> =</span><span> std</span><span style="color: #94E2D5;">.</span><span>time</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">nanoTimestamp</span><span>()</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> self</span><span style="color: #94E2D5;">.</span><span>passes</span><span style="color: #94E2D5;">.</span><span>items[idx]</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">run_fn</span><span>(ctx</span><span style="color: #9399B2;">,</span><span> diag) </span><span style="color: #CBA6F7;">catch</span><span style="color: #94E2D5;"> |</span><span>err</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span> diag</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">warn</span><span>(</span><span style="color: #A6E3A1;">"PassManager: pass '{s}' failed with error: {any}, degrading gracefully"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{ pass_name</span><span style="color: #9399B2;">,</span><span> err })</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> pass_failures</span><span style="color: #94E2D5;"> +=</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> elapsed_ns</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;font-style: italic;"> @max</span><span>(</span><span style="color: #F38BA8;font-style: italic;">@as</span><span>(</span><span style="color: #CBA6F7;">i128</span><span style="color: #9399B2;">,</span><span style="color: #FAB387;"> 0</span><span>)</span><span style="color: #9399B2;">,</span><span> std</span><span style="color: #94E2D5;">.</span><span>time</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">nanoTimestamp</span><span>() </span><span style="color: #94E2D5;">-</span><span> t0)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> elapsed_ms</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;font-style: italic;"> @as</span><span>(</span><span style="color: #CBA6F7;">f64</span><span style="color: #9399B2;">,</span><span style="color: #F38BA8;font-style: italic;"> @floatFromInt</span><span>(elapsed_ns)) </span><span style="color: #94E2D5;">/</span><span style="color: #FAB387;"> 1_000_000.0</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (elapsed_ms</span><span style="color: #94E2D5;"> ></span><span style="color: #FAB387;"> 10</span><span>) {</span></span>
<span class="giallo-l"><span> diag</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">info</span><span>(</span><span style="color: #A6E3A1;">"[PERF] Pass '{s}: {d} ms"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{ pass_name</span><span style="color: #9399B2;">,</span><span style="color: #F38BA8;font-style: italic;"> @as</span><span>(</span><span style="color: #CBA6F7;">u32</span><span style="color: #9399B2;">,</span><span style="color: #F38BA8;font-style: italic;"> @intFromFloat</span><span>(elapsed_ms)) })</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>This is not just error swallowing. Real-world IR is messy: incomplete linking, different optimization levels, missing debug info, and compiler differences can make one pass hit an unexpected pattern. OmniScope preserves independent evidence instead of letting one failure zero out the whole run.</p>
<h2 id="addissue-is-a-normalization-layer"><code>addIssue</code> is a normalization layer</h2>
<p>Passes do not print findings directly. They create an <code>Issue</code> and send it through <code>ctx.addIssue</code> in <code>src/pass/pass.zig:458</code>. That function deduplicates, enriches classification, handles ownership, and then forwards the result to the data flow graph or issue store.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>pass-local finding</span></span>
<span class="giallo-l"><span> -> Issue.init / Issue.initWithReason</span></span>
<span class="giallo-l"><span> -> PassContext.addIssue</span></span>
<span class="giallo-l"><span> -> dedup / confidence / classification / ownership handling</span></span>
<span class="giallo-l"><span> -> DataFlowGraph issue store</span></span>
<span class="giallo-l"><span> -> output formatter</span></span></code></pre>
<p>That is why <code>Issue</code> carries <code>confidence</code>, <code>reason</code>, <code>ffi_boundary</code>, <code>trace</code>, and <code>classification</code>. The pass only needs to say what it found; the context decides how that finding enters the global reporting system.</p>
<h2 id="how-it-works-passes-produce-evidence-not-reports">How it works: passes produce evidence, not reports</h2>
<p>The pass system defines a hard boundary: passes create facts and issues, but do not decide final presentation. That gives OmniScope a few important properties:</p>
<ul>
<li>The same finding can be rendered as text, JSON, or SARIF.</li>
<li>Output schemas can evolve without changing every analysis pass.</li>
<li><code>DangerSurface</code> can populate relevance sets without emitting findings directly.</li>
<li>Rust FFI, memory safety, and return-check passes can all share the same fact layer.</li>
</ul>
<p>That is a better fit for a source-level audit tool than printing findings immediately, because what matters is not the individual rule but whether rules can exchange evidence.</p>
Zone Classification and Semantic Registry: Avoiding Blacklist-Style Reporting
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/04-zone-and-registry/
<h1 id="zone-classification-and-semantic-registry-avoiding-blacklist-style-reporting">Zone Classification and Semantic Registry: Avoiding Blacklist-Style Reporting</h1>
<p>A static analyzer without semantic layers can easily become a dangerous-function list. OmniScope separates region classification from function semantics before deciding whether a path should be analyzed in depth.</p>
<h2 id="start-with-the-problem-the-same-function-name-means-different-things-in-context">Start with the problem: the same function name means different things in context</h2>
<p>Reporting whenever <code>free</code>, <code>strcpy</code>, or <code>dlopen</code> appears is simple, but it becomes noisy in cross-language auditing. An allocator inside runtime glue may be expected; the same allocator in a user-written FFI wrapper may indicate ownership crossing a language boundary.</p>
<p>The question is not only "is this function dangerous?" It is also: which code region is it in, runtime/internal or user boundary code? Does the function represent allocation, deallocation, borrow escape, or ordinary library behavior in an FFI context?</p>
<h2 id="omniscope-s-entry-point-zone-answers-where-registry-answers-what">OmniScope’s entry point: Zone answers where, Registry answers what</h2>
<p><code>ZoneClassifier</code> classifies the region first. <code>SemanticRegistry</code> then explains function meaning in cross-language context. Together they provide context to later danger-path checks: the same call can receive different priority depending on its zone and semantic kind.</p>
<h2 id="zonekind-classifies-code-by-risk-region">ZoneKind classifies code by risk region</h2>
<p><code>ZoneKind</code> is defined at <code>src/semantics/zone_classifier.zig:24</code>. It includes <code>safe</code>, <code>unsafe</code>, <code>ffi</code>, <code>runtime_internal</code>, and <code>unknown</code>. The file-level comment states the operating principle: focus where language guarantees stop.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[Function / Instruction / Debug path] --> B[Zone Classifier]
B --> C[safe: language guarantees likely apply]
B --> D[unsafe: explicit escape]
B --> E[ffi: cross-language boundary]
B --> F[runtime_internal: stdlib/runtime]
B --> G[unknown: conservative handling]
</div>
<p>This model is not a proof that safe zones are bug-free. It is a prioritization layer. For an FFI-focused analyzer, runtime glue or container internals should usually not have the same weight as user-written unsafe wrappers.</p>
<h2 id="multi-language-escape-triggers">Multi-language escape triggers</h2>
<p><code>EscapeTrigger</code> is defined at <code>src/semantics/zone_classifier.zig:45</code>. It represents cross-language escape points across ecosystems: Rust unsafe/extern/raw pointers, Zig pointer casts and C imports, Go cgo and <code>unsafe.Pointer</code>, and C++ extern C, reinterpret casts, and manual memory.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[Rust unsafe / extern C] --> Z[EscapeTrigger]
B[Zig @ptrCast / @cImport] --> Z
C[Go cgo / unsafe.Pointer] --> Z
D[C++ reinterpret_cast / malloc] --> Z
Z --> E[ZoneKind unsafe or ffi]
</div>
<p>This gives the implementation a common vocabulary for multiple language-specific risk boundaries.</p>
<h2 id="function-level-and-llvm-level-classification">Function-level and LLVM-level classification</h2>
<p>The classifier includes both name-based and LLVM-function-based entry points:</p>
<ul>
<li><code>src/semantics/zone_classifier.zig:347</code> classifies by function name.</li>
<li><code>src/semantics/zone_classifier.zig:394</code> classifies LLVM functions using declarations, intrinsics, debug information, and path data when available.</li>
</ul>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[LLVMValueRef function] --> B{External declaration?}
B -->|Yes| C[May be ffi]
B -->|No| D[Name-based classification]
D --> E[Path/debug-info classification]
E --> F[ZoneKind]
</div>
<p>These rules are heuristic. Symbol names and debug information quality can affect classification, so Zone should be described as a risk-prioritization mechanism rather than a formal proof.</p>
<h2 id="semantic-registry-describes-function-meaning-in-ffi-context">Semantic Registry describes function meaning in FFI context</h2>
<p><code>SemanticRegistry</code> is defined at <code>src/registry/semantic_registry.zig:48</code>; lookup starts at <code>src/registry/semantic_registry.zig:90</code>. The registry is layered by ecosystem: C standard library, Rust ownership patterns, Go cgo, Zig, C++, JNI, Python C API, POSIX, and dynamic loading.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[func_name] --> B[SemanticRegistry.lookup]
B --> C[Layer1: C stdlib high-risk]
B --> D[Layer2: Rust ownership]
B --> E[Layer3: Go cgo]
B --> F[Layer5: Zig stdlib]
B --> G[Layer6: C++ stdlib]
B --> H[JNI / Python C API / POSIX]
C --> I[FunctionSemantics]
D --> I
E --> I
F --> I
G --> I
H --> I
</div>
<p>The registry helps interpret calls in context. <code>Box::into_raw</code> is not a vulnerability by itself; it changes the ownership protocol. <code>strcpy</code> in local C code and <code>strcpy</code> at a Rust-to-C boundary may require different review priorities.</p>
<h2 id="zone-registry-danger-path">Zone + Registry + Danger Path</h2>
<p>Zone and Registry should not produce most findings directly. A more controlled path is: classify the region, look up function semantics, ask whether the pointer or function is on a relevant risk path, then produce an issue if warranted.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[Call / Function] --> B[Zone]
A --> C[Registry]
B --> D[Region risk]
C --> E[Function semantics]
D --> F[Danger path]
E --> F
F --> G{Report?}
G -->|Yes| H[Issue]
G -->|No| I[Filter or lower priority]
</div><h2 id="summary">Summary</h2>
<p>OmniScope avoids pure blacklist reporting by splitting a finding into three questions: where is the code located, what does the function mean at an FFI boundary, and does the pointer or function participate in a relevant risk path?</p>
<h2 id="source-breakdown-zoneclassifier-is-a-priority-decision-tree">Source breakdown: ZoneClassifier is a priority decision tree</h2>
<p>The latter half of <code>classifyFunction</code> lives around <code>src/semantics/zone_classifier.zig:640</code>. It is not a simple lookup table. The order is C++ unsafe patterns, C++ safe patterns, C escape patterns, extern C, and only then the <code>SemanticRegistry</code> fallback.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">for</span><span> (CPP_ESCAPE_PATTERNS) </span><span style="color: #94E2D5;">|</span><span>pattern</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (std</span><span style="color: #94E2D5;">.</span><span>mem</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">indexOf</span><span>(</span><span style="color: #CBA6F7;">u8</span><span style="color: #9399B2;">,</span><span> func_name</span><span style="color: #9399B2;">,</span><span> pattern) </span><span style="color: #94E2D5;">!=</span><span style="color: #CBA6F7;"> null</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #94E2D5;"> .</span><span>unsafe</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span> (CPP_SAFE_PATTERNS) </span><span style="color: #94E2D5;">|</span><span>pattern</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (std</span><span style="color: #94E2D5;">.</span><span>mem</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">indexOf</span><span>(</span><span style="color: #CBA6F7;">u8</span><span style="color: #9399B2;">,</span><span> func_name</span><span style="color: #9399B2;">,</span><span> pattern) </span><span style="color: #94E2D5;">!=</span><span style="color: #CBA6F7;"> null</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #94E2D5;"> .</span><span>safe</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">if</span><span> (SemanticRegistry</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">lookup</span><span>(func_name)) </span><span style="color: #94E2D5;">|</span><span>sem</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> switch</span><span> (sem</span><span style="color: #94E2D5;">.</span><span>kind) {</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>allocator</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>deallocator</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>rust_ownership</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>borrow_escaped</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>zig_allocator</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>cpp_allocator</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> =></span><span style="color: #CBA6F7;"> return</span><span style="color: #94E2D5;"> .</span><span>ffi</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> else</span><span style="color: #94E2D5;"> =></span><span style="color: #CBA6F7;"> return</span><span style="color: #94E2D5;"> .</span><span>ffi</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>The important part is ordering. One function name can match multiple weak signals, such as a runtime helper that also contains allocator semantics. Without priority, Zone classification becomes noisy. OmniScope lets strong context decide the zone first and weak semantics fill in the explanation later.</p>
<h2 id="source-breakdown-semanticregistry-is-a-layered-semantic-library">Source breakdown: SemanticRegistry is a layered semantic library</h2>
<p><code>SemanticRegistry.lookup</code> in <code>src/registry/semantic_registry.zig:90</code> walks layer1 through layer6, then JNI, Python C API, file I/O, network I/O, signal, thread, process, dynamic loading, and static buffer groups.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> lookup</span><span>(func_name</span><span style="color: #94E2D5;">:</span><span> []</span><span style="color: #CBA6F7;">const u8</span><span>) </span><span style="color: #94E2D5;">?</span><span>FunctionSemantics {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> (layer1) </span><span style="color: #94E2D5;">|</span><span>sem</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (</span><span style="color: #89B4FA;font-style: italic;">matchesPattern</span><span>(func_name</span><span style="color: #9399B2;">,</span><span> sem</span><span style="color: #94E2D5;">.</span><span>pattern</span><span style="color: #9399B2;">,</span><span> sem</span><span style="color: #94E2D5;">.</span><span>match_type)) </span><span style="color: #CBA6F7;">return</span><span> sem</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> (layer2) </span><span style="color: #94E2D5;">|</span><span>sem</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (</span><span style="color: #89B4FA;font-style: italic;">matchesPattern</span><span>(func_name</span><span style="color: #9399B2;">,</span><span> sem</span><span style="color: #94E2D5;">.</span><span>pattern</span><span style="color: #9399B2;">,</span><span> sem</span><span style="color: #94E2D5;">.</span><span>match_type)) </span><span style="color: #CBA6F7;">return</span><span> sem</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ... ecosystem-specific layers ...</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> (dynamic_loading) </span><span style="color: #94E2D5;">|</span><span>sem</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (</span><span style="color: #89B4FA;font-style: italic;">matchesPattern</span><span>(func_name</span><span style="color: #9399B2;">,</span><span> sem</span><span style="color: #94E2D5;">.</span><span>pattern</span><span style="color: #9399B2;">,</span><span> sem</span><span style="color: #94E2D5;">.</span><span>match_type)) </span><span style="color: #CBA6F7;">return</span><span> sem</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return null</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span>}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> matchesPattern</span><span>(func_name</span><span style="color: #94E2D5;">:</span><span> []</span><span style="color: #CBA6F7;">const u8</span><span style="color: #9399B2;">,</span><span> pattern</span><span style="color: #94E2D5;">:</span><span> []</span><span style="color: #CBA6F7;">const u8</span><span style="color: #9399B2;">,</span><span> match_type</span><span style="color: #94E2D5;">:</span><span> MatchType) </span><span style="color: #CBA6F7;">bool</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return switch</span><span> (match_type) {</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>exact</span><span style="color: #94E2D5;"> =></span><span> std</span><span style="color: #94E2D5;">.</span><span>mem</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">eql</span><span>(</span><span style="color: #CBA6F7;">u8</span><span style="color: #9399B2;">,</span><span> func_name</span><span style="color: #9399B2;">,</span><span> pattern)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>contains</span><span style="color: #94E2D5;"> =></span><span> std</span><span style="color: #94E2D5;">.</span><span>mem</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">indexOf</span><span>(</span><span style="color: #CBA6F7;">u8</span><span style="color: #9399B2;">,</span><span> func_name</span><span style="color: #9399B2;">,</span><span> pattern) </span><span style="color: #94E2D5;">!=</span><span style="color: #CBA6F7;"> null</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>suffix</span><span style="color: #94E2D5;"> =></span><span> std</span><span style="color: #94E2D5;">.</span><span>mem</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">endsWith</span><span>(</span><span style="color: #CBA6F7;">u8</span><span style="color: #9399B2;">,</span><span> func_name</span><span style="color: #9399B2;">,</span><span> pattern)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> }</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>This is more expressive than a dangerous-function list. The registry stores semantic source and match strength. <code>exact</code> evidence is stronger than <code>contains</code>, and ecosystem-specific meaning is different from a generic libc helper. That difference is what later turns into confidence and reason.</p>
<h2 id="how-it-works-zone-reduces-noise-registry-explains-meaning">How it works: Zone reduces noise, Registry explains meaning</h2>
<p>Keeping Zone and Registry separate avoids two common mistakes:</p>
<ul>
<li>Zone alone tells you where code lives, but not what the function means.</li>
<li>Registry alone tells you a function looks like an allocator or deallocator, but not whether it sits on a local path or a cross-language path.</li>
</ul>
<p>OmniScope lets Zone answer "where", Registry answer "what", and MemoryGraph/DangerSurface answer "does it flow into a dangerous path". Once those questions are separated, false-positive control becomes actionable.</p>
MemoryGraph and DangerSurface: From Pointer Facts to Risk Paths
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/05-memory-graph-danger-surface/
<h1 id="memorygraph-and-dangersurface-from-pointer-facts-to-risk-paths">MemoryGraph and DangerSurface: From Pointer Facts to Risk Paths</h1>
<p>Cross-language memory analysis should not only ask whether <code>malloc</code> or <code>free</code> exists. It should ask whether a pointer crosses FFI, comes from an unsafe region, or propagates through aliases. OmniScope uses <code>MemoryGraph</code> and <code>DangerSurfacePass</code> to connect these facts.</p>
<h2 id="start-with-the-problem-memory-risk-lives-on-paths-not-names">Start with the problem: memory risk lives on paths, not names</h2>
<p>A pointer can start as Rust <code>Box::into_raw</code>, become <code>void*</code> through a bitcast, pass into C, get stored in a global structure, and later be released by another function. Each step can look like an ordinary IR event. The risk only appears when the path is connected.</p>
<p>This is where simple rule scanning loses information: it can see calls but not pointer families, allocators but not aliases, FFI boundaries but not which pointer actually crossed them.</p>
<h2 id="omniscope-s-entry-point-build-memorygraph-then-narrow-with-dangersurface">OmniScope’s entry point: build MemoryGraph, then narrow with DangerSurface</h2>
<p><code>MemoryGraph</code> stores allocations, frees, call arguments, returns, and aliases in one graph. <code>DangerSurfacePass</code> starts from FFI/unsafe boundaries and marks the pointers and functions worth deeper analysis. Reports then follow risky paths rather than isolated APIs.</p>
<h2 id="role-of-memorygraph">Role of MemoryGraph</h2>
<p><code>MemoryGraph</code> is defined at <code>src/semantics/memory_graph.zig:160</code>. It stores memory-related facts: allocations, frees, call arguments, call returns, alias relations, zones, and language sources.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[trackAlloc] --> M[MemoryGraph]
B[trackFree] --> M
C[trackCallArg] --> M
D[trackCallRet] --> M
E[alias relation] --> M
F[Zone / Language] --> M
M --> G[isOnDangerPath]
</div>
<p>The graph does not attempt to model every possible program path. It answers a narrower question: is there enough evidence that this pointer is relevant to an FFI/unsafe risk path?</p>
<h2 id="risk-path-classification">Risk-path classification</h2>
<p><code>MemoryGraph.isOnDangerPath</code> is implemented around <code>src/semantics/memory_graph.zig:892</code>. It may return categories such as <code>ffi_arg</code>, <code>ffi_ret</code>, and <code>unsafe_alloc</code>. <code>PassContext.isOnDangerPathFull</code> at <code>src/pass/pass.zig:866</code> provides a shared entry point for other passes.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[ptr_val] --> B[PassContext.isOnDangerPathFull]
B --> C[MemoryGraph.isOnDangerPath]
C --> D{DangerPathKind}
D -->|ffi_arg| E[Pointer passed as FFI argument]
D -->|ffi_ret| F[Pointer returned from FFI]
D -->|unsafe_alloc| G[Allocation in unsafe region]
E --> H[Prioritize analysis]
F --> H
G --> H
</div>
<p>A shared entry point keeps later passes from inventing inconsistent risk definitions.</p>
<h2 id="how-dangersurfacepass-marks-relevant-objects">How DangerSurfacePass marks relevant objects</h2>
<p><code>DangerSurfacePass</code> starts at <code>src/pass/analysis/danger_surface.zig:37</code>. It consumes <code>cross_lang_edges</code> and <code>memory_graph</code>, then updates <code>danger_surface_relevant</code>, <code>ffi_auto_relevant</code>, and <code>relevant_functions</code>.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
sequenceDiagram
participant CG as CrossLangEdges
participant MG as MemoryGraph
participant DS as DangerSurfacePass
participant Ctx as PassContext
DS->>CG: Read FFI callees
DS->>MG: Inspect call args / returns
DS->>Ctx: markRelevantAlloc(ptr)
DS->>Ctx: markFfiRelevant(ptr)
DS->>DS: traceAliasClosure(ptr)
DS->>Ctx: markRelevantFunction(func)
</div>
<p>Two implementation choices are visible in the source: known FFI arguments and returns can be marked directly, while MemoryGraph nodes already on a risk path can be expanded through alias closure.</p>
<h2 id="why-alias-closure-matters">Why alias closure matters</h2>
<p><code>traceAliasClosure</code> is at <code>src/pass/analysis/danger_surface.zig:144</code>. In LLVM IR, one memory object may appear through several SSA values after bitcasts, loads, stores, parameters, and returns. Marking only the original pointer may miss later uses.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[ptr0: FFI relevant] --> B[alias set]
B --> C[ptr1]
B --> D[ptr2]
C --> E[markRelevantAlloc]
D --> E
E --> F[Recursive alias tracing]
</div>
<p>This is not a full alias analysis. It is a focused propagation step for FFI-related pointer families.</p>
<h2 id="from-risk-paths-to-prioritized-reporting">From risk paths to prioritized reporting</h2>
<p>Later passes can use <code>isOnDangerPathFull</code> or relevance sets to filter findings. A local C allocation/free pair may be lower priority, while a pointer crossing FFI through arguments, returns, or callbacks may require review.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[Potential memory issue] --> B{On risk path?}
B -->|No| C[Filter / lower priority / local issue]
B -->|Yes| D[FFI-relevant issue]
D --> E[Higher review priority]
</div><h2 id="summary">Summary</h2>
<p>MemoryGraph stores pointer facts. DangerSurfacePass turns those facts into relevance sets tied to FFI/unsafe paths. This is the layer that connects low-level IR events with higher-level ownership and lifetime checks.</p>
<h2 id="source-breakdown-isondangerpath-is-the-central-question">Source breakdown: <code>isOnDangerPath</code> is the central question</h2>
<p><code>MemoryGraph.isOnDangerPath</code> lives around <code>src/semantics/memory_graph.zig:892</code>. The source comment calls it the ONE question that decides whether a pointer matters. It checks call edges first, allocation-node facts second, and alias closure last.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> isOnDangerPath</span><span>(</span></span>
<span class="giallo-l"><span> graph</span><span style="color: #94E2D5;">: *</span><span>MemoryGraph</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> ptr_val</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> ffi_boundaries</span><span style="color: #94E2D5;">:</span><span> []</span><span style="color: #CBA6F7;">const</span><span> MemoryGraph</span><span style="color: #94E2D5;">.</span><span>DangerSurface</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> visited</span><span style="color: #94E2D5;">: *</span><span>std</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">AutoHashMap</span><span>(</span><span style="color: #CBA6F7;">u64</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> void</span><span>)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> ffi_set</span><span style="color: #94E2D5;">: ?*</span><span style="color: #CBA6F7;">const</span><span> std</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">StringHashMap</span><span>(</span><span style="color: #CBA6F7;">void</span><span>)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span>) DangerPathKind {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (visited</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">contains</span><span>(ptr_val)) </span><span style="color: #CBA6F7;">return</span><span style="color: #94E2D5;"> .</span><span>none</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> visited</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">put</span><span>(ptr_val</span><span style="color: #9399B2;">,</span><span> {}) </span><span style="color: #CBA6F7;">catch return</span><span style="color: #94E2D5;"> .</span><span>none</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> arg_indices</span><span style="color: #94E2D5;"> =</span><span> graph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">getCallArgsForPtr</span><span>(ptr_val)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> (arg_indices) </span><span style="color: #94E2D5;">|</span><span>idx</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> arg_edge</span><span style="color: #94E2D5;"> = &</span><span>graph</span><span style="color: #94E2D5;">.</span><span>call_args</span><span style="color: #94E2D5;">.</span><span>items[idx]</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (set</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">contains</span><span>(arg_edge</span><span style="color: #94E2D5;">.</span><span>callee_name)) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #94E2D5;"> .</span><span>ffi_arg</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> ret_indices</span><span style="color: #94E2D5;"> =</span><span> graph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">getCallRetsForPtr</span><span>(ptr_val)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> (ret_indices) </span><span style="color: #94E2D5;">|</span><span>idx</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> ret_edge</span><span style="color: #94E2D5;"> = &</span><span>graph</span><span style="color: #94E2D5;">.</span><span>call_rets</span><span style="color: #94E2D5;">.</span><span>items[idx]</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (set</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">contains</span><span>(ret_edge</span><span style="color: #94E2D5;">.</span><span>callee_name)) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #94E2D5;"> .</span><span>ffi_ret</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> node</span><span style="color: #94E2D5;"> =</span><span> graph</span><span style="color: #94E2D5;">.</span><span>nodes</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">get</span><span>(ptr_val) </span><span style="color: #CBA6F7;">orelse return</span><span style="color: #94E2D5;"> .</span><span>none</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (node</span><span style="color: #94E2D5;">.</span><span>zone</span><span style="color: #94E2D5;"> == .</span><span>unsafe) </span><span style="color: #CBA6F7;">return</span><span style="color: #94E2D5;"> .</span><span>unsafe_alloc</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (node</span><span style="color: #94E2D5;">.</span><span>freed</span><span style="color: #CBA6F7;"> and</span><span> node</span><span style="color: #94E2D5;">.</span><span>alloc_lang</span><span style="color: #94E2D5;"> !=</span><span> node</span><span style="color: #94E2D5;">.</span><span>free_lang</span><span style="color: #94E2D5;">.?</span><span>) </span><span style="color: #CBA6F7;">return</span><span style="color: #94E2D5;"> .</span><span>cross_lang_lifecycle</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // alias closure follows...</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>The order matters. Call edges must come before allocation nodes because many FFI arguments have no allocation record: they may come from function parameters, external returns, or sources that earlier passes could not recover. Requiring an <code>AllocNode</code> first would miss the most important boundary-flow cases.</p>
<h2 id="source-breakdown-dangersurfacepass-is-a-pruning-pass">Source breakdown: DangerSurfacePass is a pruning pass</h2>
<p><code>DangerSurfacePass.run</code> in <code>src/pass/analysis/danger_surface.zig:37</code> takes <code>ctx.getCrossLangEdges()</code>, extracts FFI boundaries, and marks relevant arguments, returns, functions, and aliases.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">for</span><span> (ffis) </span><span style="color: #94E2D5;">|</span><span>surface</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> arg_indices</span><span style="color: #94E2D5;"> =</span><span> mg</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">getCallArgsForCallee</span><span>(surface</span><span style="color: #94E2D5;">.</span><span>callee_name)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> (arg_indices) </span><span style="color: #94E2D5;">|</span><span>arg_idx</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> arg_ptr_val</span><span style="color: #94E2D5;"> =</span><span> mg</span><span style="color: #94E2D5;">.</span><span>call_args</span><span style="color: #94E2D5;">.</span><span>items[arg_idx]</span><span style="color: #94E2D5;">.</span><span>arg_ptr</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> ctx</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">markRelevantAlloc</span><span>(arg_ptr_val)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> ctx</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">markFfiRelevant</span><span>(arg_ptr_val)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> ctx</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">markFunctionFromInst</span><span>(mg</span><span style="color: #94E2D5;">.</span><span>call_args</span><span style="color: #94E2D5;">.</span><span>items[arg_idx]</span><span style="color: #94E2D5;">.</span><span>caller_inst)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span style="color: #89B4FA;font-style: italic;"> traceAliasClosure</span><span>(mg</span><span style="color: #9399B2;">,</span><span> arg_ptr_val</span><span style="color: #9399B2;">,</span><span> ctx</span><span style="color: #9399B2;">,</span><span> diag</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>visited)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> ret_indices</span><span style="color: #94E2D5;"> =</span><span> mg</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">getCallRetsFromCallee</span><span>(surface</span><span style="color: #94E2D5;">.</span><span>callee_name)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> (ret_indices) </span><span style="color: #94E2D5;">|</span><span>ret_idx</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> ret_ptr_val</span><span style="color: #94E2D5;"> =</span><span> mg</span><span style="color: #94E2D5;">.</span><span>call_rets</span><span style="color: #94E2D5;">.</span><span>items[ret_idx]</span><span style="color: #94E2D5;">.</span><span>ret_ptr</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> ctx</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">markRelevantAlloc</span><span>(ret_ptr_val)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> ctx</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">markFfiRelevant</span><span>(ret_ptr_val)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> ctx</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">markFunctionFromInst</span><span>(mg</span><span style="color: #94E2D5;">.</span><span>call_rets</span><span style="color: #94E2D5;">.</span><span>items[ret_idx]</span><span style="color: #94E2D5;">.</span><span>caller_inst)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span style="color: #89B4FA;font-style: italic;"> traceAliasClosure</span><span>(mg</span><span style="color: #9399B2;">,</span><span> ret_ptr_val</span><span style="color: #9399B2;">,</span><span> ctx</span><span style="color: #9399B2;">,</span><span> diag</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>visited)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>It does not emit issues directly. Its output is a set of relevance maps: <code>danger_surface_relevant</code>, <code>ffi_auto_relevant</code>, and <code>relevant_functions</code>. Later passes use those maps to decide whether to run strict checks or filter local noise.</p>
<h2 id="how-it-works-build-the-fact-graph-first-then-define-the-danger-surface">How it works: build the fact graph first, then define the danger surface</h2>
<p>Instead of reporting immediately on malloc/free-like events, OmniScope records allocations, frees, call arguments, call returns, and aliases in <code>MemoryGraph</code>. Then it asks which graph nodes are close to FFI or unsafe boundaries.</p>
<p>That gives three benefits: rules reuse one pointer-fact layer, future danger-path kinds can be added centrally, and reports can distinguish local-only issues from boundary-relevant issues. The cost is also clear: the quality of <code>MemoryGraph</code> bounds the quality of every downstream pass.</p>
Rust FFI Auditor: Reconstructing and Checking Cross-Language Ownership Protocols
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/06-rust-ffi-auditor/
<h1 id="rust-ffi-auditor-reconstructing-and-checking-cross-language-ownership-protocols">Rust FFI Auditor: Reconstructing and Checking Cross-Language Ownership Protocols</h1>
<p>Rust FFI risk often appears when Rust’s ownership and borrowing protocols cross an ABI boundary. <code>RustFfiAuditor</code> maps those protocols back onto LLVM IR patterns that can be inspected statically.</p>
<h2 id="start-with-the-problem-rust-cannot-verify-what-c-does-later">Start with the problem: Rust cannot verify what C does later</h2>
<p>Inside Rust, <code>Box</code>, borrow, and drop have clear semantics. Once <code>extern "C"</code> exposes a raw pointer, those semantics become a protocol between two sides. C can store the pointer, release it later, call <code>free</code>, or pass it back through a callback. The Rust compiler no longer verifies those actions.</p>
<p>Rust FFI auditing therefore looks past "is there unsafe?" and asks whether the ownership protocol closes: who reclaims an <code>into_raw</code> pointer, whether an <code>as_ptr</code> borrow escapes, whether a stack address outlives the function, and whether allocation and deallocation use the same protocol.</p>
<h2 id="omniscope-s-entry-point-rust-specific-rules-plus-universal-ffi-rules">OmniScope’s entry point: Rust-specific rules plus universal FFI rules</h2>
<p><code>RustFfiAuditor</code> splits its rules into two layers. Rust-specific rules run on Rust modules and recover <code>into_raw/from_raw</code>, <code>as_ptr</code>, and borrow-dangling semantics. Universal FFI rules run on all modules and catch stack escape or unsafe boundary calls.</p>
<h2 id="rule-surface">Rule surface</h2>
<p><code>RustFfiAuditor</code> is defined at <code>src/pass/analysis/rust_ffi_auditor.zig:63</code>. Its function-level logic covers Rust-specific patterns and general FFI boundary checks:</p>
<ul>
<li><code>into_raw</code> without a matching <code>from_raw</code>;</li>
<li><code>as_ptr</code> borrow escape;</li>
<li>Rust allocator and C <code>free</code> mismatch;</li>
<li>ownership-transfer protocol violations;</li>
<li>dangling <code>as_ptr</code> after parent object drop;</li>
<li>unsafe FFI calls;</li>
<li>stack address escape to <code>extern C</code>.</li>
</ul>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[RustFfiAuditor] --> B[Box::into_raw / from_raw]
A --> C[as_ptr borrow escape]
A --> D[allocator mismatch]
A --> E[ownership transfer]
A --> F[dangling as_ptr]
A --> G[unsafe FFI call]
A --> H[stack escape]
</div><h2 id="as-ptr-borrow-escape-recovering-a-lifetime-risk-from-ir-calls"><code>as_ptr</code> borrow escape: recovering a lifetime risk from IR calls</h2>
<p><code>detectAsPtrEscape</code> is implemented at <code>src/pass/analysis/rust_ffi_auditor.zig:180</code>. It iterates LLVM functions, basic blocks, and instructions; handles only <code>LLVMCall</code> and <code>LLVMInvoke</code>; retrieves the callee from the final operand; reads the callee name; and matches Rust <code>as_ptr</code> patterns.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[LLVM Function] --> B[BasicBlock iterator]
B --> C[Instruction iterator]
C --> D{opcode == call/invoke?}
D -->|No| C
D -->|Yes| E[Read callee operand]
E --> F[LLVMGetValueName]
F --> G{isRustAsPtrCall?}
G -->|No| C
G -->|Yes| H[addFinding + ctx.addIssue]
</div>
<p>The risk is that <code>String</code> or <code>Vec</code> <code>as_ptr</code> returns a borrowed pointer. If C stores it, the Rust object may be dropped while C still holds the address.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
sequenceDiagram
participant R as Rust local Vec/String
participant P as raw pointer from as_ptr
participant C as C FFI callee
R->>P: as_ptr()
P->>C: pass pointer
R->>R: local value dropped
C->>P: later use
P-->>C: dangling pointer risk
</div>
<p>At <code>src/pass/analysis/rust_ffi_auditor.zig:212</code>, the rule creates a <code>borrow_escape</code> issue through <code>Issue.initWithReason</code>, with a reason explaining that a local <code>String/Vec</code> pointer passed to extern C may dangle.</p>
<h2 id="into-raw-from-raw-ownership-transfer-should-close-correctly"><code>into_raw/from_raw</code>: ownership transfer should close correctly</h2>
<p><code>Box::into_raw</code> converts Rust-managed heap ownership into a raw pointer. The caller must ensure the later deallocation protocol is correct. Missing restoration can leak; double restoration can double free; C-side release can produce allocator mismatch depending on allocation protocol.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[Box<T>] --> B[Box::into_raw]
B --> C[*mut T]
C --> D{Later protocol}
D -->|from_raw exactly once| E[Ownership restored]
D -->|Never restored| F[Leak risk]
D -->|C free + Rust drop| G[Double free / allocator mismatch]
</div>
<p><code>into_raw</code> alone is not a vulnerability. The finding depends on the surrounding protocol and subsequent pointer flow.</p>
<h2 id="cross-language-allocator-mismatch">Cross-language allocator mismatch</h2>
<p><code>detectCrossLangMismatch</code> starts at <code>src/pass/analysis/rust_ffi_auditor.zig:230</code>. It iterates call/invoke instructions and attempts to identify Rust allocation paired with C deallocation.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
sequenceDiagram
participant RA as Rust allocator
participant IR as LLVM IR pointer
participant CF as C free
RA->>IR: allocate / expose pointer
IR->>CF: pointer crosses ABI
CF->>CF: free(pointer)
CF-->>RA: allocator ownership contract may be violated
</div>
<p>The accuracy of this kind of check depends on symbol names, preserved call relationships, wrappers, inlining, and custom allocators.</p>
<h2 id="general-ffi-checks">General FFI checks</h2>
<p>The auditor also runs checks that are not Rust-only, such as unsafe FFI call scanning and stack address escape. Stack escape is relevant when a pointer to a local object is passed to C and then stored beyond the call.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[alloca / local stack object] --> B[Take address]
B --> C[Pass to extern C]
C --> D{Does C store it?}
D -->|Yes| E[Dangling after return]
D -->|No| F[Depends on call-duration use]
</div><h2 id="summary">Summary</h2>
<p><code>RustFfiAuditor</code> maps Rust ownership and borrowing concepts onto IR-level call and pointer patterns. It should be described as static protocol recovery and checking, with accuracy bounded by available IR information.</p>
<h2 id="source-breakdown-rust-rules-and-universal-ffi-rules-are-intentionally-separated">Source breakdown: Rust rules and universal FFI rules are intentionally separated</h2>
<p><code>RustFfiAuditor.auditFunction</code> around <code>src/pass/analysis/rust_ffi_auditor.zig:120</code> splits rules into Rust-specific and universal FFI boundary checks.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> auditFunction</span><span>(self</span><span style="color: #94E2D5;">: *</span><span>RustFfiAuditor</span><span style="color: #9399B2;">,</span><span> func</span><span style="color: #94E2D5;">:</span><span> c</span><span style="color: #94E2D5;">.</span><span>LLVMValueRef</span><span style="color: #9399B2;">,</span><span> ctx</span><span style="color: #94E2D5;">: *</span><span>PassContext</span><span style="color: #9399B2;">,</span><span> diag</span><span style="color: #94E2D5;">: *</span><span>DiagnosticWriter) </span><span style="color: #94E2D5;">!</span><span style="color: #CBA6F7;">void</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> func_name</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> getFunctionName</span><span>(func)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> is_rust</span><span style="color: #94E2D5;"> =</span><span> ctx</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">isRustModule</span><span>()</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (is_rust) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (ctx</span><span style="color: #94E2D5;">.</span><span>rust_into_raw_set</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">contains</span><span>(</span><span style="color: #F38BA8;font-style: italic;">@intFromPtr</span><span>(c</span><span style="color: #94E2D5;">.</span><span style="color: #F9E2AF;font-style: italic;">LLVMGetValueName</span><span>(func)))) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (ctx</span><span style="color: #94E2D5;">.</span><span>rust_from_raw_set</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">count</span><span>() </span><span style="color: #94E2D5;">==</span><span style="color: #FAB387;"> 0</span><span>) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">addFinding</span><span>(</span><span style="color: #94E2D5;">.</span><span>{</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>issue_type</span><span style="color: #94E2D5;"> = .</span><span>unpaired_into_raw</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>severity</span><span style="color: #94E2D5;"> = .</span><span>high</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>confidence</span><span style="color: #94E2D5;"> =</span><span style="color: #FAB387;"> 0.75</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>reason</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "into_raw() called but no matching from_raw() in module"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>location</span><span style="color: #94E2D5;"> =</span><span> Location</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">init</span><span>(func_name)</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> })</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">detectAsPtrEscape</span><span>(func</span><span style="color: #9399B2;">,</span><span> ctx</span><span style="color: #9399B2;">,</span><span> diag)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">detectCrossLangMismatch</span><span>(func</span><span style="color: #9399B2;">,</span><span> ctx</span><span style="color: #9399B2;">,</span><span> diag)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">detectOwnershipTransferViolations</span><span>(func</span><span style="color: #9399B2;">,</span><span> ctx</span><span style="color: #9399B2;">,</span><span> diag)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">detectAsPtrDangling</span><span>(func</span><span style="color: #9399B2;">,</span><span> ctx</span><span style="color: #9399B2;">,</span><span> diag)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">detectUnsafeFfiCalls</span><span>(func)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> try</span><span> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">detectStackEscapeToFFI</span><span>(func</span><span style="color: #9399B2;">,</span><span> ctx</span><span style="color: #9399B2;">,</span><span> diag)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>This split matters. <code>as_ptr</code>, <code>into_raw</code>, and <code>from_raw</code> are Rust semantics. Forcing those rules onto C or Zig modules would create noise. By contrast, stack-address escape and unsafe FFI calls are cross-language risks and should run on every module.</p>
<h2 id="how-it-works-language-gating-is-precision-not-conservatism">How it works: language gating is precision, not conservatism</h2>
<p>At LLVM level many values become pointers, but the source-level semantics still differ. OmniScope uses <code>ctx.isRustModule()</code> to match rules with semantic origin:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>Rust module</span></span>
<span class="giallo-l"><span> -> ownership transfer / borrow escape / from_raw pairing</span></span>
<span class="giallo-l"><span>Any module</span></span>
<span class="giallo-l"><span> -> FFI boundary / stack escape / unsafe call surface</span></span></code></pre>
<p>That is also why <code>PassContext</code> stores <code>module_language</code>, and why the Pipeline calls <code>ctx.initModuleLanguage(self.module)</code> before any pass runs. Language detection is not a UI label; it is a rule selector.</p>
<h2 id="evidence-chain-a-finding-is-not-the-final-report">Evidence chain: a finding is not the final report</h2>
<p><code>RustFfiAuditor</code> first produces <code>RustFfiFinding</code> records with <code>func_name</code>, <code>issue_type</code>, <code>severity</code>, <code>confidence</code>, <code>reason</code>, and <code>location</code>. That is an internal semantic layer, not the final output. Only later does the unified issue path convert it into the shared <code>Issue</code> schema.</p>
<p>That two-stage structure matters because the auditor can keep Rust-specific meaning such as unpaired <code>into_raw</code>, borrow escape, or stack escape, while the output layer preserves a consistent schema shared with the rest of OmniScope.</p>
<h2 id="one-concrete-rule-unpaired-into-raw">One concrete rule: unpaired <code>into_raw</code></h2>
<p>The problem with <code>Box::into_raw</code> is not "raw pointers are dangerous" in the abstract. It is that Rust intentionally gives up automatic drop and transfers release responsibility into an external protocol. A safe recovery path requires a matching <code>from_raw</code> or equivalent reclamation.</p>
<p>That is why the code checks for module-level pairing: if <code>into_raw</code> semantics exist but no <code>from_raw</code> semantics exist anywhere in the module, OmniScope emits <code>unpaired_into_raw</code>. It is a protocol-completeness check, not a function blacklist.</p>
<p>The rule still has limits: if the release path lives in another shared library, another module, or a wrapper function, a static module-local check must lower confidence. That is why the code uses <code>0.75</code> instead of <code>1.0</code>.</p>
Reporting Pipeline: Issue Objects, JSON, SARIF, and Engineering Integration
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/07-output-and-integration/
<h1 id="reporting-pipeline-issue-objects-json-sarif-and-engineering-integration">Reporting Pipeline: Issue Objects, JSON, SARIF, and Engineering Integration</h1>
<p>Static analysis results need to be reviewable and machine-readable. OmniScope’s reporting path starts with <code>Issue</code> objects, then emits text, JSON, or SARIF through <code>emitOutput</code>.</p>
<h2 id="start-with-the-problem-plain-text-alerts-do-not-integrate-well">Start with the problem: plain-text alerts do not integrate well</h2>
<p>Cross-language audit results are not only read by a human once. CI needs artifacts and status decisions, security platforms consume SARIF, secondary tooling reads JSON, and developers need reason, confidence, and FFI context. If each pass prints text directly, that information is hard to consume reliably.</p>
<h2 id="omniscope-s-entry-point-issue-as-the-shared-middle-layer">OmniScope’s entry point: Issue as the shared middle layer</h2>
<p>OmniScope makes passes produce structured <code>Issue</code> values first, then lets the output layer serialize them as text, JSON, or SARIF. Analysis logic, evidence fields, and engineering integration stay separate: passes find problems, <code>Issue</code> stores evidence, and <code>emitOutput</code> decides how to deliver it.</p>
<h2 id="issue-is-the-shared-result-type">Issue is the shared result type</h2>
<p><code>Issue</code> is defined in <code>src/diag/issue.zig</code>, and issue kinds are exposed from <code>src/common/types.zig:151</code>. The structure carries kind, message, location, severity, confidence, confidence level, reason, FFI boundary, trace, and classification.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
classDiagram
class Issue {
kind
message
location
severity
confidence
confidence_level
reason
ffi_boundary
trace
classification
}
</div>
<p>These fields support different review needs:</p>
<ul>
<li><code>severity</code> helps triage.</li>
<li><code>confidence</code> and <code>confidence_level</code> distinguish stronger evidence from heuristics.</li>
<li><code>reason</code> records the rule rationale.</li>
<li><code>ffi_boundary</code> marks cross-language context.</li>
<li><code>trace</code> leaves room for evidence paths.</li>
<li><code>classification</code> separates FFI-boundary findings from local-only findings.</li>
</ul>
<h2 id="from-pass-to-issue">From Pass to Issue</h2>
<p>When a pass finds something reportable, it creates an <code>Issue</code> with constructors such as <code>Issue.init</code> or <code>Issue.initWithReason</code>, then calls <code>ctx.addIssue</code>. The entry point is <code>src/pass/pass.zig:458</code>.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[Analysis Pass] --> B[Rule match]
B --> C[Construct Issue]
C --> D[ctx.addIssue]
D --> E[DataFlowGraph / Issue store]
E --> F[Pipeline.getIssues]
F --> G[emitOutput]
</div>
<p>This decouples rule logic from output format. Passes produce structured findings; the main program decides how to serialize them.</p>
<h2 id="emitoutput-dispatches-formats"><code>emitOutput</code> dispatches formats</h2>
<p><code>emitOutput</code> is implemented at <code>src/main.zig:207</code>. The JSON branch calls <code>formatIssuesAsJson</code> at <code>src/main.zig:494</code>; the SARIF branch uses <code>SarifOutput</code>, initialized at <code>src/main.zig:232</code>; file output is controlled by <code>config.output_file</code>.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart LR
A[issues + func_count + time_ms] --> B[emitOutput]
B --> C{OutputFormat}
C -->|json| D[formatIssuesAsJson]
C -->|sarif| E[SarifOutput.generate]
C -->|text| F[terminal diagnostics]
D --> G[stdout or file]
E --> G
F --> H[developer console]
</div><h2 id="sarif-integration">SARIF integration</h2>
<p><code>SarifOutput</code> is defined at <code>src/output/sarif.zig:36</code>, with file writing at <code>src/output/sarif.zig:167</code>. SARIF lets results be consumed by GitHub Code Scanning, CI systems, and security dashboards.</p>
<div class="mermaid-direct" data-series="omniscope" data-name="graph.mmd">
flowchart TD
A[OmniScope Issue] --> B[SARIF Result]
B --> C[sarif.json]
C --> D[CI Artifact]
C --> E[GitHub Code Scanning]
C --> F[Security Dashboard]
</div><h2 id="example-commands">Example commands</h2>
<p>Local inspection:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">omniscope</span><span style="color: #A6E3A1;"> input.ll</span></span></code></pre>
<p>Structured JSON:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">omniscope</span><span style="color: #A6E3A1;"> --json -o omniscope-report.json input.ll</span></span></code></pre>
<p>SARIF for CI or code scanning:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">omniscope</span><span style="color: #A6E3A1;"> --sarif -o omniscope.sarif input.ll</span></span></code></pre>
<p>FFI-focused mode:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">omniscope</span><span style="color: #A6E3A1;"> --ffi-only input.ll</span></span></code></pre><h2 id="confidence-and-review-boundaries">Confidence and review boundaries</h2>
<p>Confidence is not decorative. Cross-language static analysis can be affected by optimization, missing symbols, debug information quality, wrappers, and custom allocators. Exposing confidence, reason, and trace fields helps reviewers inspect the analyzer’s reasoning instead of relying only on alert color.</p>
<h2 id="summary">Summary</h2>
<p>The reporting path separates analysis from serialization: passes produce structured issues, <code>emitOutput</code> selects the output format, and JSON/SARIF make the results usable in engineering workflows.</p>
<h2 id="source-breakdown-emitoutput-is-format-dispatch-not-analysis-logic">Source breakdown: <code>emitOutput</code> is format dispatch, not analysis logic</h2>
<p><code>emitOutput</code> in <code>src/main.zig:207</code> only dispatches by <code>config.output_format</code>. It does not reinterpret risk.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> emitOutput</span><span>(allocator</span><span style="color: #94E2D5;">:</span><span> std</span><span style="color: #94E2D5;">.</span><span>mem</span><span style="color: #94E2D5;">.</span><span>Allocator</span><span style="color: #9399B2;">,</span><span> issues</span><span style="color: #94E2D5;">:</span><span> []</span><span style="color: #CBA6F7;">const</span><span> Issue</span><span style="color: #9399B2;">,</span><span> func_count</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span> time_ms</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">,</span><span> config</span><span style="color: #94E2D5;">:</span><span> Config) </span><span style="color: #94E2D5;">!</span><span style="color: #CBA6F7;">void</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (issues</span><span style="color: #94E2D5;">.</span><span>len</span><span style="color: #94E2D5;"> ==</span><span style="color: #FAB387;"> 0</span><span style="color: #CBA6F7;"> and</span><span> config</span><span style="color: #94E2D5;">.</span><span>output_format</span><span style="color: #94E2D5;"> == .</span><span>text) </span><span style="color: #CBA6F7;">return</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (config</span><span style="color: #94E2D5;">.</span><span>output_format</span><span style="color: #94E2D5;"> == .</span><span>json) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> json_output</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> formatIssuesAsJson</span><span>(allocator</span><span style="color: #9399B2;">,</span><span> issues</span><span style="color: #9399B2;">,</span><span> func_count</span><span style="color: #9399B2;">,</span><span> time_ms) </span><span style="color: #CBA6F7;">catch</span><span style="color: #94E2D5;"> |</span><span>err</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span> log</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">err</span><span>(</span><span style="color: #A6E3A1;">"Failed to format JSON output: {}</span><span style="color: #F5C2E7;">\n</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{err})</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> } </span><span style="color: #CBA6F7;">else if</span><span> (config</span><span style="color: #94E2D5;">.</span><span>output_format</span><span style="color: #94E2D5;"> == .</span><span>sarif) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> var</span><span> sarif</span><span style="color: #94E2D5;"> =</span><span> SarifOutput</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">init</span><span>(allocator</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "OmniScope"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "0.1.8"</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span> sarif_output</span><span style="color: #94E2D5;"> =</span><span> sarif</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">generate</span><span>(issues) </span><span style="color: #CBA6F7;">catch</span><span style="color: #94E2D5;"> |</span><span>err</span><span style="color: #94E2D5;">|</span><span> {</span></span>
<span class="giallo-l"><span> log</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">err</span><span>(</span><span style="color: #A6E3A1;">"Failed to generate SARIF output: {}</span><span style="color: #F5C2E7;">\n</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{err})</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> } </span><span style="color: #CBA6F7;">else</span><span> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> (issues</span><span style="color: #94E2D5;">.</span><span>len</span><span style="color: #94E2D5;"> ></span><span style="color: #FAB387;"> 0</span><span>) {</span></span>
<span class="giallo-l"><span> log</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">info</span><span>(</span><span style="color: #A6E3A1;">"Issues detected: {d}</span><span style="color: #F5C2E7;">\n</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{issues</span><span style="color: #94E2D5;">.</span><span>len})</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span> }</span></span>
<span class="giallo-l"><span>}</span></span></code></pre>
<p>That boundary is important. Analysis passes should not know whether a finding will be consumed by GitHub Code Scanning, a CI artifact, or a terminal. The output layer should not reinterpret severity or confidence either, otherwise JSON and SARIF could drift semantically.</p>
<h2 id="source-breakdown-json-exposes-evidence-strength">Source breakdown: JSON exposes evidence strength</h2>
<p><code>formatIssuesAsJson</code> in <code>src/main.zig:494</code> writes more than kind, message, and location. It includes summary, confidence, confidence score, CWE, and reason.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="zig"><span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #A6E3A1;">"{</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">schema_version</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">1.0.0</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">,</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">tool</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">omniscope</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">,</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">tool_version</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">0.1.8</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">,</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">timestamp</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:"</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #A6E3A1;">",</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">summary</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:{"</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">print</span><span>(</span><span style="color: #A6E3A1;">"</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">functions</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:{d},</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">issues</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:{d},</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">time_ms</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:{d}"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{ func_count</span><span style="color: #9399B2;">,</span><span> issues</span><span style="color: #94E2D5;">.</span><span>len</span><span style="color: #9399B2;">,</span><span> analysis_time_ms })</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #A6E3A1;">"{</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">id</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">"</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">print</span><span>(</span><span style="color: #A6E3A1;">"OMI-{d:0>3}"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{idx</span><span style="color: #94E2D5;"> +</span><span style="color: #FAB387;"> 1</span><span>})</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #A6E3A1;">"</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">,</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">kind</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">"</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #F38BA8;font-style: italic;">@tagName</span><span>(issue</span><span style="color: #94E2D5;">.</span><span>kind))</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #A6E3A1;">"</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">,</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">severity</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">"</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #F38BA8;font-style: italic;">@tagName</span><span>(issue</span><span style="color: #94E2D5;">.</span><span>severity))</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #A6E3A1;">"</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">,</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">confidence</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">"</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(issue</span><span style="color: #94E2D5;">.</span><span>confidence_level</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">toString</span><span>())</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #A6E3A1;">"</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">,</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">confidence_score</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:"</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">print</span><span>(</span><span style="color: #A6E3A1;">"{d:.2}"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{issue</span><span style="color: #94E2D5;">.</span><span>confidence})</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">writeAll</span><span>(</span><span style="color: #A6E3A1;">",</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">cwe_id</span><span style="color: #F5C2E7;">\"</span><span style="color: #A6E3A1;">:"</span><span>)</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">try</span><span> writer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">print</span><span>(</span><span style="color: #A6E3A1;">"{d}"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> .</span><span>{cwe_id})</span><span style="color: #9399B2;">;</span></span></code></pre>
<p>The reporting design is built around auditability. <code>confidence_score</code> exposes evidence strength, <code>reason</code> explains the rule, <code>cwe_id</code> connects to security programs, and <code>summary.time_ms</code> gives CI a performance signal.</p>
<h2 id="issue-is-the-external-contract"><code>Issue</code> is the external contract</h2>
<p><code>Issue</code> in <code>src/diag/issue.zig</code> standardizes findings as kind, message, location, severity, confidence, confidence level, reason, FFI boundary, trace, and classification.</p>
<p>Those fields form OmniScope’s external contract:</p>
<ul>
<li><code>kind</code> and <code>severity</code> describe type and priority.</li>
<li><code>confidence</code> and <code>confidence_level</code> describe evidence strength.</li>
<li><code>reason</code> explains why the rule fired.</li>
<li><code>ffi_boundary</code> and <code>classification</code> distinguish local-only issues from boundary-relevant issues.</li>
<li><code>trace</code> leaves room for evidence paths and visualization.</li>
</ul>
<p>Keeping these fields in <code>Issue</code>, rather than scattering them across JSON/SARIF formatters, makes the report schema part of the analysis model rather than a string-rendering detail.</p>
<h2 id="engineering-integration-needs-stable-schema-not-pretty-output">Engineering integration needs stable schema, not pretty output</h2>
<p>Terminal output is useful for demos, but engineering systems need stable structure. CI needs artifacts, security platforms need SARIF, secondary tooling needs JSON, and developers need reason and trace. OmniScope’s <code>Issue -> JSON/SARIF/Text</code> path keeps those consumers on the same semantics.</p>
<p>The important part is bigger than "supports JSON and SARIF": OmniScope models cross-language findings as structured issues with confidence, CWE, location, FFI classification, and explanatory evidence, then serializes that contract into machine-consumable formats.</p>
GoAgent Source Deep Dive Series
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/00-series-index/
<h1 id="goagent-source-deep-dive-series">GoAgent Source Deep Dive Series</h1>
<p>This series walks through the GoAgent multi-agent framework implementation, from API client to embedding service.</p>
<p>GoAgent is a custom multi-agent framework in Go. It is not a wrapper around an existing framework — it is built from scratch to explore how agents can collaborate, remember, and execute tasks through a clean API boundary.</p>
<p>The series follows one narrative thread: <strong>Problem → Limitations of existing approaches → GoAgent's approach → Architecture naturally emerges</strong>.</p>
<h2 id="articles">Articles</h2>
<ol>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/01-api-client/">API Client: Converging Framework Capabilities Into a Usable Entry Point</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/02-llm-client/">LLM Client: Unifying Multi-Model Call Boundaries</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/03-agent-system/">Agent System: The Leader/Sub-Agent Collaboration Skeleton</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/04-ahp-protocol/">AHP Protocol: Messages, Queues, and Heartbeats</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/05-memory-system/">Memory System: Sessions, Tasks, and Distilled Memories</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/06-tool-system/">Tool System: Registration, Capability Matching, and Execution</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/07-workflow-engine/">Workflow Engine: DAG-Based Agent Orchestration</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/08-storage-retrieval/">Storage and Retrieval: PostgreSQL, pgvector, and Hybrid Search</a></li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/09-embedding-service/">Embedding Service: The Engineering Boundary of Vector Generation</a></li>
</ol>
<h2 id="reading-order">Reading Order</h2>
<p>Read in the numbered order above. Each article builds on the previous one:</p>
<ul>
<li>Start with <strong>API Client</strong> to understand how internal modules are isolated from external callers.</li>
<li><strong>LLM Client</strong> shows how the framework stays provider-agnostic.</li>
<li><strong>Agent System</strong> introduces the leader/sub-agent collaboration model.</li>
<li><strong>AHP Protocol</strong> explains how agents communicate.</li>
<li><strong>Memory System</strong> covers sessions, tasks, and distilled memories.</li>
<li><strong>Tool System</strong> shows how agents execute actions, not just talk.</li>
<li><strong>Workflow Engine</strong> handles multi-step task orchestration.</li>
<li><strong>Storage and Retrieval</strong> covers where memories and knowledge live.</li>
<li><strong>Embedding Service</strong> closes the loop with vector generation.</li>
</ul>
<h2 id="source-reading-map">Source Reading Map</h2>
<ul>
<li>Entry: <code>api/client.go</code>, <code>api/simple_client.go</code>, <code>api/workflow_client.go</code></li>
<li>LLM: <code>internal/llm/output/</code>, <code>internal/llm/provider/</code></li>
<li>Agents: <code>internal/agents/leader/</code>, <code>internal/agents/sub/</code></li>
<li>Protocol: <code>internal/protocol/ahp/</code></li>
<li>Memory: <code>internal/memory/</code></li>
<li>Tools: <code>internal/tools/</code></li>
<li>Workflow: <code>internal/workflow/</code></li>
<li>Storage: <code>internal/storage/postgres/</code></li>
<li>Embedding: <code>internal/embedding/</code></li>
</ul>
<h2 id="writing-principles">Writing Principles</h2>
<ul>
<li>Describe implemented source-code behavior, not wishful roadmap items.</li>
<li>Include code references so readers can verify claims in the repository.</li>
<li>Use Mermaid diagrams only; no screenshots or image dependencies.</li>
<li>Analyze design trade-offs, not just advantages.</li>
</ul>
GoAgent Source Deep Dive 01: API Client — Converging Framework Capabilities Into a Usable Entry Point
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/01-api-client/
<h1 id="goagent-source-deep-dive-01-api-client-converging-framework-capabilities-into-a-usable-entry-point">GoAgent Source Deep Dive 01: API Client — Converging Framework Capabilities Into a Usable Entry Point</h1>
<h2 id="the-problem-how-should-a-framework-s-internal-modules-be-called-externally">The Problem: How Should a Framework's Internal Modules Be Called Externally</h2>
<p>You've built a multi-agent framework with LLM adapters, task orchestrators, memory managers, vector retrieval, workflow engines — a dozen packages, dozens of interfaces. Now someone wants to use your framework. How do you let them?</p>
<p>The most intuitive approach: let callers directly <code>import</code> your internal packages. But problems appear immediately.</p>
<h2 id="limitations-of-existing-approaches">Limitations of Existing Approaches</h2>
<p><strong>Approach A: Expose internal packages directly</strong></p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #CBA6F7;">import</span><span style="color: #A6E3A1;"> "github.com/Timwood0x10/goagent/internal/llm/output"</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">import</span><span style="color: #A6E3A1;"> "github.com/Timwood0x10/goagent/internal/agents/leader"</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">import</span><span style="color: #A6E3A1;"> "github.com/Timwood0x10/goagent/internal/memory"</span></span></code></pre>
<p>Problems:</p>
<ul>
<li>Rename an internal variable and every caller has to change.</li>
<li>Callers need to understand internal module topology to assemble a working Agent.</li>
<li>Can't mock anything in tests — you have to boot the entire internal dependency chain.</li>
</ul>
<p><strong>Approach B: Build a massive Facade package</strong></p>
<p>Cram everything into one package, expose one giant struct.</p>
<p>Problems:</p>
<ul>
<li>The struct balloons to hundreds of methods.</li>
<li>Callers who only need LLM capabilities are forced to depend on Agent, Memory, and Workflow types.</li>
<li>No layering; changing one service's interface ripples through everything.</li>
</ul>
<p>Neither approach is a good answer. What GoAgent needs is: <strong>a thin isolation boundary that keeps internal complexity outside, while letting callers depend only on the capabilities they need</strong>.</p>
<h2 id="goagent-s-approach">GoAgent's Approach</h2>
<p>GoAgent uses three-layer Client + service interface isolation:</p>
<ol>
<li><strong>Service interface layer</strong> (<code>api/core/</code>): Defines <code>LLMService</code>, <code>AgentService</code>, <code>MemoryService</code>, <code>RetrievalService</code> interfaces. Callers depend on interfaces, not implementations.</li>
<li><strong>Capability layer</strong> (<code>api.Client</code>): Holds all service instances, exposes service access through methods.</li>
<li><strong>Entry layer</strong> (<code>SimpleClient</code>, <code>WorkflowClient</code>): Minimal encapsulation for specific scenarios.</li>
</ol>
<p>This isn't a "designed architecture" — it's a layering that naturally emerges from the isolation requirement:</p>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart TB
subgraph "Callers only see these"
SimpleClient[SimpleClient<br/>One Execute call]
Client[api.Client<br/>Access all services]
WorkflowClient[WorkflowClient<br/>Run workflows]
end
subgraph "Callers only depend on these interfaces"
LLMService[LLMService]
AgentService[AgentService]
MemoryService[MemoryService]
RetrievalService[RetrievalService]
end
subgraph "Callers never see these"
LLM[llm/output]
Agents[agents/leader+sub]
Memory[memory/]
Storage[storage/postgres]
end
SimpleClient --> Client
Client --> LLMService
Client --> AgentService
Client --> MemoryService
Client --> RetrievalService
LLMService -.->|impl| LLM
AgentService -.->|impl| Agents
MemoryService -.->|impl| Memory
RetrievalService -.->|impl| Storage
</div>
<p>Key point: <strong>callers never see any implementation under <code>internal/</code></strong>. Dashed lines mean "implementation relationships are completed inside the api package, invisible to the outside."</p>
<h2 id="architecture-naturally-emerges">Architecture Naturally Emerges</h2>
<h3 id="service-interfaces-the-core-of-isolation">Service Interfaces: The Core of Isolation</h3>
<p>Interfaces defined in <code>api/core/</code> are the linchpin:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// api/core/llm.go</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> LLMService</span><span style="color: #CBA6F7;"> interface</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Generate</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> prompt</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> opts</span><span style="color: #94E2D5;"> ...</span><span style="color: #F9E2AF;font-style: italic;">GenerateOption</span><span style="color: #9399B2;">) (</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> GenerateSimple</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> prompt</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">) (</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> GenerateEmbedding</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> text</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">) ([]</span><span style="color: #CBA6F7;">float64</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> GetConfig</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">LLMConfig</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> IsEnabled</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> bool</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> GetProvider</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> string</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> GetModel</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> string</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Callers don't know whether it's OpenRouter, Ollama, or OpenAI underneath. This isn't just "wrapping" — it's <strong>dependency inversion</strong>: callers depend on abstract interfaces, implementation details are isolated inside the api package.</p>
<h3 id="api-client-the-convergence-point">api.Client: The Convergence Point</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// api/client.go</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> Client</span><span style="color: #CBA6F7;"> struct</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> config</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">Config</span></span>
<span class="giallo-l"><span> llm</span><span style="color: #F9E2AF;font-style: italic;"> LLMService</span></span>
<span class="giallo-l"><span> agent</span><span style="color: #F9E2AF;font-style: italic;"> AgentService</span></span>
<span class="giallo-l"><span> memory</span><span style="color: #F9E2AF;font-style: italic;"> MemoryService</span></span>
<span class="giallo-l"><span> retrieval</span><span style="color: #F9E2AF;font-style: italic;"> RetrievalService</span></span>
<span class="giallo-l"><span> closed</span><span style="color: #CBA6F7;"> bool</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p><code>Client</code> does no business logic — only two things:</p>
<ol>
<li>Hold service instances.</li>
<li>Expose service access through methods.</li>
</ol>
<h3 id="simpleclient-the-minimal-entry">SimpleClient: The Minimal Entry</h3>
<p><code>SimpleClient</code> solves the "I just want to call an LLM, not understand the whole framework" need:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// api/client/simple.go:73</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">func</span><span style="color: #9399B2;"> (</span><span style="color: #EBA0AC;font-style: italic;">c </span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">SimpleClient</span><span style="color: #9399B2;">)</span><span style="color: #89B4FA;font-style: italic;"> Execute</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> query</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">) (</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> query</span><span style="color: #94E2D5;"> ==</span><span style="color: #A6E3A1;"> ""</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> ""</span><span style="color: #9399B2;">,</span><span> errors</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">New</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"query cannot be empty"</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> llm</span><span style="color: #94E2D5;"> :=</span><span> c</span><span style="color: #9399B2;">.</span><span>client</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">LLM</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> llm</span><span style="color: #94E2D5;"> ==</span><span style="color: #F38BA8;"> nil</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> ""</span><span style="color: #9399B2;">,</span><span> errors</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">New</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"LLM service not available"</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> prompt</span><span style="color: #94E2D5;"> :=</span><span> fmt</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">Sprintf</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"Please help with: %s"</span><span style="color: #9399B2;">,</span><span> query</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span> llm</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">GenerateSimple</span><span style="color: #9399B2;">(</span><span>ctx</span><span style="color: #9399B2;">,</span><span> prompt</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p><strong>Important clarification</strong>: <code>SimpleClient.Execute</code> does NOT enter multi-agent orchestration — it wraps the query into a prompt and calls the LLM directly. This isn't a deficiency, it's design: lowering the barrier and providing full capabilities are two different entry points.</p>
<h2 id="design-trade-offs">Design Trade-offs</h2>
<h3 id="trade-off-1-manual-di-vs-framework-di">Trade-off 1: Manual DI vs Framework DI</h3>
<p>GoAgent manually assembles services, no Wire or dig. Cost: slightly more initialization code. Benefit: dependencies are explicitly visible in code, no DI framework magic to understand.</p>
<h3 id="trade-off-2-simpleclient-doesn-t-go-through-agent-orchestration">Trade-off 2: SimpleClient Doesn't Go Through Agent Orchestration</h3>
<p>Intentional. Lowering the barrier and providing full capabilities are two different needs that shouldn't be crammed into one entry point.</p>
<h3 id="trade-off-3-no-caching-at-api-layer">Trade-off 3: No Caching at API Layer</h3>
<p>Caching strategies are decided by individual internal services. The API layer stays thin and simple, avoiding cache consistency issues spreading across layers.</p>
<h2 id="summary">Summary</h2>
<p>API Client isn't "wrapping HTTP." It's GoAgent's isolation boundary: callers depend on interfaces, not implementations; internal refactoring doesn't affect external integration. Three Client types target different scenarios, enabling progressive adoption. The architecture wasn't pre-designed — it naturally emerged from the isolation requirement.</p>
GoAgent Source Deep Dive 02: LLM Client — Unifying Multi-Model Call Boundaries
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/02-llm-client/
<h1 id="goagent-source-deep-dive-02-llm-client-unifying-multi-model-call-boundaries">GoAgent Source Deep Dive 02: LLM Client — Unifying Multi-Model Call Boundaries</h1>
<h2 id="the-problem-how-to-keep-an-agent-from-being-tied-to-one-model-provider">The Problem: How to Keep an Agent From Being Tied to One Model Provider</h2>
<p>You've built an agent that needs to call an LLM to understand user input, generate task plans, and perform reasoning. The most intuitive approach is to hardcode HTTP requests to the OpenAI API directly in agent code.</p>
<p>But problems appear immediately.</p>
<h2 id="limitations-of-existing-approaches">Limitations of Existing Approaches</h2>
<p><strong>Approach A: Hardcode one provider</strong></p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span>resp</span><span style="color: #9399B2;">,</span><span> err</span><span style="color: #94E2D5;"> :=</span><span> http</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">Post</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"https://api.openai.com/v1/chat/completions"</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> ...</span><span style="color: #9399B2;">)</span></span></code></pre>
<p>Problems:</p>
<ul>
<li>Switch models (e.g., local Ollama deployment), and every call site needs changing.</li>
<li>Can't mock in tests — you have to actually call the OpenAI API.</li>
<li>Different providers have different request formats, auth methods, and response structures.</li>
</ul>
<p><strong>Approach B: Provider switching inside each agent</strong></p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #CBA6F7;">func</span><span style="color: #9399B2;"> (</span><span style="color: #EBA0AC;font-style: italic;">a </span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">Agent</span><span style="color: #9399B2;">)</span><span style="color: #89B4FA;font-style: italic;"> callLLM</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">prompt</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">) (</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> switch</span><span> a</span><span style="color: #9399B2;">.</span><span>config</span><span style="color: #9399B2;">.</span><span>Provider</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> case</span><span style="color: #A6E3A1;"> "openai"</span><span style="color: #9399B2;">:</span><span style="color: #CBA6F7;"> return</span><span style="color: #89B4FA;font-style: italic;"> callOpenAI</span><span style="color: #9399B2;">(</span><span>prompt</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> case</span><span style="color: #A6E3A1;"> "ollama"</span><span style="color: #9399B2;">:</span><span style="color: #CBA6F7;"> return</span><span style="color: #89B4FA;font-style: italic;"> callOllama</span><span style="color: #9399B2;">(</span><span>prompt</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> case</span><span style="color: #A6E3A1;"> "openrouter"</span><span style="color: #9399B2;">:</span><span style="color: #CBA6F7;"> return</span><span style="color: #89B4FA;font-style: italic;"> callOpenRouter</span><span style="color: #9399B2;">(</span><span>prompt</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Problems:</p>
<ul>
<li>Every agent repeats this switch.</li>
<li>Agent code couples with provider details.</li>
<li>Adding a new provider requires modifying every agent.</li>
</ul>
<p>Both approaches tie the agent to model invocation details. What GoAgent needs is: <strong>a unified model call boundary where upper layers only depend on <code>Generate</code> capability, without caring which provider is underneath</strong>.</p>
<h2 id="goagent-s-approach">GoAgent's Approach</h2>
<p>GoAgent uses a two-layer architecture:</p>
<ul>
<li><strong>Bottom layer</strong>: A lightweight HTTP client that communicates directly with providers.</li>
<li><strong>Top layer</strong>: <code>LLMAdapter</code> interface + Factory pattern, unifying behavior and supporting structured output.</li>
</ul>
<p>Two layers isn't redundancy — it's division of labor: the bottom handles "how to send requests," the top handles "how to use the model."</p>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart TD
subgraph "Top: How to use the model"
Adapter[LLMAdapter Interface]
Factory[Factory Dispatch]
Template[Prompt Template]
Parser[Response Parser]
Validator[Output Validator]
end
subgraph "Bottom: How to send requests"
Client[llm.Client]
OpenRouter[OpenRouter HTTP]
Ollama[Ollama HTTP]
end
Adapter --> Factory
Factory --> Client
Client --> OpenRouter
Client --> Ollama
</div><h2 id="architecture-naturally-emerges">Architecture Naturally Emerges</h2>
<h3 id="bottom-layer-http-direct-connect">Bottom Layer: HTTP Direct-Connect</h3>
<p><code>internal/llm/client.go</code>'s <code>Client</code> is the most primitive call layer — direct HTTP requests, no abstraction:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// internal/llm/client.go:89</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">func</span><span style="color: #9399B2;"> (</span><span style="color: #EBA0AC;font-style: italic;">c </span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">Client</span><span style="color: #9399B2;">)</span><span style="color: #89B4FA;font-style: italic;"> Generate</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> prompt</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">) (</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> prompt</span><span style="color: #94E2D5;"> ==</span><span style="color: #A6E3A1;"> ""</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> ""</span><span style="color: #9399B2;">,</span><span> ErrEmptyPrompt</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span style="color: #89B4FA;font-style: italic;"> len</span><span style="color: #9399B2;">(</span><span>prompt</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> ></span><span> MaxPromptLength</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> ""</span><span style="color: #9399B2;">,</span><span> ErrPromptTooLong</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> switch</span><span> c</span><span style="color: #9399B2;">.</span><span>config</span><span style="color: #9399B2;">.</span><span>Provider</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> case</span><span style="color: #A6E3A1;"> "openrouter"</span><span style="color: #9399B2;">:</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span> c</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">generateOpenRouter</span><span style="color: #9399B2;">(</span><span>ctx</span><span style="color: #9399B2;">,</span><span> prompt</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> case</span><span style="color: #A6E3A1;"> "ollama"</span><span style="color: #9399B2;">:</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span> c</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">generateOllama</span><span style="color: #9399B2;">(</span><span>ctx</span><span style="color: #9399B2;">,</span><span> prompt</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> default</span><span style="color: #9399B2;">:</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #A6E3A1;"> ""</span><span style="color: #9399B2;">,</span><span> ErrUnsupportedProvider</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The value isn't "supports OpenRouter and Ollama" — it's <strong>converging all provider HTTP details into a single <code>Generate</code> method</strong>. Input validation happens upfront — empty and over-length prompts are intercepted before dispatch.</p>
<h3 id="top-layer-llmadapter-interface">Top Layer: LLMAdapter Interface</h3>
<p>The bottom solves "how to send requests," but agents need more — structured output, prompt templates, response validation. These shouldn't live in the bottom Client (that would bloat it) or in agents (that would couple agents to provider details).</p>
<p>The answer is a new abstraction: <code>LLMAdapter</code>.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// internal/llm/output/adapter.go:10</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> LLMAdapter</span><span style="color: #CBA6F7;"> interface</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Generate</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> prompt</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">) (</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> GenerateStructured</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> prompt</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> schema</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">) (</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">RecommendResult</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> GetModel</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> string</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>What does <code>LLMAdapter</code> add over the bottom <code>Client</code>?</p>
<ul>
<li><strong><code>GenerateStructured</code></strong>: Not just text — structured results conforming to JSON Schema. This is key for agent capabilities: agents need to extract structured task plans from LLM output, not receive free-form text.</li>
<li><strong><code>GetModel()</code></strong>: Lets upper layers know which model is in use.</li>
</ul>
<h3 id="factory-the-provider-dispatch-point">Factory: The Provider Dispatch Point</h3>
<p>With the interface defined, you need a way to create the right adapter at runtime based on configuration. The Factory pattern emerges naturally:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// internal/llm/output/factory.go:17</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> Factory</span><span style="color: #CBA6F7;"> struct</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> adapters</span><span style="color: #CBA6F7;"> map</span><span style="color: #9399B2;">[</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">]</span><span style="color: #CBA6F7;">func</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">Config</span><span style="color: #9399B2;">)</span><span style="color: #F9E2AF;font-style: italic;"> LLMAdapter</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">func</span><span style="color: #89B4FA;font-style: italic;"> NewFactory</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">Factory</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> f</span><span style="color: #94E2D5;"> := &</span><span style="color: #F9E2AF;font-style: italic;">Factory</span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span> adapters</span><span style="color: #9399B2;">:</span><span style="color: #89B4FA;font-style: italic;"> make</span><span style="color: #9399B2;">(</span><span style="color: #CBA6F7;">map</span><span style="color: #9399B2;">[</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">]</span><span style="color: #CBA6F7;">func</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">Config</span><span style="color: #9399B2;">)</span><span style="color: #F9E2AF;font-style: italic;"> LLMAdapter</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span> f</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">register</span><span style="color: #9399B2;">(</span><span>ProviderOpenAI</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> func</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">cfg</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">Config</span><span style="color: #9399B2;">)</span><span style="color: #F9E2AF;font-style: italic;"> LLMAdapter</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #89B4FA;font-style: italic;"> NewOpenAIAdapter</span><span style="color: #9399B2;">(</span><span>cfg</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span> f</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">register</span><span style="color: #9399B2;">(</span><span>ProviderOllama</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> func</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">cfg</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">Config</span><span style="color: #9399B2;">)</span><span style="color: #F9E2AF;font-style: italic;"> LLMAdapter</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #89B4FA;font-style: italic;"> NewOllamaAdapter</span><span style="color: #9399B2;">(</span><span>cfg</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span> f</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">register</span><span style="color: #9399B2;">(</span><span>ProviderOpenRouter</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> func</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">cfg</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">Config</span><span style="color: #9399B2;">)</span><span style="color: #F9E2AF;font-style: italic;"> LLMAdapter</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #89B4FA;font-style: italic;"> NewOpenRouterAdapter</span><span style="color: #9399B2;">(</span><span>cfg</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span> f</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Registry pattern: <code>adapters</code> maps <code>provider -> constructor</code>. Instantiation happens on <code>Create</code>, supporting runtime registration of new providers.</p>
<h3 id="structured-output-from-free-text-to-structured-results">Structured Output: From Free Text to Structured Results</h3>
<p><code>GenerateStructured</code>'s workflow illustrates the top-layer adapter's value:</p>
<ol>
<li>Inject JSON Schema into prompt, telling the model "return in this format."</li>
<li>Call <code>Generate</code> for raw response.</li>
<li>Extract JSON from response.</li>
<li>Validate against Schema.</li>
</ol>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
sequenceDiagram
participant Agent
participant Adapter as LLMAdapter
participant Template as Prompt Template
participant LLM as LLM Provider
participant Parser as Response Parser
Agent->>Adapter: GenerateStructured(ctx, prompt, schema)
Adapter->>Template: Inject schema into prompt
Template-->>Adapter: Complete prompt
Adapter->>LLM: Generate(ctx, fullPrompt)
LLM-->>Adapter: Raw response
Adapter->>Parser: Extract + validate JSON
Parser-->>Adapter: RecommendResult
Adapter-->>Agent: (*RecommendResult, error)
</div><h2 id="three-adapters-compared">Three Adapters Compared</h2>
<table><thead><tr><th>Feature</th><th>OpenAI</th><th>Ollama</th><th>OpenRouter</th></tr></thead><tbody>
<tr><td>API format</td><td>Chat Completions</td><td>Generate</td><td>Chat Completions</td></tr>
<tr><td>Auth</td><td>Bearer Token</td><td>None (local)</td><td>Bearer Token</td></tr>
<tr><td>Structured output</td><td>Native function calling</td><td>Prompt injection</td><td>OpenAI-compatible</td></tr>
</tbody></table>
<p>Adding a new provider only requires implementing <code>LLMAdapter</code> and registering via <code>RegisterProvider</code> — no changes to agents, factory, or config parsing.</p>
<h2 id="design-trade-offs">Design Trade-offs</h2>
<h3 id="trade-off-1-are-two-layers-redundant">Trade-off 1: Are Two Layers Redundant</h3>
<p>Bottom Client sends HTTP; top adapter also sends HTTP. Looks redundant, but different responsibilities: bottom is a minimal-dependency communication layer for internal modules; top is an agent-facing capability layer with templates, parsing, and validation.</p>
<h3 id="trade-off-2-switch-vs-registry">Trade-off 2: Switch vs Registry</h3>
<p>Bottom uses switch dispatch (2-3 providers, clear and direct); top uses registry (extensible, supports runtime registration). Different layers, different trade-offs.</p>
<h3 id="trade-off-3-no-native-function-calling">Trade-off 3: No Native Function Calling</h3>
<p>Structured output via prompt injection, not provider-native capabilities. This trades off native optimization for cross-provider compatibility — Ollama has no function calling, but prompt injection works everywhere.</p>
<h2 id="summary">Summary</h2>
<p>LLM Client is GoAgent's model boundary. The two-layer architecture has each layer doing its job: bottom converges HTTP details, top unifies interfaces and structured output. The Factory pattern minimizes new provider onboarding cost. Agent code depends only on the <code>LLMAdapter</code> interface, never caring which provider is underneath — this is the engineering realization of "not tied to one provider."</p>
GoAgent Source Deep Dive 03: Agent System — The Leader/Sub-Agent Collaboration Skeleton
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/03-agent-system/
<h1 id="goagent-source-deep-dive-03-agent-system-the-leader-sub-agent-collaboration-skeleton">GoAgent Source Deep Dive 03: Agent System — The Leader/Sub-Agent Collaboration Skeleton</h1>
<h2 id="the-problem-one-agent-can-t-handle-everything">The Problem: One Agent Can't Handle Everything</h2>
<p>You've built an agent with an LLM that understands user input, calls tools, and returns results. But you quickly discover — when a user says "plan a trip to Tokyo, budget 5000, and recommend some restaurants," a single agent trying to do itinerary planning, budget calculation, restaurant recommendation, and information retrieval all at once either becomes impossibly complex or does each task poorly.</p>
<p>You need to split tasks across different agents. But who splits them? Who coordinates the results?</p>
<h2 id="limitations-of-existing-approaches">Limitations of Existing Approaches</h2>
<p><strong>Approach A: One giant agent with if-else</strong></p>
<p>A monolithic agent with branching logic. Problems: hard to test, tasks run serially, adding capabilities requires modifying core agent code.</p>
<p><strong>Approach B: Let the LLM decide everything via function calling</strong></p>
<p>Let the LLM dynamically choose tools. Problems: LLM decisions are unpredictable, no task decomposition, result aggregation depends entirely on the LLM with no engineering guarantees.</p>
<p>Both approaches lack an <strong>orchestration layer</strong> — a component responsible for understanding input, decomposing tasks, dispatching execution, and aggregating results.</p>
<h2 id="goagent-s-approach">GoAgent's Approach</h2>
<p>The Leader/Sub-agent pattern:</p>
<ul>
<li><strong>Leader Agent</strong> is the orchestrator, not the executor. It understands input, plans tasks, dispatches to Sub Agents, aggregates results.</li>
<li><strong>Sub Agent</strong> is the execution unit, not the decision maker. It receives a concrete task, calls LLM and tools to complete it.</li>
</ul>
<p>This division isn't a "designed architecture" — it naturally emerges from the task decomposition requirement: decomposition implies separation of planner and executor.</p>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart TD
Input[User Input] --> Leader[Leader Agent<br/>Orchestrator]
Leader --> Parse[Parse User Profile]
Parse --> Plan[Plan Task List]
Plan --> Dispatch[Parallel Dispatch]
Dispatch --> SubA[Sub Agent A<br/>Itinerary]
Dispatch --> SubB[Sub Agent B<br/>Restaurants]
Dispatch --> SubC[Sub Agent C<br/>Budget]
SubA --> Results[TaskResults]
SubB --> Results
SubC --> Results
Results --> Aggregate[Aggregate]
Aggregate --> Output[Final Output]
</div><h2 id="architecture-naturally-emerges">Architecture Naturally Emerges</h2>
<h3 id="interface-hierarchy">Interface Hierarchy</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// base.Agent — foundation for all agents</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> Agent</span><span style="color: #CBA6F7;"> interface</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> ID</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> string</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Type</span><span style="color: #9399B2;">()</span><span style="color: #F9E2AF;font-style: italic;"> models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">AgentType</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Status</span><span style="color: #9399B2;">()</span><span style="color: #F9E2AF;font-style: italic;"> models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">AgentStatus</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Start</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> error</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Stop</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> error</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Process</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> input</span><span style="color: #F9E2AF;font-style: italic;"> any</span><span style="color: #9399B2;">) (</span><span style="color: #F9E2AF;font-style: italic;">any</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;">// sub.Agent — adds Execute</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> Agent</span><span style="color: #CBA6F7;"> interface</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> base</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Agent</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Execute</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> task</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Task</span><span style="color: #9399B2;">) (</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">TaskResult</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre><h3 id="leader-s-four-replaceable-components">Leader's Four Replaceable Components</h3>
<p>Leader isn't an "all-powerful LLM object" — it's an orchestrator depending on four strategy interfaces:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> ProfileParser</span><span style="color: #CBA6F7;"> interface</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Parse</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> input</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">) (</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">UserProfile</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> TaskPlanner</span><span style="color: #CBA6F7;"> interface</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Plan</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> profile</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">UserProfile</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> inputText</span><span style="color: #CBA6F7;"> string</span><span style="color: #9399B2;">) ([]</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Task</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> TaskDispatcher</span><span style="color: #CBA6F7;"> interface</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Dispatch</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> tasks</span><span style="color: #9399B2;"> []</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Task</span><span style="color: #9399B2;">) ([]</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">TaskResult</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> RegisterExecutor</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">agentType</span><span style="color: #F9E2AF;font-style: italic;"> models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">AgentType</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> fn</span><span style="color: #F9E2AF;font-style: italic;"> TaskExecutorFunc</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> ResultAggregator</span><span style="color: #CBA6F7;"> interface</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Aggregate</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> results</span><span style="color: #9399B2;"> []</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">TaskResult</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> tasks</span><span style="color: #9399B2;"> []</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Task</span><span style="color: #9399B2;">) (</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">RecommendResult</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre><h3 id="dispatcher-the-concurrency-core">Dispatcher: The Concurrency Core</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #CBA6F7;">func</span><span style="color: #9399B2;"> (</span><span style="color: #EBA0AC;font-style: italic;">d </span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">taskDispatcher</span><span style="color: #9399B2;">)</span><span style="color: #89B4FA;font-style: italic;"> Dispatch</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> tasks</span><span style="color: #9399B2;"> []</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Task</span><span style="color: #9399B2;">) ([]</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">TaskResult</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span> g</span><span style="color: #9399B2;">,</span><span> ctx</span><span style="color: #94E2D5;"> :=</span><span> errgroup</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">WithContext</span><span style="color: #9399B2;">(</span><span>ctx</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span> sem</span><span style="color: #94E2D5;"> :=</span><span style="color: #89B4FA;font-style: italic;"> make</span><span style="color: #9399B2;">(</span><span style="color: #CBA6F7;">chan struct</span><span style="color: #9399B2;">{},</span><span> d</span><span style="color: #9399B2;">.</span><span>maxParallel</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> var</span><span> resultsMu</span><span style="color: #F9E2AF;font-style: italic;"> sync</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Mutex</span></span>
<span class="giallo-l"><span> results</span><span style="color: #94E2D5;"> :=</span><span style="color: #89B4FA;font-style: italic;"> make</span><span style="color: #9399B2;">([]</span><span style="color: #94E2D5;">*</span><span style="color: #F9E2AF;font-style: italic;">models</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">TaskResult</span><span style="color: #9399B2;">,</span><span style="color: #89B4FA;font-style: italic;"> len</span><span style="color: #9399B2;">(</span><span>tasks</span><span style="color: #9399B2;">))</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> i</span><span style="color: #9399B2;">,</span><span> task</span><span style="color: #94E2D5;"> :=</span><span style="color: #CBA6F7;"> range</span><span> tasks</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> task</span><span style="color: #94E2D5;"> :=</span><span> task</span></span>
<span class="giallo-l"><span> g</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">Go</span><span style="color: #9399B2;">(</span><span style="color: #CBA6F7;">func</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> select</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> case</span><span> sem</span><span style="color: #94E2D5;"> <-</span><span style="color: #CBA6F7;"> struct</span><span style="color: #9399B2;">{}{}:</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> case</span><span style="color: #94E2D5;"> <-</span><span>ctx</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">Done</span><span style="color: #9399B2;">():</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span> ctx</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">Err</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> defer func</span><span style="color: #9399B2;">() {</span><span style="color: #94E2D5;"> <-</span><span>sem</span><span style="color: #9399B2;"> }()</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> execResult</span><span style="color: #94E2D5;"> :=</span><span> d</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">executeTask</span><span style="color: #9399B2;">(</span><span>ctx</span><span style="color: #9399B2;">,</span><span> task</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span> resultsMu</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">Lock</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span> results</span><span style="color: #9399B2;">[</span><span>i</span><span style="color: #9399B2;">]</span><span style="color: #94E2D5;"> =</span><span> execResult</span></span>
<span class="giallo-l"><span> resultsMu</span><span style="color: #9399B2;">.</span><span style="color: #89B4FA;font-style: italic;">Unlock</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #F38BA8;"> nil</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ...</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Three concurrency decisions: errgroup for cancellation propagation, semaphore for bounded parallelism, mutex for safe result collection.</p>
<h3 id="task-planning-pragmatics">Task Planning Pragmatics</h3>
<p>The planner matches trigger keywords to sub-agents. If no match and fallback is enabled, all sub-agents are dispatched. <strong>Better to over-dispatch than leave input with no response.</strong></p>
<h2 id="design-trade-offs">Design Trade-offs</h2>
<ul>
<li><strong>Strategy pattern vs inheritance</strong>: Composition + interfaces, replaceable at runtime.</li>
<li><strong>errgroup vs worker pool</strong>: errgroup has built-in context cancellation; semaphore limits concurrency on-demand.</li>
<li><strong>Local execution vs message queue</strong>: Both supported, auto-selected based on whether <code>executorFuncs</code> or <code>messageSender</code> exists.</li>
</ul>
<h2 id="summary">Summary</h2>
<p>The Agent system isn't "mysterious intelligent agents" — it's an engineering-driven task orchestration pipeline. Leader plans and orchestrates, Sub Agent executes, Dispatcher handles concurrency. Architecture wasn't pre-designed — it naturally emerged from the task decomposition requirement.</p>
GoAgent Source Deep Dive 04: AHP Protocol — Messages, Queues, and Heartbeats
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/04-ahp-protocol/
<h1 id="goagent-source-deep-dive-04-ahp-protocol-messages-queues-and-heartbeats">GoAgent Source Deep Dive 04: AHP Protocol — Messages, Queues, and Heartbeats</h1>
<h2 id="the-problem-how-do-agents-communicate">The Problem: How Do Agents Communicate</h2>
<p>Leader dispatches tasks to Sub Agents, Sub Agents return results — sounds like function calls. But when you want to run agents across processes or machines, function calls become network calls.</p>
<h2 id="limitations-of-existing-approaches">Limitations of Existing Approaches</h2>
<p><strong>Approach A: Direct function calls</strong> — Ties agents to the same process, no buffering, no retry/timeout.</p>
<p><strong>Approach B: Direct gRPC/HTTP</strong> — Every agent defines its own API routes, no unified message format, no protocol semantics, no heartbeat.</p>
<p>Both lack a <strong>protocol layer</strong>: unified message format, reliable queue mechanism, health monitoring.</p>
<h2 id="goagent-s-approach">GoAgent's Approach</h2>
<p>AHP (Agent Handshake Protocol) solves three problems:</p>
<ol>
<li><strong>Message format</strong>: Unified <code>AHPMessage</code> structure with method type, sender, receiver, task ID, payload.</li>
<li><strong>Message queue</strong>: Channel-based in-memory queue with enqueue, dequeue, peek.</li>
<li><strong>Heartbeat</strong>: Periodic heartbeat signals so Leader can detect Sub Agent liveness.</li>
</ol>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart LR
Leader[Leader Agent] -->|TASK| Queue[MessageQueue]
Queue -->|TASK| Sub[Sub Agent]
Sub -->|RESULT| Queue
Queue -->|RESULT| Leader
Sub -->|HEARTBEAT| HB[HeartbeatMonitor]
HB -->|IsAlive?| Leader
</div><h2 id="architecture-naturally-emerges">Architecture Naturally Emerges</h2>
<h3 id="five-message-types">Five Message Types</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #CBA6F7;">const</span><span style="color: #9399B2;"> (</span></span>
<span class="giallo-l"><span style="color: #FAB387;"> AHPMethodTask</span><span style="color: #F9E2AF;font-style: italic;"> AHPMethod</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "TASK"</span><span style="color: #9399B2;font-style: italic;"> // Leader dispatches task</span></span>
<span class="giallo-l"><span style="color: #FAB387;"> AHPMethodResult</span><span style="color: #F9E2AF;font-style: italic;"> AHPMethod</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "RESULT"</span><span style="color: #9399B2;font-style: italic;"> // Sub Agent returns result</span></span>
<span class="giallo-l"><span style="color: #FAB387;"> AHPMethodProgress</span><span style="color: #F9E2AF;font-style: italic;"> AHPMethod</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "PROGRESS"</span><span style="color: #9399B2;font-style: italic;"> // Sub Agent reports progress</span></span>
<span class="giallo-l"><span style="color: #FAB387;"> AHPMethodACK</span><span style="color: #F9E2AF;font-style: italic;"> AHPMethod</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "ACK"</span><span style="color: #9399B2;font-style: italic;"> // Acknowledgment</span></span>
<span class="giallo-l"><span style="color: #FAB387;"> AHPMethodHeartbeat</span><span style="color: #F9E2AF;font-style: italic;"> AHPMethod</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "HEARTBEAT"</span><span style="color: #9399B2;font-style: italic;"> // Heartbeat signal</span></span>
<span class="giallo-l"><span style="color: #9399B2;">)</span></span></code></pre>
<p>Message ID: <code>timestamp.atomic_counter.random_suffix</code> — three dimensions ensure uniqueness.</p>
<h3 id="messagequeue-backup-buffer-prevents-lost-messages">MessageQueue: Backup Buffer Prevents Lost Messages</h3>
<p>The key design: <strong>Peek never loses messages</strong>. When Peek takes a message out to inspect, if it can't put it back (queue full), the message goes into a backup buffer. Next Dequeue consumes backup first.</p>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart TD
Peek[Peek] --> Check{backup has data?}
Check -->|Yes| ReturnB[Return backup[0]]
Check -->|No| Receive{Receive from channel}
Receive -->|Got msg| PutBack{Try put back}
PutBack -->|OK| Return[Return msg]
PutBack -->|Full| Save[Save to backup]
Save --> Return
Receive -->|Empty| Nil[Return nil]
</div>
<p>Enqueue: <code>select</code> with three paths — success, context cancelled, queue full (never blocks).
Dequeue: prioritizes backup buffer, then main channel.</p>
<h3 id="heartbeat">Heartbeat</h3>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
sequenceDiagram
participant Sub as Sub Agent
participant Monitor as HeartbeatMonitor
participant Leader as Leader Agent
loop Every HeartbeatInterval
Sub->>Monitor: Heartbeat(agentID)
Monitor->>Monitor: Update lastSeen
end
Leader->>Monitor: IsAlive(agentID)
Monitor-->>Leader: true / false
</div><h2 id="design-trade-offs">Design Trade-offs</h2>
<ul>
<li><strong>In-memory queue vs external middleware</strong>: Zero dependencies, low latency. No persistence — sufficient for single-process, extend MessageSender for distributed.</li>
<li><strong>Backup buffer vs unbounded queue</strong>: Bounded channel + backup buffer controls memory while ensuring Peek doesn't lose messages.</li>
<li><strong>JSON vs Protobuf</strong>: Readable, debuggable. Agent message volume is modest, JSON overhead acceptable.</li>
</ul>
<h2 id="summary">Summary</h2>
<p>AHP moves agent collaboration from "local function calls" to "message semantics." It's not distributed middleware — it's the protocol skeleton for inter-agent communication, naturally derived from the question "how do agents talk to each other."</p>
GoAgent Source Deep Dive 05: Memory System — Sessions, Tasks, and Distilled Memories
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/05-memory-system/
<h1 id="goagent-source-deep-dive-05-memory-system-sessions-tasks-and-distilled-memories">GoAgent Source Deep Dive 05: Memory System — Sessions, Tasks, and Distilled Memories</h1>
<h2 id="the-problem-agents-start-from-scratch-every-conversation">The Problem: Agents Start From Scratch Every Conversation</h2>
<p>You've built an agent with LLM calls. A user asks "recommend a Go book," and the agent recommends "The Go Programming Language." The user follows up with "is there a Chinese edition?" — the agent has no idea what "Chinese edition" refers to.</p>
<p>LLM calls are stateless. Each request is independent. This means: no conversation context, no task tracking, no experience reuse.</p>
<h2 id="limitations-of-existing-approaches">Limitations of Existing Approaches</h2>
<p><strong>Approach A: Stuff all history into the prompt</strong> — Token cost grows linearly, exceeds context window, doesn't extract reusable knowledge.</p>
<p><strong>Approach B: Simple key-value cache</strong> — Semantic matches miss, no classification, noise (code blocks, stack traces) gets stored too.</p>
<p>Both lack a <strong>tiered memory architecture</strong>: information at different lifecycles needs different storage and retrieval strategies.</p>
<h2 id="goagent-s-approach">GoAgent's Approach</h2>
<p>Three memory tiers for three time scales:</p>
<table><thead><tr><th>Tier</th><th>Lifecycle</th><th>What</th><th>How Used</th></tr></thead><tbody>
<tr><td>Session</td><td>Current conversation</td><td>Multi-turn message history</td><td>Build context</td></tr>
<tr><td>Task</td><td>Current task</td><td>Task input/output</td><td>Track execution</td></tr>
<tr><td>Distilled</td><td>Long-term</td><td>Extracted experience</td><td>Retrieve for similar tasks</td></tr>
</tbody></table>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart TD
Input[User Input] --> Session[Session Memory<br/>Short-term]
Input --> Task[Task Memory<br/>Mid-term]
Task --> Distill[Distilled Memory<br/>Long-term]
Session --> BuildCtx[BuildContext]
Distill --> Search[SearchSimilarTasks]
Search --> BuildCtx
BuildCtx --> Agent[Agent Execution]
</div><h2 id="architecture-naturally-emerges">Architecture Naturally Emerges</h2>
<h3 id="session-memory">Session Memory</h3>
<p><code>BuildContext</code> combines current input with message history. Key designs: TTL cleanup, max history limit, fault tolerance (returns original input on error).</p>
<h3 id="distilled-memory-pipeline">Distilled Memory Pipeline</h3>
<p>Not simple caching — a full pipeline:</p>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart TD
Conv[Conversation] --> Extract[ExperienceExtractor]
Extract --> Classify[MemoryClassifier]
Classify --> Score[ImportanceScorer]
Score --> Filter[NoiseFilter]
Filter --> TopN[Top-N Selection]
TopN --> Conflict[ConflictResolver]
Conflict --> Store[(Repository)]
</div>
<p><strong>Classifier</strong>: Knowledge, Preference, Interaction, Profile.
<strong>Noise filter</strong>: Code blocks, stack traces, logs, markdown tables.
<strong>Conflict resolver</strong>: Vector similarity detection, keep the more important memory.</p>
<h2 id="two-implementations">Two Implementations</h2>
<table><thead><tr><th>Feature</th><th>Old (Hash)</th><th>New (Distiller)</th></tr></thead><tbody>
<tr><td>Vectors</td><td>Hash function</td><td>Embedding Service</td></tr>
<tr><td>Storage</td><td>Memory</td><td>PostgreSQL + pgvector</td></tr>
<tr><td>Retrieval</td><td>Cosine similarity</td><td>Vector + BM25</td></tr>
<tr><td>Capacity</td><td>5000 (LRU)</td><td>Unlimited</td></tr>
<tr><td>Quality</td><td>Basic</td><td>Full pipeline</td></tr>
</tbody></table>
<p>Switchable via <code>useNewDistill</code> flag; old method as fallback.</p>
<h2 id="summary">Summary</h2>
<p>Three tiers naturally emerge from three time-scale needs: session for context, task for tracking, distilled for experience. The distillation pipeline's complexity reflects real "experience management" needs — not archiving logs, but extracting, classifying, scoring, filtering, and resolving conflicts.</p>
GoAgent Source Deep Dive 06: Tool System — Registration, Capability Matching, and Execution
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/06-tool-system/
<h1 id="goagent-source-deep-dive-06-tool-system-registration-capability-matching-and-execution">GoAgent Source Deep Dive 06: Tool System — Registration, Capability Matching, and Execution</h1>
<h2 id="the-problem-how-do-agents-do-things-instead-of-just-talking">The Problem: How Do Agents "Do Things" Instead of Just "Talking"</h2>
<p>LLMs excel at understanding and generating text, but when a user says "calculate a 15% tip" or "check today's weather," the agent needs to call external capabilities. The question: how does an agent discover, select, and call these capabilities?</p>
<h2 id="limitations-of-existing-approaches">Limitations of Existing Approaches</h2>
<p><strong>Approach A: Hardcode tool logic in prompts</strong> — Tool list changes require prompt changes, LLM selection is unpredictable, no parameter validation.</p>
<p><strong>Approach B: Direct function calls inside agents</strong> — Agent couples with tool implementations, adding tools requires modifying agent code, no tool discovery mechanism.</p>
<p>Both lack a <strong>unified tool abstraction layer</strong>: tools should be registerable, discoverable, and callable without coupling to agent code.</p>
<h2 id="goagent-s-approach">GoAgent's Approach</h2>
<p>Three layers:</p>
<ol>
<li><strong>Unified interface</strong>: All tools implement the same <code>Tool</code> interface.</li>
<li><strong>Registry</strong>: Tools register via <code>Registry</code>, accessed by name.</li>
<li><strong>Capability matching</strong>: <code>CapabilityEngine</code> detects needed capabilities from user input, automatically filters relevant tools.</li>
</ol>
<h2 id="architecture-naturally-emerges">Architecture Naturally Emerges</h2>
<h3 id="tool-interface">Tool Interface</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> Tool</span><span style="color: #CBA6F7;"> interface</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Name</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> string</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Description</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> string</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Category</span><span style="color: #9399B2;">()</span><span style="color: #F9E2AF;font-style: italic;"> ToolCategory</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Capabilities</span><span style="color: #9399B2;">() []</span><span style="color: #F9E2AF;font-style: italic;">Capability</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Execute</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;font-style: italic;">ctx</span><span style="color: #F9E2AF;font-style: italic;"> context</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Context</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;font-style: italic;"> params</span><span style="color: #CBA6F7;"> map</span><span style="color: #9399B2;">[</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">]</span><span style="color: #CBA6F7;">interface</span><span style="color: #9399B2;">{}) (</span><span style="color: #F9E2AF;font-style: italic;">Result</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> error</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> Parameters</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">ParameterSchema</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>6 categories, 8 capabilities. Every tool provides name, description, category, capabilities, parameter schema, and execute method.</p>
<h3 id="registry">Registry</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> Registry</span><span style="color: #CBA6F7;"> struct</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> mu</span><span style="color: #F9E2AF;font-style: italic;"> sync</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">RWMutex</span></span>
<span class="giallo-l"><span> tools</span><span style="color: #CBA6F7;"> map</span><span style="color: #9399B2;">[</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">]</span><span style="color: #F9E2AF;font-style: italic;">Tool</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Prevents nil tools and duplicate names on registration; retrieves by name on execution.</p>
<h3 id="capability-engine-keyword-routing">Capability Engine: Keyword Routing</h3>
<p>Not LLM reasoning — keyword matching:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #CBA6F7;">var</span><span> capabilityKeywords</span><span style="color: #94E2D5;"> =</span><span style="color: #CBA6F7;"> map</span><span style="color: #9399B2;">[</span><span style="color: #F9E2AF;font-style: italic;">Capability</span><span style="color: #9399B2;">][]</span><span style="color: #CBA6F7;">string</span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span> CapabilityMath</span><span style="color: #9399B2;">: {</span><span style="color: #A6E3A1;">"calculate"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "sum"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "compute"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "aggregate"</span><span style="color: #9399B2;">},</span></span>
<span class="giallo-l"><span> CapabilityNetwork</span><span style="color: #9399B2;">: {</span><span style="color: #A6E3A1;">"api"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "request"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "fetch"</span><span style="color: #9399B2;">,</span><span style="color: #A6E3A1;"> "download"</span><span style="color: #9399B2;">},</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // ...</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Flow: user input → keyword detection → matched capabilities → filter tools from registry → execute. Bilingual (English + Chinese), multi-capability matching, simple and predictable.</p>
<h2 id="design-trade-offs">Design Trade-offs</h2>
<ul>
<li><strong>Keywords vs LLM reasoning</strong>: Fast, predictable, zero cost. Complex scenarios left to LLM tool selection in prompts.</li>
<li><strong>Global registry vs per-agent</strong>: Simpler; AgentTools filters per agent on top of global registry.</li>
<li><strong>map vs generics</strong>: Simple interface, sacrifices type safety. Practical for tools with wildly varying parameter types.</li>
</ul>
<h2 id="summary">Summary</h2>
<p>The Tool system is the agent's "action layer." Unified interface makes tools pluggable, registry manages lifecycle, capability engine auto-routes. Architecture naturally emerged from "how do agents do things": first the interface (decoupling), then the registry (management), then capability matching (discovery).</p>
GoAgent Source Deep Dive 07: Workflow Engine — DAG-Based Agent Orchestration
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/07-workflow-engine/
<h1 id="goagent-source-deep-dive-07-workflow-engine-dag-based-agent-orchestration">GoAgent Source Deep Dive 07: Workflow Engine — DAG-Based Agent Orchestration</h1>
<h2 id="the-problem-how-do-multi-step-tasks-execute-by-dependency">The Problem: How Do Multi-Step Tasks Execute by Dependency</h2>
<p>The Leader/Sub-agent pattern excels at "user input → real-time decision → execute → return." But some tasks are multi-step: step A's output is step B's input, step C must wait for both A and B.</p>
<h2 id="limitations-of-existing-approaches">Limitations of Existing Approaches</h2>
<p><strong>Approach A: Hardcoded execution order</strong> — Not reusable, no retry/timeout/concurrency control.</p>
<p><strong>Approach B: Dynamic planning via Agent system</strong> — Same workflow re-inferred every time, LLM planning is unpredictable, no declarative definition.</p>
<p>Both lack <strong>declarable, reusable, controllable process orchestration</strong>.</p>
<h2 id="goagent-s-approach">GoAgent's Approach</h2>
<p>Two workflow engines for different scenarios:</p>
<ul>
<li><strong>YAML DAG engine</strong>: Declare steps and dependencies in YAML, engine builds DAG, topological sort, concurrent execution.</li>
<li><strong>Dynamic graph engine</strong>: Build graphs in Go code, conditional edges, multiple node types, pluggable schedulers.</li>
</ul>
<p>Not duplicate implementations: YAML DAG for "known processes," dynamic graph for "processes needing runtime decisions."</p>
<h2 id="architecture-naturally-emerges">Architecture Naturally Emerges</h2>
<h3 id="yaml-dag-engine">YAML DAG Engine</h3>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart TD
A[collect] --> B[analyze]
A --> C[retrieve]
B --> D[synthesize]
C --> D
</div>
<p>Executor follows topological order, semaphore controls concurrency, supports retry and template variable substitution.</p>
<h3 id="dynamic-graph-engine">Dynamic Graph Engine</h3>
<p>Three node types: AgentNode, ToolNode, FuncNode. Conditional edges for dynamic routing. Three schedulers: FIFO, Priority, Shortest-Job-First.</p>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart TD
Start --> Analyze
Analyze -->|confidence > 0.8| Fast[Fast Path]
Analyze -->|confidence <= 0.8| Deep[Deep Analysis]
Fast --> End
Deep --> End
</div><h2 id="design-trade-offs">Design Trade-offs</h2>
<ul>
<li><strong>Two vs unified</strong>: Each serves its purpose better than a unified complexity.</li>
<li><strong>Panic vs error</strong>: <code>NewGraph</code> panics on empty id — startup programming errors should fail fast.</li>
<li><strong>Shared state vs message passing</strong>: Simpler, requires single-threaded execution (controlled by scheduler).</li>
</ul>
<h2 id="summary">Summary</h2>
<p>The Workflow engine extends agents from "real-time decisions" to "declarable, reusable process orchestration." Architecture naturally emerged from "how do multi-step tasks execute by dependency."</p>
GoAgent Source Deep Dive 08: Storage and Retrieval — PostgreSQL, pgvector, and Hybrid Search
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/08-storage-retrieval/
<h1 id="goagent-source-deep-dive-08-storage-and-retrieval-postgresql-pgvector-and-hybrid-search">GoAgent Source Deep Dive 08: Storage and Retrieval — PostgreSQL, pgvector, and Hybrid Search</h1>
<h2 id="the-problem-where-do-agent-memories-and-knowledge-live">The Problem: Where Do Agent Memories and Knowledge Live</h2>
<p>Agent conversation history, knowledge base, and distilled experience need persistence. In-memory storage is lost on process restart. You need a database — but storage alone isn't enough. You also need efficient retrieval: "how was a similar problem solved last time" requires vector similarity search; "knowledge containing a keyword" requires full-text search.</p>
<h2 id="limitations-of-existing-approaches">Limitations of Existing Approaches</h2>
<p><strong>SQLite + file storage</strong> — No vector retrieval, no concurrent writes, not production-ready.
<strong>Direct PostgreSQL writes</strong> — High-frequency writes perform poorly; no vectorization pipeline.
<strong>Pure vector databases (Pinecone, Weaviate)</strong> — External dependency, no relational queries, can't do structured + semantic retrieval simultaneously.</p>
<h2 id="goagent-s-approach">GoAgent's Approach</h2>
<p>PostgreSQL + pgvector as unified storage, with four capability layers:</p>
<ol>
<li><strong>Connection pool</strong>: Connection management with timeout and reuse.</li>
<li><strong>WriteBuffer</strong>: Batch writes, content-hash dedup, graceful shutdown.</li>
<li><strong>EmbeddingQueue</strong>: Async vectorization, idempotent dedup, dead letter queue.</li>
<li><strong>RetrievalService</strong>: Hybrid search (vector + BM25), query rewriting, time decay, multi-source fusion.</li>
</ol>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart TD
Write[Write Text] --> Buffer[WriteBuffer<br/>Batch Write]
Buffer --> DB[(PostgreSQL<br/>pending)]
DB --> Queue[EmbeddingQueue<br/>Async Vectorize]
Queue --> Embed[Embedding Service]
Embed --> DB2[(PostgreSQL<br/>completed)]
Search[Search Request] --> RS[RetrievalService]
RS --> Vector[Vector Search<br/>pgvector]
RS --> BM25[Keyword Search<br/>BM25]
Vector --> Merge[Fuse & Rank]
BM25 --> Merge
Merge --> Results[SearchResult]
</div><h2 id="architecture-naturally-emerges">Architecture Naturally Emerges</h2>
<h3 id="writebuffer">WriteBuffer</h3>
<p>Buffers high-frequency writes, flushes on batchSize or flushInterval. Content-hash dedup prevents duplicate entries. <code>safeSend</code> uses <code>recover()</code> to handle write-on-closed-channel races.</p>
<h3 id="embeddingqueue">EmbeddingQueue</h3>
<p>Async vectorization pipeline: text → DB (pending) → EmbeddingQueue → Embedding Service → DB (completed). Dedupe key ensures idempotency, dead letter queue handles repeated failures, reconciliation periodically retries pending records.</p>
<h3 id="retrievalservice-hybrid-search">RetrievalService: Hybrid Search</h3>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
sequenceDiagram
participant User
participant RS as RetrievalService
participant PG as PostgreSQL
User->>RS: Search(ctx, request)
RS->>RS: Build RetrievalPlan
par Vector Search
RS->>PG: ORDER BY vector <-> query_vector
and Keyword Search
RS->>PG: WHERE content LIKE '%keyword%'
end
RS->>RS: Weight, merge, rank
RS-->>User: SearchResult list
</div><h2 id="design-trade-offs">Design Trade-offs</h2>
<ul>
<li><strong>WriteBuffer vs direct write</strong>: Batch + timer flush balances throughput and latency.</li>
<li><strong>Hybrid vs pure vector</strong>: Semantic + exact match, but more complexity. RetrievalPlan lets callers choose.</li>
<li><strong>Async vs sync vectorization</strong>: Writes don't wait; retrieval degrades to keyword search if vectors aren't ready.</li>
</ul>
<h2 id="summary">Summary</h2>
<p>Storage and Retrieval isn't "just connect to a database." Connection pool, repository, batch writes, async vectorization, hybrid search, multi-tenant isolation — together they form the agent's knowledge infrastructure. Architecture naturally emerged from "where do memories live and how are they retrieved."</p>
GoAgent Source Deep Dive 09: Embedding Service — The Engineering Boundary of Vector Generation
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/09-embedding-service/
<h1 id="goagent-source-deep-dive-09-embedding-service-the-engineering-boundary-of-vector-generation">GoAgent Source Deep Dive 09: Embedding Service — The Engineering Boundary of Vector Generation</h1>
<h2 id="the-problem-how-do-you-turn-text-into-vectors">The Problem: How Do You Turn Text Into Vectors</h2>
<p>GoAgent's retrieval depends on vector similarity search. PostgreSQL + pgvector can do vector search, but you need vectors first. How does text become a vector?</p>
<h2 id="limitations-of-existing-approaches">Limitations of Existing Approaches</h2>
<p><strong>CGO bindings for vector models in Go</strong> — Complex compilation, cross-platform difficulties, Go's ML ecosystem is immature.
<strong>Python subprocess from Go</strong> — High latency, complex process management.
<strong>External API (OpenAI Embedding)</strong> — External dependency, network latency and cost, not suitable for local deployment.</p>
<p>All three have engineering pain points. What GoAgent needs: <strong>vector generation decoupled from the Go main program, supporting local deployment and flexible backend switching</strong>.</p>
<h2 id="goagent-s-approach">GoAgent's Approach</h2>
<p>Deploy embedding as an independent HTTP service:</p>
<ul>
<li>Python FastAPI implementation, leveraging Python's ML ecosystem.</li>
<li>Go calls via HTTP, no CGO or subprocess dependency.</li>
<li>Supports Ollama and SentenceTransformers backends, switchable via environment variable.</li>
<li>Optional Redis cache, L2 normalization for retrieval consistency.</li>
</ul>
<div class="mermaid-direct" data-series="goagent" data-name="graph.mmd">
flowchart LR
subgraph "Go Main Program"
Retrieval[RetrievalService]
Client[EmbeddingClient]
end
subgraph "Embedding Service"
FastAPI[FastAPI]
Cache[Redis]
end
subgraph "Backends"
Ollama[Ollama]
ST[SentenceTransformers]
end
Retrieval --> Client
Client -->|HTTP| FastAPI
FastAPI --> Cache
FastAPI --> Ollama
FastAPI --> ST
</div><h2 id="architecture-naturally-emerges">Architecture Naturally Emerges</h2>
<h3 id="go-side-embeddingclient">Go Side: EmbeddingClient</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="go"><span class="giallo-l"><span style="color: #CBA6F7;">type</span><span style="color: #F9E2AF;font-style: italic;"> EmbeddingClient</span><span style="color: #CBA6F7;"> struct</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> baseURL</span><span style="color: #CBA6F7;"> string</span></span>
<span class="giallo-l"><span> httpClient</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;font-style: italic;">http</span><span style="color: #9399B2;">.</span><span style="color: #F9E2AF;font-style: italic;">Client</span></span>
<span class="giallo-l"><span> retries</span><span style="color: #CBA6F7;"> int</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Two core methods: <code>Embed</code> (single text) and <code>EmbedBatch</code> (batch). Constructs JSON, POSTs to <code>/embed</code> or <code>/embed_batch</code>, parses response.</p>
<h3 id="python-side-fastapi-service">Python Side: FastAPI Service</h3>
<p>Three endpoints: <code>/embed</code>, <code>/embed_batch</code>, <code>/health</code>.</p>
<p>Single text flow: check cache → generate vector → L2 normalize → write cache → return.
Batch flow: filter cached → batch generate uncached → normalize and cache each → return.</p>
<h3 id="backend-switching">Backend Switching</h3>
<p><code>EMBEDDING_BACKEND</code> environment variable switches Ollama or SentenceTransformers. No code changes. Dev uses sentence-transformers (zero config), production uses Ollama (high performance).</p>
<h3 id="vector-normalization">Vector Normalization</h3>
<p>All vectors L2-normalized before return (norm = 1). After normalization, cosine similarity equals dot product — faster. Without normalization, long text vectors may have larger magnitudes, causing retrieval bias.</p>
<h3 id="cache">Cache</h3>
<p>Redis optional: same text doesn't recompute, reduces backend pressure. Service works without Redis, just uncached.</p>
<h2 id="design-trade-offs">Design Trade-offs</h2>
<ul>
<li><strong>Independent service vs embedded</strong>: Extra network hop and deployment component, but language decoupling, independent scaling, flexible backend.</li>
<li><strong>HTTP vs gRPC</strong>: Easy debugging, language-agnostic. For request-response patterns, HTTP overhead acceptable.</li>
<li><strong>Optional vs mandatory cache</strong>: Zero config for dev, on-demand for production. Service availability independent of Redis.</li>
</ul>
<h2 id="summary">Summary</h2>
<p>The Embedding Service is the prerequisite for GoAgent's retrieval. Deploying independently is the trade-off between engineering complexity and operational flexibility. Architecture naturally emerged from "how does text become vectors" — Python ecosystem for vector generation, Go for business logic, HTTP as the bridge.</p>
GoAgent Source Deep Dive Series
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/goagent/readme/
<h1 id="goagent-source-deep-dive-series">GoAgent Source Deep Dive Series</h1>
<p>This directory contains the English drafts for the GoAgent blog series. All diagrams use Mermaid for blog-friendly publishing.</p>
<h2 id="narrative-thread">Narrative Thread</h2>
<p>Every article follows the same thread: <strong>Problem → Limitations of existing approaches → GoAgent's approach → Architecture naturally emerges</strong>.</p>
<ol>
<li><strong>Problem</strong>: Start with a concrete pain point — readers immediately understand why the module exists.</li>
<li><strong>Limitations</strong>: List 1-2 common approaches and explain why they fall short.</li>
<li><strong>GoAgent's approach</strong>: Present the solution as a natural response to the problem, not a pre-designed architecture.</li>
<li><strong>Architecture naturally emerges</strong>: From the approach, the architecture follows as a natural derivation.</li>
</ol>
<h2 id="reading-order">Reading Order</h2>
<ol>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/readme/01-api-client.md">API Client: Converging Framework Capabilities Into a Usable Entry Point</a> — How should internal modules be called externally</li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/readme/02-llm-client.md">LLM Client: Unifying Multi-Model Call Boundaries</a> — How to keep agents from being tied to one provider</li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/readme/03-agent-system.md">Agent System: The Leader/Sub-Agent Collaboration Skeleton</a> — One agent can't handle everything</li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/readme/04-ahp-protocol.md">AHP Protocol: Messages, Queues, and Heartbeats</a> — How do agents communicate</li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/readme/05-memory-system.md">Memory System: Sessions, Tasks, and Distilled Memories</a> — Agents start from scratch every conversation</li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/readme/06-tool-system.md">Tool System: Registration, Capability Matching, and Execution</a> — How agents "do things" not just "talk"</li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/readme/07-workflow-engine.md">Workflow Engine: DAG-Based Agent Orchestration</a> — How do multi-step tasks execute by dependency</li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/readme/08-storage-retrieval.md">Storage and Retrieval: PostgreSQL, pgvector, and Hybrid Search</a> — Where do agent memories and knowledge live</li>
<li><a href="https://Timwood0x10.github.io/myblog/log/goagent/readme/09-embedding-service.md">Embedding Service: The Engineering Boundary of Vector Generation</a> — How does text become vectors</li>
</ol>
<h2 id="writing-principles">Writing Principles</h2>
<ul>
<li>Describe implemented source-code behavior, not wishful roadmap items.</li>
<li>Include code references so readers can verify claims in the repository.</li>
<li>Use Mermaid diagrams only; no screenshots or image dependencies.</li>
<li>Analyze design trade-offs, not just advantages.</li>
<li>Style reference: <a rel="noopener external" target="_blank" href="https://medium.com/@jinhopers/in-depth-llvm-ir-how-omniscope-tracks-ownership-across-languages-2919e418ca61">In-depth LLVM IR</a>.</li>
</ul>
Architecture Deep Dive: From Facade to Event Store, Snapshot, Analysis, and Render
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/article-architecture-facade-pipeline/
<h1 id="architecture-deep-dive-from-facade-to-event-store-snapshot-analysis-and-render">Architecture Deep Dive: From Facade to Event Store, Snapshot, Analysis, and Render</h1>
<p><code>memscope-rs</code> is not a single monolithic tracker. The project is organized as a set of connected engines:</p>
<ul>
<li>capture;</li>
<li>event storage;</li>
<li>metadata;</li>
<li>snapshot construction;</li>
<li>query;</li>
<li>analysis;</li>
<li>timeline;</li>
<li>rendering.</li>
</ul>
<p>This article explains the architecture as it exists in the source code, not as a marketing diagram.</p>
<p>The main idea is:</p>
<blockquote>
<p>capture memory events once, store them centrally, then build snapshots, analysis views, exports, and dashboards from that event stream.</p>
</blockquote>
<hr />
<h2 id="1-the-facade-layer">1. The Facade Layer</h2>
<p>The <code>MemScope</code> facade wires the major engines together.</p>
<p>The source structure is explicit:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> MemScope</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> event_store</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">EventStore</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> capture</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">CaptureEngine</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> metadata</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">MetadataEngine</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> snapshot</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">SnapshotEngine</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> query</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">QueryEngine</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> analysis</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Mutex</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">AnalysisEngine</span><span style="color: #89DCEB;">>></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> timeline</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">TimelineEngine</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> render</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">RenderEngine</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The facade exists to provide a unified entry point, but the internals remain modular.</p>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[MemScope Facade] --> B[CaptureEngine]
A --> C[EventStore]
A --> D[SnapshotEngine]
A --> E[AnalysisEngine]
A --> F[QueryEngine]
A --> G[TimelineEngine]
A --> H[RenderEngine]
B --> C
C --> D
D --> E
D --> F
D --> G
D --> H
</div>
<p>This architecture is useful because it separates event collection from analysis and presentation.</p>
<hr />
<h2 id="2-captureengine-backend-abstraction">2. CaptureEngine: Backend Abstraction</h2>
<p><code>CaptureEngine</code> is responsible for turning allocation operations into <code>MemoryEvent</code>s and sending them to <code>EventStore</code>.</p>
<p>Its key fields are:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> CaptureEngine</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> backend</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Box</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">dyn</span><span style="color: #F9E2AF;font-style: italic;"> CaptureBackend</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> event_store</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> SharedEventStore</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The allocation path is simple:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> capture_alloc</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> thread_id</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> event</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>backend</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">capture_alloc</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;">,</span><span> thread_id</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>event_store</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">record</span><span style="color: #9399B2;">(</span><span>event</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The backend abstraction supports different capture strategies:</p>
<ul>
<li><code>Core</code></li>
<li><code>Lockfree</code></li>
<li><code>Async</code></li>
<li><code>Unified</code></li>
</ul>
<p>The important point is that capture does not own the data. It forwards events into a shared event stream.</p>
<hr />
<h2 id="3-eventstore-the-central-event-stream">3. EventStore: The Central Event Stream</h2>
<p>The <code>EventStore</code> is the central storage layer:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> EventStore</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> queue</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> SegQueue</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">MemoryEvent</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> cache</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> RwLock</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">MemoryEvent</span><span style="color: #89DCEB;">>></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> count</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicUsize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> clearing</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicUsize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Recording is queue-based:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> record</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> event</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> MemoryEvent</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>clearing</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">load</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Acquire</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> !=</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>queue</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span>event</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>count</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">fetch_add</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Release</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Snapshotting flushes the queue into the cache:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> flush_to_cache</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let mut</span><span> cache</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>cache</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">write</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> while let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>event</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>queue</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">pop</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span> cache</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span>event</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is not a pure append-only immutable log. It is a practical event store with a queue, cache, count, and clear coordination.</p>
<hr />
<h2 id="4-snapshotengine-reconstructing-state-from-events">4. SnapshotEngine: Reconstructing State from Events</h2>
<p>The <code>SnapshotEngine</code> builds point-in-time views from <code>EventStore</code>.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> SnapshotEngine</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> event_store</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> SharedEventStore</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Its main method reads all events and constructs a snapshot:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> build_snapshot</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> MemorySnapshot</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> events</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>event_store</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">snapshot</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> build_snapshot_from_events</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>events</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is important architecturally:</p>
<blockquote>
<p>snapshots are derived from events; they are not the primary source of truth.</p>
</blockquote>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
graph LR
A[MemoryEvent Queue] --> B[EventStore Snapshot]
B --> C[build snapshot from events]
C --> D[MemorySnapshot]
D --> E[Analysis / Query / Render]
</div>
<hr />
<h2 id="5-renderengine-output-as-a-separate-concern">5. RenderEngine: Output as a Separate Concern</h2>
<p><code>RenderEngine</code> consumes snapshots and renders output.</p>
<p>It has a registry of renderers:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> RenderEngine</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> snapshot_engine</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> SharedSnapshotEngine</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> renderers</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Box</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">dyn</span><span style="color: #F9E2AF;font-style: italic;"> Renderer</span><span style="color: #89DCEB;">>></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>By default, it registers a JSON renderer:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> engine</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> Self</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> snapshot_engine</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> renderers</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;">};</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>engine</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">register_renderer</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">Box</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">JsonRenderer</span><span style="color: #9399B2;">));</span></span></code></pre>
<p>This separation means rendering can evolve without changing capture or event storage.</p>
<hr />
<h2 id="6-globaltracker-vs-memscope">6. GlobalTracker vs MemScope</h2>
<p>The project has both a lower-level facade architecture and a higher-level global tracker path.</p>
<p>The <code>GlobalTracker</code> path is used by many examples and exposes convenience methods such as:</p>
<ul>
<li><code>track_as()</code></li>
<li><code>create_passport()</code></li>
<li><code>record_handover()</code></li>
<li><code>export_json()</code></li>
<li><code>export_html()</code></li>
</ul>
<p>This is convenient, but it also means there are multiple entry points into the system. Articles and documentation should be clear about which path they describe.</p>
<hr />
<h2 id="7-known-architectural-tradeoffs">7. Known Architectural Tradeoffs</h2>
<h3 id="strengths">Strengths</h3>
<ul>
<li>Capture and analysis are decoupled.</li>
<li>Events are centralized.</li>
<li>Snapshot construction is derived from events.</li>
<li>Rendering/export is separate from tracking.</li>
<li>Multiple backends can share the same event model.</li>
</ul>
<h3 id="tradeoffs">Tradeoffs</h3>
<ul>
<li>Some paths use global state.</li>
<li>Some components use <code>Mutex</code> internally.</li>
<li>There are multiple APIs (<code>Tracker</code>, <code>GlobalTracker</code>, <code>MemScope</code>) that can confuse users if not documented clearly.</li>
<li>Some conversion paths use placeholder values, such as reconstructing <code>ThreadId</code> from <code>u64</code> being impossible.</li>
</ul>
<p>The architecture is practical and modular, but it is not minimal.</p>
<hr />
<h2 id="8-summary">8. Summary</h2>
<p>The core architecture can be summarized as:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>Capture -> EventStore -> Snapshot -> Analysis / Query / Render</span></span></code></pre>
<p><code>memscope-rs</code> is strongest when this event-driven architecture is kept clear. Runtime events should remain the source of truth; everything else should be understood as a derived view or analysis layer.</p>
Benchmark Deep Dive: What the Numbers Actually Say
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/article-benchmark-deep-dive/
<h1 id="benchmark-deep-dive-what-the-numbers-actually-say">Benchmark Deep Dive: What the Numbers Actually Say</h1>
<p>Performance claims are easy to overstate. This article keeps the interpretation narrow and tied to the benchmark source and log that exist in the repository.</p>
<p>The benchmark suite is in:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>benches/comprehensive_benchmarks.rs</span></span></code></pre>
<p>The raw benchmark log reviewed here is:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>benches/run.log</span></span></code></pre>
<p>The goal is not to claim that <code>memscope-rs</code> is always fast. The goal is to identify which operations are currently cheap, which operations scale linearly, and where the log already shows regressions.</p>
<hr />
<h2 id="1-benchmark-coverage">1. Benchmark Coverage</h2>
<p>The benchmark source includes groups such as:</p>
<ul>
<li>tracker creation;</li>
<li>single variable tracking;</li>
<li>multiple variable tracking;</li>
<li>analysis;</li>
<li>stats;</li>
<li>backend alloc/dealloc/realloc/move;</li>
<li>type classification;</li>
<li>concurrent tracking;</li>
<li>parallel tracking;</li>
<li>shared tracker concurrent tracking;</li>
<li>allocation patterns;</li>
<li>analysis operations;</li>
<li>tracking stats;</li>
<li>IO operations.</li>
</ul>
<p>This is broad coverage, but not every feature has an isolated benchmark. For example, there is no standalone benchmark specifically isolating <code>StackOwner</code> grouping or async attribution overhead.</p>
<hr />
<h2 id="2-tracker-creation">2. Tracker Creation</h2>
<p>The log reports:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>tracker_creation time: [867.65 ns 873.02 ns 879.91 ns]</span></span>
<span class="giallo-l"><span>change: [+551.47% +559.50% +568.23%]</span></span>
<span class="giallo-l"><span>Performance has regressed.</span></span></code></pre>
<p>Interpretation:</p>
<ul>
<li>absolute time is still sub-microsecond;</li>
<li>relative performance regressed significantly against the previous baseline;</li>
<li>this should not be described as an unqualified win.</li>
</ul>
<hr />
<h2 id="3-single-variable-tracking">3. Single Variable Tracking</h2>
<p>Representative results:</p>
<table><thead><tr><th>Benchmark</th><th style="text-align: right">Approximate Median</th></tr></thead><tbody>
<tr><td><code>track_single/vec/64</code></td><td style="text-align: right">653.02 ns</td></tr>
<tr><td><code>track_single/vec/256</code></td><td style="text-align: right">662.81 ns</td></tr>
<tr><td><code>track_single/vec/1024</code></td><td style="text-align: right">666.22 ns</td></tr>
<tr><td><code>track_single/vec/4096</code></td><td style="text-align: right">725.49 ns</td></tr>
<tr><td><code>track_single/vec/65536</code></td><td style="text-align: right">1.1087 µs</td></tr>
<tr><td><code>track_single/vec/1048576</code></td><td style="text-align: right">4.9307 µs</td></tr>
</tbody></table>
<p>The small-object tracking path is sub-microsecond, but the log marks several of these as regressions relative to earlier runs.</p>
<p>Interpretation:</p>
<ul>
<li>single tracking overhead is practically usable for profiling;</li>
<li>overhead is not zero;</li>
<li>recent changes appear to have increased latency for smaller payloads.</li>
</ul>
<hr />
<h2 id="4-multiple-variable-tracking">4. Multiple Variable Tracking</h2>
<p>Representative results:</p>
<table><thead><tr><th style="text-align: right">Variables</th><th style="text-align: right">Approximate Median</th></tr></thead><tbody>
<tr><td style="text-align: right">10</td><td style="text-align: right">6.5364 µs</td></tr>
<tr><td style="text-align: right">50</td><td style="text-align: right">33.240 µs</td></tr>
<tr><td style="text-align: right">100</td><td style="text-align: right">66.963 µs</td></tr>
<tr><td style="text-align: right">1000</td><td style="text-align: right">669.67 µs</td></tr>
<tr><td style="text-align: right">5000</td><td style="text-align: right">3.3126 ms</td></tr>
<tr><td style="text-align: right">10000</td><td style="text-align: right">6.5949 ms</td></tr>
</tbody></table>
<p>This is close to linear scaling.</p>
<p>Interpretation:</p>
<ul>
<li>batch tracking cost grows predictably;</li>
<li>the per-item cost is roughly stable;</li>
<li>the log marks these paths as regressions against the prior baseline.</li>
</ul>
<hr />
<h2 id="5-analysis-cost">5. Analysis Cost</h2>
<p>Representative analysis results:</p>
<table><thead><tr><th style="text-align: right">Records</th><th style="text-align: right">Approximate Median</th></tr></thead><tbody>
<tr><td style="text-align: right">10</td><td style="text-align: right">5.2939 µs</td></tr>
<tr><td style="text-align: right">50</td><td style="text-align: right">16.028 µs</td></tr>
<tr><td style="text-align: right">100</td><td style="text-align: right">29.759 µs</td></tr>
<tr><td style="text-align: right">1000</td><td style="text-align: right">285.64 µs</td></tr>
<tr><td style="text-align: right">5000</td><td style="text-align: right">1.5684 ms</td></tr>
<tr><td style="text-align: right">10000</td><td style="text-align: right">4.1880 ms</td></tr>
<tr><td style="text-align: right">50000</td><td style="text-align: right">33.887 ms</td></tr>
</tbody></table>
<p>Interpretation:</p>
<ul>
<li>small and medium analysis is fast enough for interactive use;</li>
<li>large analysis becomes millisecond-scale;</li>
<li>50,000 records is tens of milliseconds;</li>
<li>several larger analysis cases show significant relative regressions.</li>
</ul>
<hr />
<h2 id="6-backend-event-construction">6. Backend Event Construction</h2>
<p>Representative backend allocation times:</p>
<table><thead><tr><th>Backend</th><th style="text-align: right">Alloc Median</th></tr></thead><tbody>
<tr><td>Core</td><td style="text-align: right">23.022 ns</td></tr>
<tr><td>Lockfree</td><td style="text-align: right">39.265 ns</td></tr>
<tr><td>Async</td><td style="text-align: right">23.008 ns</td></tr>
<tr><td>Unified</td><td style="text-align: right">39.512 ns</td></tr>
</tbody></table>
<p>Representative deallocation times:</p>
<table><thead><tr><th>Backend</th><th style="text-align: right">Dealloc Median</th></tr></thead><tbody>
<tr><td>Core</td><td style="text-align: right">22.859 ns</td></tr>
<tr><td>Lockfree</td><td style="text-align: right">38.586 ns</td></tr>
<tr><td>Async</td><td style="text-align: right">22.632 ns</td></tr>
<tr><td>Unified</td><td style="text-align: right">39.147 ns</td></tr>
</tbody></table>
<p>Interpretation:</p>
<ul>
<li>backend event construction is nanosecond-scale;</li>
<li>Core and Async are similar in this benchmark;</li>
<li>Lockfree and Unified are slower, roughly in the high-30ns range;</li>
<li>this benchmark measures event construction, not full end-to-end application overhead.</li>
</ul>
<hr />
<h2 id="7-concurrent-tracking">7. Concurrent Tracking</h2>
<p>Representative concurrent tracking results:</p>
<table><thead><tr><th style="text-align: right">Threads</th><th style="text-align: right">Approximate Median</th></tr></thead><tbody>
<tr><td style="text-align: right">1</td><td style="text-align: right">19.174 µs</td></tr>
<tr><td style="text-align: right">2</td><td style="text-align: right">40.599 µs</td></tr>
<tr><td style="text-align: right">4</td><td style="text-align: right">55.303 µs</td></tr>
<tr><td style="text-align: right">8</td><td style="text-align: right">134.74 µs</td></tr>
<tr><td style="text-align: right">16</td><td style="text-align: right">372.96 µs</td></tr>
<tr><td style="text-align: right">32</td><td style="text-align: right">961.58 µs</td></tr>
<tr><td style="text-align: right">64</td><td style="text-align: right">1.8646 ms</td></tr>
<tr><td style="text-align: right">128</td><td style="text-align: right">4.6714 ms</td></tr>
</tbody></table>
<p>Interpretation:</p>
<ul>
<li>concurrency works;</li>
<li>scaling is not linear;</li>
<li>thread scheduling and shared state costs are visible;</li>
<li>48-thread results show regression, while some larger thread-count cases show improvement against the previous baseline.</li>
</ul>
<hr />
<h2 id="8-shared-tracker-concurrent-tracking">8. Shared Tracker Concurrent Tracking</h2>
<p>Representative shared tracker results:</p>
<table><thead><tr><th style="text-align: right">Threads</th><th style="text-align: right">Approximate Median</th></tr></thead><tbody>
<tr><td style="text-align: right">1</td><td style="text-align: right">98.300 µs</td></tr>
<tr><td style="text-align: right">2</td><td style="text-align: right">231.96 µs</td></tr>
<tr><td style="text-align: right">4</td><td style="text-align: right">363.82 µs</td></tr>
<tr><td style="text-align: right">8</td><td style="text-align: right">924.19 µs</td></tr>
<tr><td style="text-align: right">16</td><td style="text-align: right">1.8448 ms</td></tr>
<tr><td style="text-align: right">32</td><td style="text-align: right">3.5680 ms</td></tr>
<tr><td style="text-align: right">64</td><td style="text-align: right">7.0581 ms</td></tr>
</tbody></table>
<p>Interpretation:</p>
<ul>
<li>sharing one tracker across many threads is a stress scenario;</li>
<li>costs grow clearly with thread count;</li>
<li>the log shows improvements against previous runs, but absolute shared-state cost remains visible.</li>
</ul>
<hr />
<h2 id="9-allocation-patterns">9. Allocation Patterns</h2>
<p>Representative results:</p>
<table><thead><tr><th>Pattern</th><th style="text-align: right">Approximate Median</th><th>Log Status</th></tr></thead><tbody>
<tr><td>many small allocations</td><td style="text-align: right">809.84 µs</td><td>regressed</td></tr>
<tr><td>few large allocations</td><td style="text-align: right">96.308 µs</td><td>improved</td></tr>
<tr><td>mixed size allocations</td><td style="text-align: right">111.44 µs</td><td>regressed</td></tr>
<tr><td>burst allocations</td><td style="text-align: right">789.08 µs</td><td>no significant change</td></tr>
</tbody></table>
<p>Interpretation:</p>
<ul>
<li>many small allocations remain expensive compared with few large allocations;</li>
<li>allocation pattern matters;</li>
<li>performance should be discussed by workload shape, not one global number.</li>
</ul>
<hr />
<h2 id="10-tracking-stats">10. Tracking Stats</h2>
<p>Some statistics operations are extremely cheap:</p>
<table><thead><tr><th>Operation</th><th style="text-align: right">Approximate Median</th></tr></thead><tbody>
<tr><td><code>stats_record_attempt</code></td><td style="text-align: right">1.8200 ns</td></tr>
<tr><td><code>stats_record_success</code></td><td style="text-align: right">1.8211 ns</td></tr>
<tr><td><code>stats_record_miss</code></td><td style="text-align: right">3.2607 ns</td></tr>
<tr><td><code>stats_get_completeness</code></td><td style="text-align: right">548.58 ps</td></tr>
<tr><td><code>stats_get_detailed_stats</code></td><td style="text-align: right">1.6463 ns</td></tr>
</tbody></table>
<p>These are tiny operations, but they should not be confused with full tracking or analysis cost.</p>
<hr />
<h2 id="11-what-the-benchmark-does-not-prove">11. What the Benchmark Does Not Prove</h2>
<p>The benchmark log does not prove:</p>
<ul>
<li>production overhead under all workloads;</li>
<li>async attribution overhead in isolation;</li>
<li><code>StackOwner</code> grouping cost in isolation;</li>
<li>dashboard rendering cost under large reports;</li>
<li>memory overhead under long-running services;</li>
<li>correctness of relation inference.</li>
</ul>
<p>Benchmarks measure performance of specific paths, not the whole tool in every environment.</p>
<hr />
<h2 id="12-honest-summary">12. Honest Summary</h2>
<p>The accurate performance story is:</p>
<ul>
<li>core event construction is nanosecond-scale;</li>
<li>explicit variable tracking is usually sub-microsecond for small values;</li>
<li>batch tracking scales roughly linearly;</li>
<li>analysis becomes millisecond-scale for large record counts;</li>
<li>concurrency is supported but not free;</li>
<li>shared tracker scenarios show real contention cost;</li>
<li>benchmark logs include both improvements and regressions.</li>
</ul>
<p>The most honest phrasing is:</p>
<blockquote>
<p><code>memscope-rs</code> has practical profiling overhead for many measured paths, but it is not zero-cost and the benchmark log shows active performance evolution.</p>
</blockquote>
Export and Dashboard Pipeline: Turning Memory Events into Reports
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/article-export-dashboard-pipeline/
<h1 id="export-and-dashboard-pipeline-turning-memory-events-into-reports">Export and Dashboard Pipeline: Turning Memory Events into Reports</h1>
<p>Collecting memory data is only useful if developers can inspect it.</p>
<p><code>memscope-rs</code> exports both machine-readable JSON and HTML dashboards. The export pipeline is not a thin wrapper around one report; it combines snapshots, passports, async task data, ownership graphs, system resources, and task graphs.</p>
<p>This article explains what the export pipeline actually writes and how it is assembled.</p>
<hr />
<h2 id="1-export-starts-from-the-event-store">1. Export Starts from the Event Store</h2>
<p>The main export entry point is <code>export_all_json()</code>.</p>
<p>The first step is to read events from the tracker’s event store:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> events</span><span style="color: #94E2D5;"> =</span><span> tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">event_store</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">snapshot</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> allocations</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> rebuild_allocations_from_events</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>events</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> snapshot</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> MemorySnapshot</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">from_allocation_infos</span><span style="color: #9399B2;">(</span><span>allocations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">());</span></span></code></pre>
<p>This is important:</p>
<blockquote>
<p>export does not only use a live allocation map; it rebuilds report data from the event stream.</p>
</blockquote>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[EventStore] --> B[snapshot]
B --> C[rebuild_allocations_from_events]
C --> D[MemorySnapshot]
D --> E[JSON Reports]
D --> F[Dashboard Context]
</div>
<hr />
<h2 id="2-what-export-all-json-writes">2. What <code>export_all_json()</code> Writes</h2>
<p>The export pipeline writes multiple files:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">export_snapshot_to_json</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>snapshot</span><span style="color: #9399B2;">,</span><span> path_ref</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>options</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">export_memory_passports_json</span><span style="color: #9399B2;">(</span><span>path_ref</span><span style="color: #9399B2;">,</span><span> passport_tracker</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">export_leak_detection_json</span><span style="color: #9399B2;">(</span><span>path_ref</span><span style="color: #9399B2;">,</span><span> passport_tracker</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">export_unsafe_ffi_json</span><span style="color: #9399B2;">(</span><span>path_ref</span><span style="color: #9399B2;">,</span><span> passport_tracker</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">export_system_resources_json</span><span style="color: #9399B2;">(</span><span>path_ref</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">export_async_analysis_json</span><span style="color: #9399B2;">(</span><span>path_ref</span><span style="color: #9399B2;">,</span><span> async_tracker</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">export_ownership_graph_json</span><span style="color: #9399B2;">(</span><span>path_ref</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>typed_allocations</span><span style="color: #9399B2;">,</span><span> tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">event_store</span><span style="color: #9399B2;">())</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">export_task_graph_json</span><span style="color: #9399B2;">(</span><span>path_ref</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span></code></pre>
<p>The actual outputs include:</p>
<ul>
<li>memory snapshot JSON;</li>
<li>memory passport JSON;</li>
<li>leak detection JSON;</li>
<li>unsafe/FFI JSON;</li>
<li>system resources JSON;</li>
<li>async analysis JSON;</li>
<li>ownership graph JSON;</li>
<li>task graph JSON.</li>
</ul>
<p>This makes the export pipeline one of the main integration points in the project.</p>
<hr />
<h2 id="3-memory-passport-export">3. Memory Passport Export</h2>
<p>Passport export summarizes lifecycle records:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #F9E2AF;">serde_json</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "passport_id"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span>passport_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "allocation_ptr"</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"0x</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:x</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> p</span><span style="color: #94E2D5;">.</span><span>allocation_ptr</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "size_bytes"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span>size_bytes</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "created_at"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span>created_at</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "lifecycle_events"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span>lifecycle_events</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">len</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "status"</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:?</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> p</span><span style="color: #94E2D5;">.</span><span>status_at_shutdown</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #9399B2;">})</span></span></code></pre>
<p>This is intentionally a summary. It does not dump every internal detail by default.</p>
<hr />
<h2 id="4-unsafe-ffi-export">4. Unsafe/FFI Export</h2>
<p>Unsafe/FFI export filters passports by FFI-relevant status:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> ffi_reports</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span>_</span><span style="color: #89DCEB;">></span><span style="color: #94E2D5;"> =</span><span> passports</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">values</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">filter</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>p</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> matches!</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">status_at_shutdown</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> PassportStatus</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">HandoverToFfi</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> |</span><span style="color: #F9E2AF;"> PassportStatus</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">InForeignCustody</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> |</span><span style="color: #F9E2AF;"> PassportStatus</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">FreedByForeign</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> )</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">collect</span><span style="color: #9399B2;">();</span></span></code></pre>
<p>This means <code>unsafe_ffi.json</code> is not just a raw memory dump. It is a filtered report focused on boundary-related memory.</p>
<hr />
<h2 id="5-async-export">5. Async Export</h2>
<p>Async export writes <code>async_analysis.json</code> with summary, task profiles, and active allocations.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> async_data</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "summary"</span><span style="color: #94E2D5;">:</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_tasks"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>total_tasks</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "active_tasks"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>active_tasks</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_allocations"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>total_allocations</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_memory_bytes"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>total_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "active_memory_bytes"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>active_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "peak_memory_bytes"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>peak_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "task_profiles"</span><span style="color: #94E2D5;">:</span><span> profiles</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>p</span><span style="color: #94E2D5;">|</span><span style="color: #89B4FA;font-style: italic;"> json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "task_id"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">task_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "task_name"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">task_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "current_memory"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">current_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "peak_memory"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">peak_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_allocations"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">total_allocations</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_deallocations"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">total_deallocations</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "is_completed"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">is_completed</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "has_potential_leak"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">has_potential_leak</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "allocations"</span><span style="color: #94E2D5;">:</span><span> snapshot</span><span style="color: #94E2D5;">.</span><span>allocations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>a</span><span style="color: #94E2D5;">|</span><span style="color: #89B4FA;font-style: italic;"> json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "ptr"</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"0x</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:x</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> a</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">ptr</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "size"</span><span style="color: #94E2D5;">:</span><span> a</span><span style="color: #94E2D5;">.</span><span>size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "task_id"</span><span style="color: #94E2D5;">:</span><span> a</span><span style="color: #94E2D5;">.</span><span>task_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "var_name"</span><span style="color: #94E2D5;">:</span><span> a</span><span style="color: #94E2D5;">.</span><span>var_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "type_name"</span><span style="color: #94E2D5;">:</span><span> a</span><span style="color: #94E2D5;">.</span><span>type_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })),</span></span>
<span class="giallo-l"><span style="color: #9399B2;">});</span></span></code></pre>
<p>This is the bridge between runtime async attribution and offline analysis.</p>
<hr />
<h2 id="6-ownership-graph-export">6. Ownership Graph Export</h2>
<p>Ownership graph export includes nodes and edges.</p>
<p>Node export:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "id"</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"0x</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:x</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> node</span><span style="color: #94E2D5;">.</span><span>id</span><span style="color: #94E2D5;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "type_name"</span><span style="color: #94E2D5;">:</span><span> node</span><span style="color: #94E2D5;">.</span><span>type_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "size"</span><span style="color: #94E2D5;">:</span><span> node</span><span style="color: #94E2D5;">.</span><span>size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "stack_ptr"</span><span style="color: #94E2D5;">:</span><span> node</span><span style="color: #94E2D5;">.</span><span>stack_ptr</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>p</span><span style="color: #94E2D5;">|</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"0x</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:x</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> p</span><span style="color: #9399B2;">)),</span></span>
<span class="giallo-l"><span style="color: #9399B2;">})</span></span></code></pre>
<p>Edge export:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #A6E3A1;">"kind"</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> match</span><span> edge</span><span style="color: #94E2D5;">.</span><span>op </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Owns</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "Owns"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Contains</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "Contains"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Borrows</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "Borrows"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">RcClone</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "RcClone"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">ArcClone</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "ArcClone"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Move</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "Move"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">SharedBorrow</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "SharedBorrow"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">MutBorrow</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "MutBorrow"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This output is only as accurate as the underlying relation inference. It should be read as an analysis graph, not a compiler proof.</p>
<hr />
<h2 id="7-dashboard-rendering">7. Dashboard Rendering</h2>
<p>The HTML dashboard path builds a dashboard context from tracker data and renders it with a template.</p>
<p>The export function chooses a dashboard template:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">match</span><span> template</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> DashboardTemplate</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Final</span><span style="color: #94E2D5;"> =></span><span> renderer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">render_final_dashboard</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>context</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> DashboardTemplate</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Unified</span><span style="color: #94E2D5;"> =></span><span> renderer</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">render_unified_dashboard</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>context</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The dashboard is therefore a rendered view over the same reconstructed data used by JSON export.</p>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[Tracker + EventStore] --> B[Dashboard Context]
C[Passport Tracker] --> B
D[Async Tracker] --> B
B --> E[Handlebars Renderer]
E --> F[Unified Dashboard HTML]
</div>
<hr />
<h2 id="8-practical-value">8. Practical Value</h2>
<p>The export pipeline matters because it gives different audiences different entry points:</p>
<ul>
<li>JSON for tooling and CI;</li>
<li>HTML dashboard for human inspection;</li>
<li>ownership graph JSON for visual relation analysis;</li>
<li>async JSON for task-level diagnosis;</li>
<li>passport JSON for unsafe/FFI audit trails.</li>
</ul>
<hr />
<h2 id="9-limitations">9. Limitations</h2>
<ul>
<li>Exported reports are derived from recorded events; missing events mean missing report data.</li>
<li>Some reports summarize data rather than dumping every internal detail.</li>
<li>Ownership graph export includes inferred relations.</li>
<li>Async export depends on explicit async context attribution.</li>
<li>Unsafe/FFI export depends on passports being created and updated.</li>
</ul>
<hr />
<h2 id="10-summary">10. Summary</h2>
<p>The export and dashboard pipeline is the point where <code>memscope-rs</code> becomes usable as a developer tool.</p>
<p>It turns event streams into:</p>
<ul>
<li>snapshots;</li>
<li>JSON files;</li>
<li>ownership graphs;</li>
<li>async task reports;</li>
<li>memory passport reports;</li>
<li>dashboards.</li>
</ul>
<p>The most accurate description is:</p>
<blockquote>
<p>export is not just serialization; it is the integration layer that combines runtime facts, explicit metadata, and analysis results into inspectable artifacts.</p>
</blockquote>
Ownership Graph and Relation Inference: From Memory Events to Rust Semantics
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/article-ownership-graph-relation-inference/
<h1 id="ownership-graph-and-relation-inference-from-memory-events-to-rust-semantics">Ownership Graph and Relation Inference: From Memory Events to Rust Semantics</h1>
<p>The previous articles focused on data collection:</p>
<ul>
<li>single-thread tracking records allocation facts and Rust-level metadata;</li>
<li>lockfree tracking handles concurrent event capture;</li>
<li>memory passports describe unsafe and FFI lifecycles;</li>
<li><code>StackOwner</code> makes <code>Arc/Rc</code> shared ownership observable;</li>
<li>async attribution connects allocations to logical tasks.</li>
</ul>
<p>This article focuses on the next layer:</p>
<blockquote>
<p>How does <code>memscope-rs</code> turn collected memory records into relationships?</p>
</blockquote>
<p>This is where raw events become a graph of ownership, containment, sharing, slices, clones, and inferred relations.</p>
<p>The key word is <strong>inferred</strong>. This graph is not a compiler proof of Rust ownership. It is a post-analysis structure built from runtime facts, explicit metadata, memory scanning, and heuristics.</p>
<hr />
<h2 id="1-why-a-relation-graph-exists">1. Why a Relation Graph Exists</h2>
<p>Raw allocation records are useful, but they do not explain structure.</p>
<p>An allocation list can tell us:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>ptr=0x1000 size=1024 type=Vec<T></span></span>
<span class="giallo-l"><span>ptr=0x5000 size=64 type=HashMap<K,V></span></span>
<span class="giallo-l"><span>ptr=0x9000 size=24 type=Arc<T></span></span></code></pre>
<p>But it does not directly answer:</p>
<ul>
<li>Which object owns another object?</li>
<li>Which container likely contains which heap allocations?</li>
<li>Which <code>Arc/Rc</code> values share the same heap data?</li>
<li>Which object is a slice/view into another?</li>
<li>Which allocations look like clones?</li>
<li>Are there cycles in the relationship graph?</li>
</ul>
<p>The relation graph exists to answer those questions as an analysis layer.</p>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[MemoryEvent Stream] --> B[Snapshot]
B --> C[Active Allocations]
C --> D[Inference Records]
D --> E[Relation Detectors]
E --> F[Owns]
E --> G[Contains]
E --> H[Shares]
E --> I[Slice]
E --> J[Clone]
E --> K[ArcClone / RcClone]
F --> L[RelationGraph]
G --> L
H --> L
I --> L
J --> L
K --> L
L --> M[Diagnostics]
L --> N[JSON Export]
L --> O[Dashboard]
</div>
<hr />
<h2 id="2-relation-types">2. Relation Types</h2>
<p>The relation inference module defines the core relationship types:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> Relation</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Owns</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Contains</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Shares</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Slice</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Clone</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Evolution</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> ArcClone</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> RcClone</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> ImmutableBorrow</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> MutableBorrow</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>These represent analysis edges between allocation records.</p>
<p>The important distinction is that not all of these relations have the same confidence level.</p>
<p>For example:</p>
<ul>
<li><code>Owns</code> can be inferred from pointer scanning.</li>
<li><code>Contains</code> is inferred from <code>Container</code> metadata and temporal locality.</li>
<li><code>ArcClone</code> can be inferred from <code>StackOwner</code> records sharing a <code>heap_ptr</code>.</li>
<li><code>Clone</code> is inferred from type, size, call stack, time window, and content similarity.</li>
<li>borrow/move-related edges are higher-level semantic approximations, not direct runtime hooks.</li>
</ul>
<hr />
<h2 id="3-the-builder-pipeline">3. The Builder Pipeline</h2>
<p>The main pipeline is in <code>RelationGraphBuilder::build()</code>.</p>
<p>At a high level, it does this:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> build</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> allocations</span><span style="color: #94E2D5;">: &</span><span style="color: #9399B2;">[</span><span style="color: #F9E2AF;font-style: italic;">ActiveAllocation</span><span style="color: #9399B2;">],</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> config</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">GraphBuilderConfig</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> RelationGraph</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> scan_results</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> HeapScanner</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">scan</span><span style="color: #9399B2;">(</span><span>allocations</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> records</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">InferenceRecord</span><span style="color: #89DCEB;">></span><span style="color: #94E2D5;"> =</span><span> allocations</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">enumerate</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;">(</span><span>id</span><span style="color: #9399B2;">,</span><span> alloc</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> InferenceRecord</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>ptr</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">unwrap_or</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> size</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> memory</span><span style="color: #94E2D5;">:</span><span> scanned_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> type_kind</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> confidence</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> call_stack_hash</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>call_stack_hash</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> alloc_time</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>allocated_at</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> stack_ptr</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>stack_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">collect</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> range_map</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> RangeMap</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span>allocations</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let mut</span><span> graph</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> RelationGraph</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> record</span><span style="color: #CBA6F7;"> in</span><span style="color: #94E2D5;"> &</span><span>records</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> graph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">add_edges</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">detect_owner</span><span style="color: #9399B2;">(</span><span>record</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>range_map</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> graph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">add_edges</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">detect_slice</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>records</span><span style="color: #9399B2;">,</span><span> allocations</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>range_map</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"><span> graph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">add_edges</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">detect_clones</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>records</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>config</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">clone_config</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"><span> graph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">add_edges</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">detect_containers</span><span style="color: #9399B2;">(</span><span>allocations</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>config</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">container_config</span><span style="color: #9399B2;">)));</span></span>
<span class="giallo-l"><span> graph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">add_edges</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">detect_variable_evolution</span><span style="color: #9399B2;">(</span><span>allocations</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"><span> graph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">add_edges</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">detect_shared</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>records</span><span style="color: #9399B2;">,</span><span style="color: #94E2D5;"> &</span><span>graph</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">edges</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> graph</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The real implementation runs these steps:</p>
<ol>
<li>scan heap memory;</li>
<li>build inference records;</li>
<li>build an address range map;</li>
<li>detect owner relations;</li>
<li>detect slice relations;</li>
<li>detect clone relations;</li>
<li>detect container relations;</li>
<li>detect variable evolution;</li>
<li>detect shared ownership.</li>
</ol>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[ActiveAllocation List] --> B[HeapScanner]
B --> C[ScanResult]
C --> D[UTI / Type Inference]
D --> E[InferenceRecord]
E --> F[RangeMap]
F --> G[Owner Detector]
F --> H[Slice Detector]
E --> I[Clone Detector]
A --> J[Container Detector]
A --> K[Variable Evolution]
E --> L[Shared Detector]
G --> M[RelationGraph]
H --> M
I --> M
J --> M
K --> M
L --> M
</div>
<hr />
<h2 id="4-inferencerecord-the-analysis-unit">4. <code>InferenceRecord</code>: The Analysis Unit</h2>
<p>The relation builder converts runtime allocations into inference records.</p>
<p>The fields include:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;">InferenceRecord</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>ptr</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">unwrap_or</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span> size</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> type_kind</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> confidence</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> call_stack_hash</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>call_stack_hash</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> alloc_time</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>allocated_at</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> stack_ptr</span><span style="color: #94E2D5;">:</span><span> alloc</span><span style="color: #94E2D5;">.</span><span>stack_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This record is the bridge between raw memory data and relation inference.</p>
<p>It may contain:</p>
<ul>
<li>pointer address;</li>
<li>allocation size;</li>
<li>scanned memory bytes;</li>
<li>inferred type kind;</li>
<li>inference confidence;</li>
<li>call stack hash;</li>
<li>allocation timestamp;</li>
<li><code>stack_ptr</code> for <code>StackOwner</code> values such as <code>Arc/Rc</code>.</li>
</ul>
<p>This is also where confidence boundaries begin. If memory cannot be scanned, type inference and pointer-based relation detection become less informative.</p>
<hr />
<h2 id="5-owner-detection-pointer-scanning">5. Owner Detection: Pointer Scanning</h2>
<p>Owner detection scans an allocation’s memory for pointer-sized values that point into another allocation.</p>
<p>Simplified:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> detect_owner</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;">record</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">InferenceRecord</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> range_map</span><span style="color: #94E2D5;">: &</span><span style="color: #F9E2AF;font-style: italic;">RangeMap</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">RelationEdge</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> memory</span><span style="color: #94E2D5;"> =</span><span style="color: #CBA6F7;"> match</span><span style="color: #94E2D5;"> &</span><span>record</span><span style="color: #94E2D5;">.</span><span>memory </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>m</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =></span><span> m</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> None</span><span style="color: #94E2D5;"> =></span><span style="color: #CBA6F7;"> return</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> offset</span><span style="color: #CBA6F7;"> in</span><span style="color: #9399B2;"> (</span><span style="color: #FAB387;">0</span><span style="color: #94E2D5;">..</span><span>memory</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">len</span><span style="color: #9399B2;">())</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">step_by</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">mem</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">size_of</span><span style="color: #94E2D5;">::</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">()) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>ptr_val</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span> memory</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">read_usize</span><span style="color: #9399B2;">(</span><span>offset</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> else</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> continue</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> ptr_val</span><span style="color: #94E2D5;"> ==</span><span style="color: #FAB387;"> 0</span><span style="color: #94E2D5;"> ||</span><span> ptr_val</span><span style="color: #94E2D5;"> <</span><span style="color: #FAB387;"> MIN_VALID_POINTER</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> continue</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>target_id</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span> range_map</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">find_containing</span><span style="color: #9399B2;">(</span><span>ptr_val</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span> relations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">RelationEdge</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> from</span><span style="color: #94E2D5;">:</span><span> record</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> to</span><span style="color: #94E2D5;">:</span><span> target_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> relation</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;"> Relation</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Owns</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> });</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> relations</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is a useful heuristic: if allocation A contains a pointer into allocation B, A may own or reference B.</p>
<p>But it is still a heuristic. A pointer-looking value in memory does not always mean Rust ownership.</p>
<hr />
<h2 id="6-slice-detection">6. Slice Detection</h2>
<p>Slice detection looks for records whose pointer falls inside another allocation rather than at the beginning.</p>
<p>Conceptually:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">if</span><span> ptr</span><span style="color: #94E2D5;"> !=</span><span> target_start</span><span style="color: #94E2D5;"> &&</span><span> ptr</span><span style="color: #94E2D5;"> >=</span><span> target_start</span><span style="color: #94E2D5;"> &&</span><span> ptr</span><span style="color: #94E2D5;"> +</span><span> size</span><span style="color: #94E2D5;"> <=</span><span> target_end</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> relations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">RelationEdge</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> from</span><span style="color: #94E2D5;">:</span><span> record</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> to</span><span style="color: #94E2D5;">:</span><span> target_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> relation</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;"> Relation</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Slice</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> });</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is useful for views such as slices or sub-regions.</p>
<p>The interpretation should be conservative:</p>
<blockquote>
<p>This allocation looks like a view into another allocation.</p>
</blockquote>
<p>Not:</p>
<blockquote>
<p>This is definitely a Rust slice with exact lifetime semantics.</p>
</blockquote>
<hr />
<h2 id="7-clone-detection">7. Clone Detection</h2>
<p>Clone detection groups allocations and compares content similarity within a time window.</p>
<p>The configuration includes:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> CloneConfig</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> max_time_diff_ns</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> compare_bytes</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> min_similarity</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> f64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> min_similarity_no_stack_hash</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> f64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> max_clone_edges_per_node</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> detect_smart_pointers</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> bool</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> arc_threshold</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> f64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> rc_threshold</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> f64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The module comments describe the grouping strategy as:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>(type, size, stack_hash) + sliding time window + content similarity</span></span></code></pre>
<p>This is explicitly probabilistic. It reduces false positives with thresholds, but it is not a compiler-level clone hook.</p>
<hr />
<h2 id="8-container-detection">8. Container Detection</h2>
<p>Containers such as <code>HashMap</code>, <code>BTreeMap</code>, and <code>VecDeque</code> often do not expose a stable user-level heap pointer.</p>
<p><code>memscope-rs</code> tracks them as <code>TrackKind::Container</code> metadata, then infers <code>Contains</code> relations using temporal locality and filters.</p>
<p>The container detector algorithm is described in the source as:</p>
<ol>
<li>filter allocations into containers and heap owners;</li>
<li>for each container, examine subsequent heap owners within a time window;</li>
<li>apply thread affinity and size-ratio filters;</li>
<li>add <code>Contains</code> edges for candidates.</li>
</ol>
<p>Simplified:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">for</span><span> container</span><span style="color: #CBA6F7;"> in</span><span> containers</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> candidate</span><span style="color: #CBA6F7;"> in</span><span style="color: #89B4FA;font-style: italic;"> heap_owners_after</span><span style="color: #9399B2;">(</span><span>container</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> container</span><span style="color: #94E2D5;">.</span><span>thread_id </span><span style="color: #94E2D5;">!=</span><span> candidate</span><span style="color: #94E2D5;">.</span><span>thread_id </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> continue</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> time_diff</span><span style="color: #94E2D5;"> ></span><span> config</span><span style="color: #94E2D5;">.</span><span>time_window_ns </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> break</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> candidate</span><span style="color: #94E2D5;">.</span><span>size</span><span style="color: #94E2D5;"> ></span><span> container</span><span style="color: #94E2D5;">.</span><span>size </span><span style="color: #94E2D5;">*</span><span> config</span><span style="color: #94E2D5;">.</span><span>size_ratio </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> continue</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> edges</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">RelationEdge</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> from</span><span style="color: #94E2D5;">:</span><span> container_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> to</span><span style="color: #94E2D5;">:</span><span> candidate_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> relation</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;"> Relation</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Contains</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> });</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is one of the places where the tool is intentionally honest: <code>Contains</code> is inferred from metadata and timing, not directly proven from Rust internals.</p>
<hr />
<h2 id="9-shared-ownership-detection">9. Shared Ownership Detection</h2>
<p>Shared ownership detection has two strategies.</p>
<p>The first strategy looks for multiple owner edges into a target that looks like <code>Arc/Rc</code>-like data.</p>
<p>The second strategy is more important for the newer <code>StackOwner</code> model:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> stack_owners</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">(</span><span style="color: #CBA6F7;">usize</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)</span><span style="color: #89DCEB;">></span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #9399B2;"> (</span><span>i</span><span style="color: #9399B2;">,</span><span> record</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span> records</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">enumerate</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>stack_ptr</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span> record</span><span style="color: #94E2D5;">.</span><span>stack_ptr </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> stack_ptr</span><span style="color: #94E2D5;"> ></span><span style="color: #FAB387;"> 0x1000</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> stack_owners</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">((</span><span>i</span><span style="color: #9399B2;">,</span><span> record</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">ptr</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> heap_to_records</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> HashMap</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">>></span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> HashMap</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #9399B2;"> (</span><span>record_id</span><span style="color: #9399B2;">,</span><span> heap_ptr</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span> stack_owners</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> heap_to_records</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">entry</span><span style="color: #9399B2;">(</span><span>heap_ptr</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">or_default</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span>record_id</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #9399B2;"> (</span><span>_heap_ptr</span><span style="color: #9399B2;">,</span><span> record_ids</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span> heap_to_records</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> record_ids</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">len</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 2</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> // emit ArcClone edges</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is the same principle explained in the <code>StackOwner</code> article:</p>
<blockquote>
<p>multiple tracked smart pointer values pointing to the same heap data imply observed shared ownership.</p>
</blockquote>
<hr />
<h2 id="10-variable-evolution">10. Variable Evolution</h2>
<p>The builder also has a variable evolution step.</p>
<p>The source comment describes it as:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>For allocations with the same variable name, infer evolution relationships indicating that the same variable was tracked multiple times.</span></span></code></pre>
<p>Examples include:</p>
<ul>
<li>a growing <code>Vec</code>;</li>
<li>a reallocated buffer;</li>
<li>a container tracked multiple times over time.</li>
</ul>
<p>This is useful for timeline-style interpretation, but it should be described as variable evolution inference, not exact move semantics.</p>
<hr />
<h2 id="11-relationgraph-vs-ownershipgraph">11. RelationGraph vs OwnershipGraph</h2>
<p>There are two related but distinct graph concepts.</p>
<h3 id="relationgraph"><code>RelationGraph</code></h3>
<p><code>RelationGraph</code> is the direct output of relation inference:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> RelationGraph</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> edges</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">RelationEdge</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>It is centered on inferred relationships between allocation records.</p>
<h3 id="ownershipgraph"><code>OwnershipGraph</code></h3>
<p><code>OwnershipGraph</code> is a higher-level post-analysis graph:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> Node</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> id</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> ObjectId</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> type_name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> stack_ptr</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Owns</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Contains</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Borrows</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> RcClone</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> ArcClone</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Move</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> SharedBorrow</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> MutBorrow</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The ownership graph is designed for diagnostics and visualization. It can report clone edges, cycles, and possible clone storms.</p>
<hr />
<h2 id="12-exporting-the-ownership-graph">12. Exporting the Ownership Graph</h2>
<p>The render/export layer builds an ownership graph from allocation records and event store data.</p>
<p>The export includes:</p>
<ul>
<li>nodes;</li>
<li>edges;</li>
<li>cycles;</li>
<li>diagnostics;</li>
<li>borrow history.</li>
</ul>
<p>Node export looks like:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "id"</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"0x</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:x</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> node</span><span style="color: #94E2D5;">.</span><span>id</span><span style="color: #94E2D5;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "type_name"</span><span style="color: #94E2D5;">:</span><span> node</span><span style="color: #94E2D5;">.</span><span>type_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "size"</span><span style="color: #94E2D5;">:</span><span> node</span><span style="color: #94E2D5;">.</span><span>size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "stack_ptr"</span><span style="color: #94E2D5;">:</span><span> node</span><span style="color: #94E2D5;">.</span><span>stack_ptr</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>p</span><span style="color: #94E2D5;">|</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"0x</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:x</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> p</span><span style="color: #9399B2;">)),</span></span>
<span class="giallo-l"><span style="color: #9399B2;">})</span></span></code></pre>
<p>Edge export maps internal edge kinds to strings:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #A6E3A1;">"kind"</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> match</span><span> edge</span><span style="color: #94E2D5;">.</span><span>op </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Owns</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "Owns"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Contains</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "Contains"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Borrows</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "Borrows"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">RcClone</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "RcClone"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">ArcClone</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "ArcClone"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Move</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "Move"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">SharedBorrow</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "SharedBorrow"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> EdgeKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">MutBorrow</span><span style="color: #94E2D5;"> =></span><span style="color: #A6E3A1;"> "MutBorrow"</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is what makes the graph consumable by dashboards and JSON-based tooling.</p>
<hr />
<h2 id="13-a-practical-view-of-confidence">13. A Practical View of Confidence</h2>
<p>Not all graph edges should be interpreted equally.</p>
<p>High-confidence inputs:</p>
<ul>
<li>pointer addresses from allocation tracking;</li>
<li>allocation sizes;</li>
<li>explicit <code>track!</code> metadata;</li>
<li><code>StackOwner.stack_ptr</code> and <code>heap_ptr</code> when tracked;</li>
<li>explicit memory passport lifecycle events.</li>
</ul>
<p>Medium-confidence relations:</p>
<ul>
<li><code>ArcClone</code> from multiple <code>StackOwner</code> records sharing a <code>heap_ptr</code>;</li>
<li><code>Contains</code> from container metadata and temporal locality;</li>
<li><code>Slice</code> from pointer-in-range detection.</li>
</ul>
<p>Heuristic relations:</p>
<ul>
<li><code>Clone</code> from content similarity;</li>
<li><code>Owns</code> from memory pointer scanning;</li>
<li>variable <code>Evolution</code> from repeated variable names;</li>
<li>borrow/move approximations.</li>
</ul>
<p>The graph is most useful when these confidence levels are understood.</p>
<hr />
<h2 id="14-what-this-can-do">14. What This Can Do</h2>
<p>The relation and ownership graph layers can help answer:</p>
<ul>
<li>Which allocations appear connected?</li>
<li>Which tracked smart pointers share heap data?</li>
<li>Which containers likely own or group nearby heap allocations?</li>
<li>Which allocations look like slices or sub-regions?</li>
<li>Which allocations appear cloned?</li>
<li>Are there graph cycles?</li>
<li>Are there many <code>ArcClone</code> edges suggesting a clone storm candidate?</li>
</ul>
<p>This is valuable for debugging and exploration.</p>
<hr />
<h2 id="15-what-this-cannot-prove">15. What This Cannot Prove</h2>
<p>The graph cannot prove:</p>
<ul>
<li>full Rust ownership correctness;</li>
<li>exact borrow lifetimes;</li>
<li>exact move semantics;</li>
<li>every clone call site;</li>
<li>all container internals;</li>
<li>semantic ownership across untracked variables;</li>
<li>correctness of unsafe or foreign code.</li>
</ul>
<p>It is a runtime-informed analysis graph, not a Rust compiler or formal verifier.</p>
<hr />
<h2 id="16-summary">16. Summary</h2>
<p>The ownership graph layer is where <code>memscope-rs</code> starts to connect raw memory behavior back to Rust semantics.</p>
<p>It combines:</p>
<ul>
<li>allocation records;</li>
<li>heap scanning;</li>
<li>type inference;</li>
<li>range maps;</li>
<li><code>TrackKind</code> metadata;</li>
<li><code>StackOwner</code> metadata;</li>
<li>container metadata;</li>
<li>memory passport events;</li>
<li>relation detectors.</li>
</ul>
<p>The most accurate description is:</p>
<blockquote>
<p><code>memscope-rs</code> builds an explainable, runtime-informed relationship graph. It is not a compiler proof, but it can make memory structure visible enough to debug real Rust programs.</p>
</blockquote>
memscope-rs Technical Article Series Index
2026-05-21T00:00:00+00:00
2026-05-21T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/article-series-index/
<h1 id="memscope-rs-technical-article-series-index">memscope-rs Technical Article Series Index</h1>
<p>This repository contains a set of pragmatic, source-driven articles that decompose <code>memscope-rs</code> by module.</p>
<p>The goal of the series is not to over-market the project, but to explain what the tool actually tracks, how it collects data, where inference starts, and what the current limitations are.</p>
<h2 id="suggested-reading-order">Suggested Reading Order</h2>
<ol>
<li>
<p><code>article-single-thread-tracking.md</code></p>
<ul>
<li><code>GlobalAlloc</code></li>
<li><code>track!</code></li>
<li><code>Trackable</code></li>
<li><code>TrackKind</code></li>
<li>standard and custom Rust types</li>
</ul>
</li>
<li>
<p><code>article-lockfree-multithread-tracking.md</code></p>
<ul>
<li>concurrent event capture</li>
<li><code>SegQueue</code></li>
<li><code>DashMap</code></li>
<li>atomics</li>
<li>CAS peak memory tracking</li>
</ul>
</li>
<li>
<p><code>article-unsafe-ffi-memory-passport.md</code></p>
<ul>
<li>unsafe allocation</li>
<li>FFI allocation</li>
<li>memory passport</li>
<li>handover/free lifecycle</li>
<li>audit boundaries</li>
</ul>
</li>
<li>
<p><code>article-arc-rc-stackowner.md</code></p>
<ul>
<li><code>Arc/Rc</code> shared ownership</li>
<li><code>StackOwner</code></li>
<li><code>stack_ptr</code></li>
<li><code>heap_ptr</code></li>
<li>clone/share relation inference</li>
</ul>
</li>
<li>
<p><code>article-async-task-memory-attribution.md</code></p>
<ul>
<li>async task context</li>
<li><code>AsyncTracker</code></li>
<li><code>spawn_tracked</code></li>
<li>task-level memory attribution</li>
<li>zombie task candidates</li>
</ul>
</li>
<li>
<p><code>article-ownership-graph-relation-inference.md</code></p>
<ul>
<li>relation graph</li>
<li>ownership graph</li>
<li>owner/slice/clone/container/shared detectors</li>
<li>confidence boundaries</li>
</ul>
</li>
<li>
<p><code>article-architecture-facade-pipeline.md</code></p>
<ul>
<li><code>MemScope</code> facade</li>
<li>capture engine</li>
<li>event store</li>
<li>snapshot engine</li>
<li>render engine</li>
</ul>
</li>
<li>
<p><code>article-export-dashboard-pipeline.md</code></p>
<ul>
<li>JSON export</li>
<li>dashboard rendering</li>
<li>memory passports</li>
<li>async reports</li>
<li>ownership graph export</li>
</ul>
</li>
<li>
<p><code>article-benchmark-deep-dive.md</code></p>
<ul>
<li>benchmark coverage</li>
<li>raw benchmark numbers</li>
<li>regressions</li>
<li>limitations of the benchmark suite</li>
</ul>
</li>
</ol>
<h2 id="editorial-principle">Editorial Principle</h2>
<p>Each article should preserve three distinctions:</p>
<ul>
<li>Runtime facts: data captured directly from allocation events or explicit API calls.</li>
<li>Explicit metadata: variable names, source locations, task IDs, <code>TrackKind</code>, and passports.</li>
<li>Inference: ownership, containment, clone, borrow, move, leak, or cycle relationships derived during analysis.</li>
</ul>
<p>This distinction is central to presenting <code>memscope-rs</code> honestly.</p>
Arc/Rc Clone Detection: Why `StackOwner` Matters
2026-05-20T00:00:00+00:00
2026-05-20T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/arc-rc-stackowner/
<h1 id="arc-rc-clone-detection-why-stackowner-matters">Arc/Rc Clone Detection: Why <code>StackOwner</code> Matters</h1>
<p><code>Arc<T></code> and <code>Rc<T></code> are safe Rust abstractions, but they make ownership harder to observe.</p>
<p>Cloning an <code>Arc</code> or <code>Rc</code> usually does not allocate a new user object. It creates another smart pointer that shares ownership of the same heap data.</p>
<p>From a Rust perspective, ownership fan-out changed. From a raw allocator perspective, very little may have happened.</p>
<p><code>memscope-rs</code> addresses this with the <code>StackOwner</code> model.</p>
<hr />
<h2 id="1-why-arc-rc-are-hard-to-observe">1. Why Arc/Rc Are Hard to Observe</h2>
<p>Consider:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> root</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">vec!</span><span style="color: #9399B2;">[</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">,</span><span style="color: #FAB387;"> 2</span><span style="color: #9399B2;">,</span><span style="color: #FAB387;"> 3</span><span style="color: #9399B2;">]);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> worker_a</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>root</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> worker_b</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>root</span><span style="color: #9399B2;">);</span></span></code></pre>
<p>Allocator-level tracking may only see the original heap allocation. But logically there are now three smart pointer values sharing the same data.</p>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[Arc::new T] --> H[Heap Data]
B[Arc::clone] --> C[New Stack Owner]
C --> H
G[GlobalAlloc] --> H
G -. "cannot directly see" .-> B
</div>
<p>The question is not just "where was memory allocated?" but:</p>
<blockquote>
<p>Which tracked smart pointers point to the same heap object?</p>
</blockquote>
<hr />
<h2 id="2-the-key-insight-arc-rc-are-stack-owners">2. The Key Insight: Arc/Rc Are Stack Owners</h2>
<p><code>Arc<T></code> and <code>Rc<T></code> are smart pointer values. The smart pointer value has its own address, and it points to heap data.</p>
<p><code>memscope-rs</code> models this as:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;">TrackKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">StackOwner</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #94E2D5;">:</span><span> stack_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> heap_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The full <code>TrackKind</code> enum includes:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> HeapOwner</span><span style="color: #9399B2;"> {</span><span> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Container</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Value</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> StackOwner</span><span style="color: #9399B2;"> {</span><span> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span> heap_ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The important distinction is:</p>
<ul>
<li><code>stack_ptr</code> identifies the smart pointer value;</li>
<li><code>heap_ptr</code> identifies the shared heap data.</li>
</ul>
<hr />
<h2 id="3-arc-t-as-trackable">3. <code>Arc<T></code> as <code>Trackable</code></h2>
<p>The <code>Arc<T></code> implementation records both pointers:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">impl</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #F9E2AF;font-style: italic;"> Trackable</span><span style="color: #CBA6F7;"> for</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">sync</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> track_kind</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> stack_ptr</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #CBA6F7;"> as</span><span style="color: #94E2D5;"> *</span><span style="color: #CBA6F7;">const</span><span> _</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> heap_ptr</span><span style="color: #94E2D5;"> = &**</span><span style="color: #F38BA8;">self</span><span style="color: #CBA6F7;"> as</span><span style="color: #94E2D5;"> *</span><span style="color: #CBA6F7;">const</span><span style="color: #FAB387;"> T</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">StackOwner</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #94E2D5;">:</span><span> stack_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> heap_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> size</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">mem</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">size_of</span><span style="color: #94E2D5;">::</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_type_name</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> &</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #F9E2AF;font-style: italic;"> str</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "Arc<T>"</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_ref_count</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">sync</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Arc</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">strong_count</span><span style="color: #9399B2;">(</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">))</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p><code>Rc<T></code> uses the same model:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">impl</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #F9E2AF;font-style: italic;"> Trackable</span><span style="color: #CBA6F7;"> for</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">rc</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Rc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> track_kind</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> stack_ptr</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #CBA6F7;"> as</span><span style="color: #94E2D5;"> *</span><span style="color: #CBA6F7;">const</span><span> _</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> heap_ptr</span><span style="color: #94E2D5;"> = &**</span><span style="color: #F38BA8;">self</span><span style="color: #CBA6F7;"> as</span><span style="color: #94E2D5;"> *</span><span style="color: #CBA6F7;">const</span><span style="color: #FAB387;"> T</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">StackOwner</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #94E2D5;">:</span><span> stack_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> heap_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> size</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">mem</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">size_of</span><span style="color: #94E2D5;">::</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_ref_count</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">rc</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Rc</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">strong_count</span><span style="color: #9399B2;">(</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">))</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This does not read private <code>ArcInner</code> or <code>RcBox</code> layout. It uses the smart pointer’s observable relationship to the data it dereferences.</p>
<hr />
<h2 id="4-event-recording-for-stackowner">4. Event Recording for <code>StackOwner</code></h2>
<p>When <code>track!</code> sees a <code>StackOwner</code>, it records the heap pointer as the event pointer and stores the stack pointer in metadata:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;">TrackKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">StackOwner</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #94E2D5;">:</span><span> stack_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> heap_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span><span style="color: #94E2D5;"> =></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>inner</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">track_allocation</span><span style="color: #9399B2;">(</span><span>stack_ptr</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let mut</span><span> event</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> MemoryEvent</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">allocate</span><span style="color: #9399B2;">(</span><span>heap_ptr</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;">,</span><span> thread_id</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span> event</span><span style="color: #94E2D5;">.</span><span>var_name </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>name</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span> event</span><span style="color: #94E2D5;">.</span><span>type_name </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>type_name</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span> event</span><span style="color: #94E2D5;">.</span><span>source_file </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span> event</span><span style="color: #94E2D5;">.</span><span>source_line </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>line</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span> event</span><span style="color: #94E2D5;">.</span><span>module_path </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>module_path</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span> event</span><span style="color: #94E2D5;">.</span><span>stack_ptr </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>stack_ptr</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>event_store</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">record</span><span style="color: #9399B2;">(</span><span>event</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This gives the analysis layer enough information to see:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>root stack_ptr = S1, heap_ptr = H</span></span>
<span class="giallo-l"><span>worker_a stack_ptr = S2, heap_ptr = H</span></span>
<span class="giallo-l"><span>worker_b stack_ptr = S3, heap_ptr = H</span></span></code></pre><div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[root: Arc<T>] --> S1[stack_ptr S1]
B[worker_a: Arc<T>] --> S2[stack_ptr S2]
C[worker_b: Arc<T>] --> S3[stack_ptr S3]
S1 --> H[heap_ptr H]
S2 --> H
S3 --> H
H --> G[Same heap_ptr]
G --> R[Shared Ownership Group]
</div>
<hr />
<h2 id="5-detection-strategy-group-by-heap-ptr">5. Detection Strategy: Group by <code>heap_ptr</code></h2>
<p>The relation detector looks for records with <code>stack_ptr</code> metadata and groups them by heap pointer.</p>
<p>Simplified:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> stack_owners</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">(</span><span style="color: #CBA6F7;">usize</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)</span><span style="color: #89DCEB;">></span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #9399B2;"> (</span><span>i</span><span style="color: #9399B2;">,</span><span> record</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span> records</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">enumerate</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>stack_ptr</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span> record</span><span style="color: #94E2D5;">.</span><span>stack_ptr </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> stack_ptr</span><span style="color: #94E2D5;"> ></span><span style="color: #FAB387;"> 0x1000</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> stack_owners</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">((</span><span>i</span><span style="color: #9399B2;">,</span><span> record</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">ptr</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> heap_to_records</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> HashMap</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">>></span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> HashMap</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #9399B2;"> (</span><span>record_id</span><span style="color: #9399B2;">,</span><span> heap_ptr</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span> stack_owners</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> heap_to_records</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">entry</span><span style="color: #9399B2;">(</span><span>heap_ptr</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">or_default</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span>record_id</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">for</span><span style="color: #9399B2;"> (</span><span>_heap_ptr</span><span style="color: #9399B2;">,</span><span> record_ids</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span> heap_to_records</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> record_ids</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">len</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> >=</span><span style="color: #FAB387;"> 2</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> i</span><span style="color: #CBA6F7;"> in</span><span style="color: #FAB387;"> 0</span><span style="color: #94E2D5;">..</span><span>record_ids</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">len</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span> j</span><span style="color: #CBA6F7;"> in</span><span style="color: #9399B2;"> (</span><span>i</span><span style="color: #94E2D5;"> +</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">..</span><span>record_ids</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">len</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span> relations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">RelationEdge</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> from</span><span style="color: #94E2D5;">:</span><span> record_ids</span><span style="color: #9399B2;">[</span><span>i</span><span style="color: #9399B2;">],</span></span>
<span class="giallo-l"><span> to</span><span style="color: #94E2D5;">:</span><span> record_ids</span><span style="color: #9399B2;">[</span><span>j</span><span style="color: #9399B2;">],</span></span>
<span class="giallo-l"><span> relation</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;"> Relation</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">ArcClone</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> });</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre><div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart LR
A[Memory Events] --> B[Filter stack_ptr != None]
B --> C[Use event.ptr as heap_ptr]
C --> D[Group by heap_ptr]
D --> E{group size >= 2?}
E -->|yes| F[Emit ArcClone relation]
E -->|no| G[No shared clone relation]
</div>
<hr />
<h2 id="6-why-this-avoids-fragile-layout-assumptions">6. Why This Avoids Fragile Layout Assumptions</h2>
<p>Some approaches to smart pointer analysis might try to inspect internal layout:</p>
<ul>
<li>strong count offset;</li>
<li>weak count offset;</li>
<li>data pointer offset;</li>
<li>Rust-version-specific implementation details.</li>
</ul>
<p><code>memscope-rs</code> avoids that.</p>
<p>It does not need to know the internal layout of <code>ArcInner<T></code>. It only needs to observe that multiple tracked smart pointer values point to the same heap data.</p>
<p>This is a practical design choice, not a perfect compiler-level ownership trace.</p>
<hr />
<h2 id="7-example">7. Example</h2>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">use</span><span style="color: #F9E2AF;"> memscope_rs</span><span style="color: #94E2D5;">::</span><span style="color: #9399B2;">{</span><span>global_tracker</span><span style="color: #9399B2;">,</span><span> init_global_tracking</span><span style="color: #9399B2;">,</span><span> track</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> MemScopeResult</span><span style="color: #9399B2;">};</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">use</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">sync</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Arc</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> main</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> MemScopeResult</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">()</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> init_global_tracking</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> tracker</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> global_tracker</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> root</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">vec!</span><span style="color: #9399B2;">[</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">,</span><span style="color: #FAB387;"> 2</span><span style="color: #9399B2;">,</span><span style="color: #FAB387;"> 3</span><span style="color: #9399B2;">,</span><span style="color: #FAB387;"> 4</span><span style="color: #9399B2;">]);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> worker_a</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>root</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> worker_b</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>root</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> track!</span><span style="color: #9399B2;">(</span><span>tracker</span><span style="color: #9399B2;">,</span><span> root</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> track!</span><span style="color: #9399B2;">(</span><span>tracker</span><span style="color: #9399B2;">,</span><span> worker_a</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> track!</span><span style="color: #9399B2;">(</span><span>tracker</span><span style="color: #9399B2;">,</span><span> worker_b</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">export_json</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"MemoryAnalysis/arc_clone_demo"</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">export_html</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"MemoryAnalysis/arc_clone_demo"</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Ok</span><span style="color: #9399B2;">(())</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The important part is that each clone must be explicitly tracked if it should appear as a <code>StackOwner</code> record.</p>
<hr />
<h2 id="8-what-this-can-detect">8. What This Can Detect</h2>
<p>The current implementation can detect:</p>
<ul>
<li>multiple tracked <code>Arc<T></code> values pointing to the same heap data;</li>
<li>multiple tracked <code>Rc<T></code> values pointing to the same heap data;</li>
<li>observed shared ownership fan-out;</li>
<li><code>ArcClone</code> relation candidates in the relation graph;</li>
<li>a basis for later clone-storm or cycle analysis.</li>
</ul>
<p>The careful description is:</p>
<blockquote>
<p>It detects observed shared ownership among tracked <code>StackOwner</code> values.</p>
</blockquote>
<hr />
<h2 id="9-what-this-cannot-detect">9. What This Cannot Detect</h2>
<p>It cannot guarantee detection of:</p>
<ul>
<li>clones that were never passed to <code>track!</code>;</li>
<li>the exact call site of every clone;</li>
<li>every strong count change;</li>
<li>short-lived smart pointer values that were not tracked;</li>
<li>complete borrow/move semantics;</li>
<li>foreign or custom reference-counted objects.</li>
</ul>
<p>There is also a practical implementation note: a <code>track_clone!</code> macro exists, but the current main reliable path is <code>track!</code> → <code>StackOwner</code> → relation inference. The article should not present <code>track_clone!</code> as the primary clone detection path unless its event recording path is verified.</p>
<hr />
<h2 id="10-performance-claims">10. Performance Claims</h2>
<p>The clone relation itself is reconstructed during post-analysis by grouping records with the same heap pointer.</p>
<p>Runtime overhead mainly comes from explicitly tracking <code>Arc/Rc</code> variables as <code>StackOwner</code> events.</p>
<p>The benchmark suite contains general tracking benchmarks, but not a standalone benchmark isolating <code>StackOwner</code> grouping. Performance claims should therefore stay conservative.</p>
<hr />
<h2 id="11-summary">11. Summary</h2>
<p><code>StackOwner</code> matters because it maps shared ownership into observable runtime data:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>smart pointer value -> stack_ptr</span></span>
<span class="giallo-l"><span>shared heap object -> heap_ptr</span></span>
<span class="giallo-l"><span>multiple stack_ptrs -> same heap_ptr</span></span></code></pre><div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[track! arc_var] --> B[Trackable for Arc/Rc]
B --> C[TrackKind::StackOwner]
C --> D[MemoryEvent]
D --> E[event.ptr = heap_ptr]
D --> F[event.stack_ptr = stack_ptr]
E --> G[Relation Inference]
F --> G
G --> H[Group by heap_ptr]
H --> I[ArcClone relation]
</div>
<p>This is not perfect ownership tracing. It is a clear, explainable, and useful model for observing shared ownership in Rust programs.</p>
Async Task Memory Attribution: Tracking Memory Beyond Threads
2026-05-20T00:00:00+00:00
2026-05-20T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/async-task-memory-attribution/
<h1 id="async-task-memory-attribution-tracking-memory-beyond-threads">Async Task Memory Attribution: Tracking Memory Beyond Threads</h1>
<p>Rust async applications often run many logical tasks on a small number of runtime worker threads.</p>
<p>That makes thread-level memory tracking insufficient. A thread can execute many tasks, and a task may resume on different worker threads after an <code>.await</code>.</p>
<p>The more useful question is:</p>
<blockquote>
<p>Which async task should be responsible for this allocation?</p>
</blockquote>
<p><code>memscope-rs</code> provides an async attribution layer that associates allocations with logical task IDs where task context is available.</p>
<hr />
<h2 id="1-why-thread-level-tracking-fails-in-async-rust">1. Why Thread-Level Tracking Fails in Async Rust</h2>
<p>In a synchronous multithreaded program, a thread often maps reasonably well to a unit of work.</p>
<p>In async Rust, that assumption breaks down:</p>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
R[Tokio Runtime] --> T1[Worker Thread 1]
R --> T2[Worker Thread 2]
T1 --> A[Task A]
T1 --> B[Task B]
T2 --> A
T2 --> C[Task C]
A --> M1[Allocation]
B --> M2[Allocation]
C --> M3[Allocation]
M1 --> Q[Need task attribution]
M2 --> Q
M3 --> Q
</div>
<p>Thread ID tells us where an allocation ran. Task attribution tells us which logical async unit it belongs to.</p>
<hr />
<h2 id="2-trackercontext">2. <code>TrackerContext</code></h2>
<p>The async context structure records thread and task information:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> TrackerContext</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> thread_id</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> task_id</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">u64</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> tokio_task_id</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">u64</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Context capture attempts to read both memscope task context and Tokio task ID:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> capture</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> -></span><span style="color: #F38BA8;"> Self</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> task_id_from_context</span><span style="color: #94E2D5;"> =</span><span style="color: #FAB387;"> TASK_CONTEXT</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">try_with</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>ctx</span><span style="color: #94E2D5;">| *</span><span>ctx</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">ok</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">flatten</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> tokio_task_id</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;"> tokio</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">task</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">try_id</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">and_then</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>id</span><span style="color: #94E2D5;">|</span><span> id</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">parse</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">ok</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F38BA8;"> Self</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> thread_id</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> current_thread_id</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> task_id</span><span style="color: #94E2D5;">:</span><span> task_id_from_context</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">or</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">CURRENT_TASK_ID</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">with</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>cell</span><span style="color: #94E2D5;">|</span><span> cell</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">get</span><span style="color: #9399B2;">())),</span></span>
<span class="giallo-l"><span> tokio_task_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This provides the basic attribution context.</p>
<hr />
<h2 id="3-internal-task-ids">3. Internal Task IDs</h2>
<p><code>memscope-rs</code> uses its own task ID counter:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">static</span><span style="color: #FAB387;"> TASK_COUNTER</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicU64</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> AtomicU64</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> generate_unique_task_id</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #FAB387;"> TASK_COUNTER</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">fetch_add</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This matters because runtime task IDs may be recycled. Internal IDs are more stable for analysis and export.</p>
<hr />
<h2 id="4-context-storage">4. Context Storage</h2>
<p>The async tracker uses both thread-local and Tokio task-local context:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">thread_local!</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> static</span><span style="color: #FAB387;"> CURRENT_TASK_ID</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">cell</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Cell</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">u64</span><span style="color: #89DCEB;">>></span><span style="color: #94E2D5;"> =</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span style="color: #9399B2;"> {</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">cell</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Cell</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">None</span><span style="color: #9399B2;">) };</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F9E2AF;">tokio</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">task_local!</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> static</span><span style="color: #FAB387;"> TASK_CONTEXT</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">u64</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This design reflects an important truth: async task attribution requires explicit context management. It is not automatically inferred for every future in the program.</p>
<hr />
<h2 id="5-asynctracker">5. <code>AsyncTracker</code></h2>
<p>The core async tracker stores allocations, statistics, and task profiles:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> AsyncTracker</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> allocations</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Mutex</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">HashMap</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> AsyncAllocation</span><span style="color: #89DCEB;">>>></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> stats</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Mutex</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">AsyncStats</span><span style="color: #89DCEB;">>></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> profiles</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Mutex</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">HashMap</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">u64</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> TaskMemoryProfile</span><span style="color: #89DCEB;">>>></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> initialized</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Mutex</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">bool</span><span style="color: #89DCEB;">>></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is not a lock-free async tracker. It uses <code>Mutex<HashMap<...>></code> internally. The focus of this module is attribution, not zero-lock performance.</p>
<hr />
<h2 id="6-tracking-task-lifecycle">6. Tracking Task Lifecycle</h2>
<p>When a task starts, the tracker creates a profile and updates counters:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span>stats</span><span style="color: #94E2D5;">.</span><span>total_tasks </span><span style="color: #94E2D5;">+=</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span>stats</span><span style="color: #94E2D5;">.</span><span>active_tasks </span><span style="color: #94E2D5;">+=</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #F38BA8;">Self</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">set_current_task</span><span style="color: #9399B2;">(</span><span>task_id</span><span style="color: #9399B2;">);</span></span></code></pre>
<p>When it ends:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span>profile</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">mark_completed</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span>stats</span><span style="color: #94E2D5;">.</span><span>active_tasks </span><span style="color: #94E2D5;">=</span><span> stats</span><span style="color: #94E2D5;">.</span><span>active_tasks</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">saturating_sub</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F38BA8;">Self</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">clear_current_task</span><span style="color: #9399B2;">();</span></span></code></pre><div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
stateDiagram-v2
[*] --> Started
Started --> Running
Running --> Completed
Completed --> [*]
Running --> ZombieCandidate: started but not completed
</div>
<hr />
<h2 id="7-tracking-allocations-by-task">7. Tracking Allocations by Task</h2>
<p>Allocation attribution records pointer, size, task ID, and optional metadata:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> track_allocation_with_location</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> &</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> task_id</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> var_name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">String</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> type_name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">String</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> source_location</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">SourceLocation</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> allocation</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> AsyncAllocation</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> timestamp</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> Self</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">now</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> task_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> var_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> type_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> source_location</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> allocations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">insert</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #9399B2;">,</span><span> allocation</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>profile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span> profiles</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">get_mut</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>task_id</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span> profile</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">record_allocation</span><span style="color: #9399B2;">(</span><span>size</span><span style="color: #CBA6F7;"> as u64</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> stats</span><span style="color: #94E2D5;">.</span><span>total_allocations </span><span style="color: #94E2D5;">+=</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> stats</span><span style="color: #94E2D5;">.</span><span>total_memory </span><span style="color: #94E2D5;">+=</span><span> size</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> stats</span><span style="color: #94E2D5;">.</span><span>active_memory </span><span style="color: #94E2D5;">+=</span><span> size</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Deallocation removes the pointer and updates the relevant task profile:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span style="color: #9399B2;"> (</span><span>task_id</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span> allocations</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">remove</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>ptr</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>alloc</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;"> (</span><span>alloc</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">task_id</span><span style="color: #9399B2;">,</span><span> alloc</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">size</span><span style="color: #9399B2;">))</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">unwrap_or</span><span style="color: #9399B2;">((</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">,</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">if</span><span> task_id</span><span style="color: #94E2D5;"> !=</span><span style="color: #FAB387;"> 0</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>profile</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span> profiles</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">get_mut</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>task_id</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span> profile</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">record_deallocation</span><span style="color: #9399B2;">(</span><span>size</span><span style="color: #CBA6F7;"> as u64</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<hr />
<h2 id="8-globaltracker-integration">8. <code>GlobalTracker</code> Integration</h2>
<p><code>GlobalTracker::track_as()</code> first records the normal memory event, then tries to associate heap-owner allocations with the current async task:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #F38BA8;">self</span><span style="color: #94E2D5;">.</span><span>tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">track_as</span><span style="color: #9399B2;">(</span><span>var</span><span style="color: #9399B2;">,</span><span> name</span><span style="color: #9399B2;">,</span><span> file</span><span style="color: #9399B2;">,</span><span> line</span><span style="color: #9399B2;">,</span><span> module_path</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">if let</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>task_id</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> AsyncTracker</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">get_current_task</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> kind</span><span style="color: #94E2D5;"> =</span><span> var</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">track_kind</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if let</span><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">HeapOwner</span><span style="color: #9399B2;"> {</span><span> ptr</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;"> }</span><span style="color: #94E2D5;"> =</span><span> kind</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>async_tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">track_allocation_with_location</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> task_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>name</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">()),</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>type_name</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> None</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> );</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Important limitation: this path reads <code>CURRENT_TASK_ID</code>, while <code>spawn_tracked()</code> sets Tokio task-local <code>TASK_CONTEXT</code>. <code>TrackerContext::capture()</code> can see the task-local context, but automatic attribution through <code>track!</code> depends on the current integration path.</p>
<p>This should be described carefully.</p>
<hr />
<h2 id="9-spawn-tracked">9. <code>spawn_tracked</code></h2>
<p><code>spawn_tracked()</code> creates a memscope task ID and scopes a future with Tokio task-local context:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> spawn_tracked</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">F</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;">future</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> F</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;"> tokio</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">task</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">JoinHandle</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">F</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Output</span><span style="color: #89DCEB;">></span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">where</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> F</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Future</span><span style="color: #94E2D5;"> +</span><span style="color: #F9E2AF;font-style: italic;"> Send</span><span style="color: #94E2D5;"> +</span><span style="color: #9399B2;"> '</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> F</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Output</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Send</span><span style="color: #94E2D5;"> +</span><span style="color: #9399B2;"> '</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> task_id</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> generate_unique_task_id</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> tokio</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">spawn</span><span style="color: #9399B2;">(</span><span style="color: #CBA6F7;">async move</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #FAB387;"> TASK_CONTEXT</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">scope</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">Some</span><span style="color: #9399B2;">(</span><span>task_id</span><span style="color: #9399B2;">),</span><span> future</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">.</span><span style="color: #CBA6F7;">await</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Example usage:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> handles</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span>_</span><span style="color: #89DCEB;">></span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> (</span><span style="color: #FAB387;">0</span><span style="color: #94E2D5;">..</span><span style="color: #FAB387;">4</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>i</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> spawn_tracked</span><span style="color: #9399B2;">(</span><span style="color: #CBA6F7;">async move</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> tracker</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> global_tracker</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">unwrap</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> ctx</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;"> TrackerContext</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">capture</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> data</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> vec!</span><span style="color: #9399B2;">[</span><span style="color: #FAB387;">0</span><span style="color: #CBA6F7;">i32</span><span style="color: #9399B2;">;</span><span style="color: #FAB387;"> 100</span><span style="color: #9399B2;">];</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> track!</span><span style="color: #9399B2;">(</span><span>tracker</span><span style="color: #9399B2;">,</span><span> data</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> ctx</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">collect</span><span style="color: #9399B2;">();</span></span></code></pre>
<p>This is useful for context capture, but it should not be overstated as fully automatic attribution for every <code>track!</code> path.</p>
<hr />
<h2 id="10-track-in-tokio-task">10. <code>track_in_tokio_task</code></h2>
<p><code>track_in_tokio_task()</code> wraps a future with explicit lifecycle tracking:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub async fn</span><span style="color: #89B4FA;font-style: italic;"> track_in_tokio_task</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">F</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> T</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> future</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> F</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #9399B2;"> (</span><span style="color: #CBA6F7;">u64</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> T</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">where</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> F</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Future</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Output</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> T</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> unique_task_id</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> generate_unique_task_id</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> tokio_task_id</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;"> tokio</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">task</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">try_id</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">and_then</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>id</span><span style="color: #94E2D5;">|</span><span> id</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">parse</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">ok</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">track_task_start_with_tokio</span><span style="color: #9399B2;">(</span><span>unique_task_id</span><span style="color: #9399B2;">,</span><span> tokio_id</span><span style="color: #9399B2;">,</span><span> name</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">(),</span><span> thread_id</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> output</span><span style="color: #94E2D5;"> =</span><span> future</span><span style="color: #94E2D5;">.</span><span style="color: #CBA6F7;">await</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">track_task_end</span><span style="color: #9399B2;">(</span><span>unique_task_id</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #9399B2;"> (</span><span>unique_task_id</span><span style="color: #9399B2;">,</span><span> output</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This path is more explicit about start/end lifecycle tracking than plain <code>spawn_tracked()</code>.</p>
<hr />
<h2 id="11-zombie-task-detection">11. Zombie Task Detection</h2>
<p>The async tracker can detect tasks that were started but not completed:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> detect_zombie_tasks</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">u64</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> profiles</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>profiles</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">lock</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">unwrap</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> profiles</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">filter</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;">(</span><span>_</span><span style="color: #9399B2;">,</span><span> p</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">| !</span><span>p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">is_completed</span><span style="color: #9399B2;">())</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>id</span><span style="color: #9399B2;">,</span><span> _</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">|</span><span> id</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">collect</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This should be interpreted as zombie task candidates. A long-running task is not always a bug.</p>
<hr />
<h2 id="12-export-async-analysis-json">12. Export: <code>async_analysis.json</code></h2>
<p>The export pipeline writes async attribution data:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> async_data</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "summary"</span><span style="color: #94E2D5;">:</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_tasks"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>total_tasks</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "active_tasks"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>active_tasks</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_allocations"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>total_allocations</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_memory_bytes"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>total_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "active_memory_bytes"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>active_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "peak_memory_bytes"</span><span style="color: #94E2D5;">:</span><span> stats</span><span style="color: #94E2D5;">.</span><span>peak_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "task_profiles"</span><span style="color: #94E2D5;">:</span><span> profiles</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>p</span><span style="color: #94E2D5;">|</span><span style="color: #89B4FA;font-style: italic;"> json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "task_id"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">task_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "task_name"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">task_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "current_memory"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">current_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "peak_memory"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">peak_memory</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_allocations"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">total_allocations</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "total_deallocations"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">total_deallocations</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "is_completed"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">is_completed</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "has_potential_leak"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">has_potential_leak</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "allocations"</span><span style="color: #94E2D5;">:</span><span> snapshot</span><span style="color: #94E2D5;">.</span><span>allocations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">iter</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span>a</span><span style="color: #94E2D5;">|</span><span style="color: #89B4FA;font-style: italic;"> json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "ptr"</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"0x</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:x</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> a</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">ptr</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "size"</span><span style="color: #94E2D5;">:</span><span> a</span><span style="color: #94E2D5;">.</span><span>size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "task_id"</span><span style="color: #94E2D5;">:</span><span> a</span><span style="color: #94E2D5;">.</span><span>task_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "var_name"</span><span style="color: #94E2D5;">:</span><span> a</span><span style="color: #94E2D5;">.</span><span>var_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "type_name"</span><span style="color: #94E2D5;">:</span><span> a</span><span style="color: #94E2D5;">.</span><span>type_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> })),</span></span>
<span class="giallo-l"><span style="color: #9399B2;">});</span></span></code></pre>
<hr />
<h2 id="13-what-this-can-track">13. What This Can Track</h2>
<p>The async layer can track:</p>
<ul>
<li>internal task ID;</li>
<li>Tokio task ID as auxiliary data;</li>
<li>thread ID;</li>
<li>task start/end;</li>
<li>active task count;</li>
<li>task-level allocation count;</li>
<li>task-level current and peak memory;</li>
<li>allocation pointer and size;</li>
<li>variable name and type name;</li>
<li>zombie task candidates;</li>
<li><code>async_analysis.json</code> export.</li>
</ul>
<hr />
<h2 id="14-limitations">14. Limitations</h2>
<p>The async layer cannot guarantee:</p>
<ul>
<li>automatic tracking of every Tokio task;</li>
<li>attribution for all third-party-library allocations;</li>
<li>exact memory changes at every <code>.await</code> point;</li>
<li>full future lifecycle reconstruction;</li>
<li>automatic bridging between all task-local and thread-local context paths;</li>
<li>lock-free async attribution.</li>
</ul>
<p>The best description is:</p>
<blockquote>
<p><code>memscope-rs</code> provides explicit async task attribution, not a fully automatic Tokio runtime profiler.</p>
</blockquote>
<hr />
<h2 id="15-summary">15. Summary</h2>
<p>Async attribution matters because Rust services are increasingly task-oriented rather than thread-oriented.</p>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart LR
A[task_scope / tracked future] --> B[Task Context]
B --> C[task_id]
C --> D[track allocation]
D --> E[AsyncTracker]
E --> F[TaskMemoryProfile]
F --> G[async_analysis.json]
</div>
<p><code>memscope-rs</code> does not magically understand all async execution. It gives developers a way to attach memory events to logical tasks when context is available.</p>
<p>That is a practical step beyond thread-level memory tracking.</p>
Lockfree Tracking: How `memscope-rs` Handles Concurrent Memory Events
2026-05-20T00:00:00+00:00
2026-05-20T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/lockfree-multithread-tracking/
<h1 id="lockfree-tracking-how-memscope-rs-handles-concurrent-memory-events">Lockfree Tracking: How <code>memscope-rs</code> Handles Concurrent Memory Events</h1>
<p>Single-thread memory tracking answers the first question: can we collect useful allocation data?</p>
<p>Multithreaded tracking asks a harder one:</p>
<blockquote>
<p>Can the tool keep collecting useful memory data while many threads allocate and free memory at the same time?</p>
</blockquote>
<p>In a real Rust service, allocation events do not arrive politely in sequence. Multiple threads may allocate concurrently, update statistics, drop values, and trigger snapshots or exports.</p>
<p>This article explains how the multithreaded path in <code>memscope-rs</code> is structured, what it actually does, and where its limits are.</p>
<hr />
<h2 id="1-why-multithread-tracking-is-different">1. Why Multithread Tracking Is Different</h2>
<p>In a single-threaded flow, event order is relatively easy to reason about.</p>
<p>In a multithreaded flow:</p>
<ul>
<li>allocation events can arrive concurrently;</li>
<li>deallocations can race with snapshots;</li>
<li>active allocation maps need concurrent updates;</li>
<li>statistics must avoid becoming a global bottleneck;</li>
<li>the profiler itself should not dominate application contention.</li>
</ul>
<p>The design uses separate structures for separate responsibilities:</p>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[Concurrent Allocation Events] --> B[Lock-free Event Queue]
A --> C[Concurrent Allocation Map]
A --> D[Atomic Counters]
B --> E[Snapshot / Export]
C --> F[Active Allocation Lookup]
D --> G[Stats / Peak Memory]
E --> H[Analysis / Report]
F --> H
G --> H
</div>
<hr />
<h2 id="2-threadlocaltracker">2. <code>ThreadLocalTracker</code></h2>
<p>The core multithreaded structure is <code>ThreadLocalTracker</code>.</p>
<p>Its important fields include:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> ThreadLocalTracker</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> thread_id</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> ThreadId</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> events</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">SegQueue</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">Event</span><span style="color: #89DCEB;">>></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> active_allocations</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Arc</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">DashMap</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #9399B2;">,</span><span style="color: #CBA6F7;"> usize</span><span style="color: #89DCEB;">>></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> total_allocations</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicU64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> total_allocated</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicU64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> total_deallocations</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicU64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> total_deallocated</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicU64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> active_memory</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicU64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> peak_memory</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicU64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> sample_rate</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> f64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> total_seen</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicUsize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> total_tracked</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> AtomicUsize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The important point is separation of responsibilities:</p>
<ul>
<li><code>SegQueue</code> stores events.</li>
<li><code>DashMap</code> tracks active allocations.</li>
<li>atomics track counters.</li>
<li>CAS logic maintains peak memory.</li>
</ul>
<p>This avoids one large global mutex around the entire tracking pipeline.</p>
<hr />
<h2 id="3-recording-allocation-events">3. Recording Allocation Events</h2>
<p>The allocation path is structured roughly like this:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> track_allocation</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> call_stack_hash</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>total_seen</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">fetch_add</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>sample_rate</span><span style="color: #94E2D5;"> <</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">.</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> sample_decision</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;"> rand</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">random</span><span style="color: #94E2D5;">::</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">f64</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> sample_decision</span><span style="color: #94E2D5;"> >=</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>sample_rate </span><span style="color: #9399B2;">{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> return</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>total_tracked</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">fetch_add</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> event</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Event</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">allocation</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;">,</span><span> call_stack_hash</span><span style="color: #9399B2;">,</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">thread_id</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>events</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span>event</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>active_allocations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">insert</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>total_allocations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">fetch_add</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>total_allocated</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">fetch_add</span><span style="color: #9399B2;">(</span><span>size</span><span style="color: #CBA6F7;"> as u64</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The design is pragmatic:</p>
<ul>
<li>event recording avoids a global event mutex;</li>
<li>active allocations use a concurrent map;</li>
<li>statistics use atomic counters;</li>
<li>sampling can reduce overhead when full fidelity is not required.</li>
</ul>
<hr />
<h2 id="4-peak-memory-with-cas-and-backoff">4. Peak Memory with CAS and Backoff</h2>
<p>Peak memory is a shared statistic and can become a contention point.</p>
<p><code>memscope-rs</code> updates it with a CAS loop and progressive backoff:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> new_active</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>active_memory</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">fetch_add</span><span style="color: #9399B2;">(</span><span>size</span><span style="color: #CBA6F7;"> as u64</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> +</span><span> size</span><span style="color: #CBA6F7;"> as u64</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> current_peak</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>peak_memory</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">load</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> backoff_count</span><span style="color: #94E2D5;"> =</span><span style="color: #FAB387;"> 0</span><span style="color: #CBA6F7;">u32</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">while</span><span> new_active</span><span style="color: #94E2D5;"> ></span><span> current_peak</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> match</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>peak_memory</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">compare_exchange_weak</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span> current_peak</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> new_active</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> ) {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Ok</span><span style="color: #9399B2;">(</span><span>_</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =></span><span style="color: #CBA6F7;"> break</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Err</span><span style="color: #9399B2;">(</span><span>actual</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> current_peak</span><span style="color: #94E2D5;"> =</span><span> actual</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span> backoff_count</span><span style="color: #94E2D5;"> +=</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span> backoff_count</span><span style="color: #94E2D5;"> <</span><span style="color: #FAB387;"> 10</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">hint</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">spin_loop</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span><span style="color: #CBA6F7;"> else if</span><span> backoff_count</span><span style="color: #94E2D5;"> <</span><span style="color: #FAB387;"> 20</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">thread</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">yield_now</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span><span style="color: #CBA6F7;"> else</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">thread</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">sleep</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;">std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">time</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">Duration</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">from_micros</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">));</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is not academic "everything is lock-free" purity. It is an engineering tradeoff: keep hot updates lightweight, but avoid wasting CPU under contention.</p>
<hr />
<h2 id="5-deallocation-and-suspicious-frees">5. Deallocation and Suspicious Frees</h2>
<p>Deallocation removes a pointer from the active map:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> track_deallocation</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> call_stack_hash</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> size</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span>active_allocations</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">remove</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span>ptr</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">map</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">|</span><span style="color: #9399B2;">(</span><span>_</span><span style="color: #9399B2;">,</span><span> v</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">|</span><span> v</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> .</span><span style="color: #89B4FA;font-style: italic;">unwrap_or</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">0</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> event</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Event</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">deallocation</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;">,</span><span> call_stack_hash</span><span style="color: #9399B2;">,</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;">thread_id</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>events</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">push</span><span style="color: #9399B2;">(</span><span>event</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>total_deallocations</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">fetch_add</span><span style="color: #9399B2;">(</span><span style="color: #FAB387;">1</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span>total_deallocated</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">fetch_add</span><span style="color: #9399B2;">(</span><span>size</span><span style="color: #CBA6F7;"> as u64</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;"> Ordering</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Relaxed</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>If the pointer is missing, that can indicate an untracked allocation, double free candidate, or another mismatch. It should be treated as a suspicious signal, not an automatic proof of memory corruption.</p>
<hr />
<h2 id="6-backend-strategy">6. Backend Strategy</h2>
<p>The capture backend abstraction exposes a common interface:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub trait</span><span style="color: #F9E2AF;font-style: italic;"> CaptureBackend</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Send</span><span style="color: #94E2D5;"> +</span><span style="color: #F9E2AF;font-style: italic;"> Sync</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> capture_alloc</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> thread_id</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> MemoryEvent</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> capture_dealloc</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> thread_id</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> MemoryEvent</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> capture_realloc</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> old_size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> new_size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> thread_id</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> MemoryEvent</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> capture_move</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> from_ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> to_ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> thread_id</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> MemoryEvent</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Available backend types include:</p>
<ul>
<li><code>Core</code></li>
<li><code>Lockfree</code></li>
<li><code>Async</code></li>
<li><code>Unified</code></li>
</ul>
<p>The <code>Unified</code> backend currently chooses simply:</p>
<ul>
<li>single core or unavailable parallelism → <code>Core</code></li>
<li>multiple cores → <code>Lockfree</code></li>
</ul>
<p>It is useful, but it should not be described as a sophisticated adaptive runtime scheduler.</p>
<hr />
<h2 id="7-benchmark-interpretation">7. Benchmark Interpretation</h2>
<p>The benchmark log shows concurrent tracking scales upward with thread count, but not linearly.</p>
<table><thead><tr><th>Threads</th><th style="text-align: right">Approximate Time</th></tr></thead><tbody>
<tr><td>1</td><td style="text-align: right">~19.17 µs</td></tr>
<tr><td>2</td><td style="text-align: right">~40.60 µs</td></tr>
<tr><td>4</td><td style="text-align: right">~55.30 µs</td></tr>
<tr><td>8</td><td style="text-align: right">~134.74 µs</td></tr>
<tr><td>16</td><td style="text-align: right">~372.96 µs</td></tr>
<tr><td>32</td><td style="text-align: right">~961.58 µs</td></tr>
<tr><td>64</td><td style="text-align: right">~1.86 ms</td></tr>
<tr><td>128</td><td style="text-align: right">~4.67 ms</td></tr>
</tbody></table>
<p>Shared-tracker scenarios also show clear concurrency cost:</p>
<table><thead><tr><th>Threads</th><th style="text-align: right">Approximate Time</th></tr></thead><tbody>
<tr><td>1</td><td style="text-align: right">~98.30 µs</td></tr>
<tr><td>8</td><td style="text-align: right">~924.19 µs</td></tr>
<tr><td>16</td><td style="text-align: right">~1.84 ms</td></tr>
<tr><td>64</td><td style="text-align: right">~7.06 ms</td></tr>
</tbody></table>
<p>The honest interpretation:</p>
<ul>
<li>the design works under concurrency;</li>
<li>contention still exists;</li>
<li>shared state is not free;</li>
<li>the benchmark log contains both improvements and regressions.</li>
</ul>
<hr />
<h2 id="8-what-makes-this-design-worth-writing-about">8. What Makes This Design Worth Writing About</h2>
<h3 id="lock-free-ish-event-capture">Lock-free-ish event capture</h3>
<p>The implementation is not fully lock-free everywhere. The accurate description is that hot paths avoid heavyweight global locks where possible:</p>
<ul>
<li><code>SegQueue</code> for event ingestion;</li>
<li><code>DashMap</code> for active allocation lookup;</li>
<li>atomics for counters;</li>
<li>CAS for peak memory.</li>
</ul>
<h3 id="separation-of-concerns">Separation of concerns</h3>
<p>The tracking path separates:</p>
<ul>
<li>event writing;</li>
<li>active allocation management;</li>
<li>statistics;</li>
<li>snapshot/export.</li>
</ul>
<p>This makes the architecture easier to reason about and extend.</p>
<h3 id="honest-performance-envelope">Honest performance envelope</h3>
<p>The benchmark data is useful because it is not uniformly positive. Some paths improve, some regress, and some remain close to noise. That is a credible measurement record, not just a promotional claim.</p>
<hr />
<h2 id="9-limitations">9. Limitations</h2>
<ul>
<li>Sampling means the trace may not be complete.</li>
<li><code>call_stack_hash</code> is a lightweight signal, not a full symbolic backtrace.</li>
<li>Contention is reduced, not eliminated.</li>
<li>Shared tracker benchmarks are stress scenarios, not proof of ideal scalability.</li>
<li>Performance should be measured under real application workloads.</li>
</ul>
<hr />
<h2 id="10-summary">10. Summary</h2>
<p>The multithreaded design answers a different question from single-thread tracking:</p>
<blockquote>
<p>When many threads write memory events at once, can the tool still collect useful data without becoming the main bottleneck?</p>
</blockquote>
<p><code>memscope-rs</code> approaches this with <code>SegQueue</code>, <code>DashMap</code>, atomics, and CAS-based statistics. It is not zero-cost, but it is a practical concurrent tracking architecture.</p>
Single-Thread Tracking: From GlobalAlloc to Rust-Level Memory Semantics
2026-05-20T00:00:00+00:00
2026-05-20T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/single-thread-tracking/
<h1 id="single-thread-tracking-from-globalalloc-to-rust-level-memory-semantics">Single-Thread Tracking: From <code>GlobalAlloc</code> to Rust-Level Memory Semantics</h1>
<p>Rust gives us strong memory safety guarantees, but memory safety does not automatically mean memory behavior is transparent.</p>
<p>In real Rust applications, developers still need answers to questions like:</p>
<ul>
<li>Which variable caused this allocation?</li>
<li>Which Rust type owns this memory?</li>
<li>Was this object a real heap owner, a container, or just metadata?</li>
<li>Which source file and line introduced this allocation?</li>
</ul>
<p><code>memscope-rs</code> approaches this problem in layers. The single-thread tracking path is the foundation: it combines raw runtime allocation facts from <code>GlobalAlloc</code> with Rust-level semantic metadata from <code>track!</code> and <code>Trackable</code>.</p>
<p>This article focuses on the base layer: how <code>memscope-rs</code> turns raw allocation events into analysis-ready Rust memory records.</p>
<hr />
<h2 id="1-why-single-thread-tracking-comes-first">1. Why Single-Thread Tracking Comes First</h2>
<p>Before multithreading, async tasks, ownership graphs, dashboards, or FFI analysis, the core question is simple:</p>
<blockquote>
<p>Can the tool reliably collect useful memory events for one execution flow?</p>
</blockquote>
<p>The single-thread model splits data collection into two complementary sources:</p>
<ol>
<li><code>GlobalAlloc</code> captures runtime allocation facts.</li>
<li><code>track!</code> and <code>Trackable</code> add Rust-level context.</li>
</ol>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[User Code] --> B[GlobalAlloc Hook]
A --> C[track! Macro]
B --> D[Raw Runtime Facts]
C --> E[Rust-Level Metadata]
D --> F[MemoryEvent]
E --> F
F --> G[EventStore]
G --> H[Analysis]
H --> I[JSON / HTML Report]
</div>
<p><code>GlobalAlloc</code> is the ground truth for pointer and size information, but it does not know Rust variable names or ownership semantics. The Rust-level semantic layer fills that gap explicitly.</p>
<hr />
<h2 id="2-runtime-ground-truth-globalalloc">2. Runtime Ground Truth: <code>GlobalAlloc</code></h2>
<p>At the lowest layer, <code>memscope-rs</code> wraps the system allocator with a custom allocator.</p>
<p>A simplified version looks like this:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">unsafe impl</span><span style="color: #F9E2AF;font-style: italic;"> GlobalAlloc</span><span style="color: #CBA6F7;"> for</span><span style="color: #F9E2AF;font-style: italic;"> TrackingAllocator</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> unsafe fn</span><span style="color: #89B4FA;font-style: italic;"> alloc</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> layout</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Layout</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> *</span><span style="color: #CBA6F7;">mut u8</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> ptr</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> System</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">alloc</span><span style="color: #9399B2;">(</span><span>layout</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span style="color: #94E2D5;"> !</span><span>ptr</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">is_null</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> _</span><span style="color: #94E2D5;"> =</span><span> tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">track_allocation</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">,</span><span> layout</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">size</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> ptr</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> unsafe fn</span><span style="color: #89B4FA;font-style: italic;"> dealloc</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> ptr</span><span style="color: #94E2D5;">: *</span><span style="color: #CBA6F7;">mut u8</span><span style="color: #9399B2;">,</span><span style="color: #EBA0AC;"> layout</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Layout</span><span style="color: #9399B2;">) {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> _</span><span style="color: #94E2D5;"> =</span><span> tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">track_deallocation</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> System</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">dealloc</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #9399B2;">,</span><span> layout</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The real implementation also includes a thread-local guard to avoid recursive tracking. Tracking itself can allocate, so the allocator must temporarily disable tracking while recording allocation metadata.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">thread_local!</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> static</span><span style="color: #FAB387;"> TRACKING_DISABLED</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">cell</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Cell</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">bool</span><span style="color: #89DCEB;">></span><span style="color: #94E2D5;"> =</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> const</span><span style="color: #9399B2;"> {</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">cell</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Cell</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span style="color: #F38BA8;">false</span><span style="color: #9399B2;">) };</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This layer provides high-confidence data:</p>
<table><thead><tr><th>Data</th><th>Source</th><th>Reliability</th></tr></thead><tbody>
<tr><td>Pointer address</td><td><code>GlobalAlloc</code></td><td>High</td></tr>
<tr><td>Allocation size</td><td><code>Layout::size()</code></td><td>High</td></tr>
<tr><td>Allocation event</td><td><code>alloc()</code></td><td>High</td></tr>
<tr><td>Deallocation event</td><td><code>dealloc()</code></td><td>High</td></tr>
</tbody></table>
<p>But this layer cannot tell whether a pointer belongs to a <code>Vec<T></code>, <code>String</code>, <code>Box<T></code>, <code>HashMap<K, V></code>, or a custom application type.</p>
<hr />
<h2 id="3-why-raw-allocation-data-is-not-enough">3. Why Raw Allocation Data Is Not Enough</h2>
<p>Allocator-level data might look like this:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>alloc ptr=0x1048a0000 size=1024</span></span>
<span class="giallo-l"><span>alloc ptr=0x1048b4000 size=24</span></span>
<span class="giallo-l"><span>dealloc ptr=0x1048a0000</span></span></code></pre>
<p>That is useful, but Rust developers usually want something closer to:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>variable: users</span></span>
<span class="giallo-l"><span>type: Vec<User></span></span>
<span class="giallo-l"><span>source: src/main.rs:42</span></span>
<span class="giallo-l"><span>size: 1024 bytes</span></span></code></pre>
<p>This is why <code>memscope-rs</code> uses an explicit macro layer:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> users</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> vec!</span><span style="color: #9399B2;">[</span><span style="color: #F9E2AF;font-style: italic;">User</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"Alice"</span><span style="color: #9399B2;">),</span><span style="color: #F9E2AF;font-style: italic;"> User</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"Bob"</span><span style="color: #9399B2;">)];</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">track!</span><span style="color: #9399B2;">(</span><span>tracker</span><span style="color: #9399B2;">,</span><span> users</span><span style="color: #9399B2;">);</span></span></code></pre>
<p>The macro injects compile-time metadata:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">macro_rules!</span><span style="color: #89B4FA;font-style: italic;"> track</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> (</span><span style="color: #F5C2E7;">$tracker</span><span style="color: #94E2D5;">:</span><span>expr</span><span style="color: #9399B2;">,</span><span style="color: #F5C2E7;"> $var</span><span style="color: #94E2D5;">:</span><span>expr</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> =></span><span style="color: #9399B2;"> {{</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> var_name</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> stringify!</span><span style="color: #9399B2;">(</span><span style="color: #F5C2E7;">$var</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F5C2E7;"> $tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">track_as</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> &$</span><span>var</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> var_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> file!</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> line!</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> module_path!</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> );</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }};</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is explicit by design. The tool does not pretend to automatically understand every variable in a Rust program.</p>
<hr />
<h2 id="4-trackable-the-semantic-layer">4. <code>Trackable</code>: The Semantic Layer</h2>
<p>To let a Rust value describe its memory role, <code>memscope-rs</code> uses the <code>Trackable</code> trait.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub trait</span><span style="color: #F9E2AF;font-style: italic;"> Trackable</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> track_kind</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_type_name</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> &</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #F9E2AF;font-style: italic;"> str</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_size_estimate</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_ref_count</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> None</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_data_ptr</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_data_size</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The most important method is <code>track_kind()</code> because not all Rust types should be modeled as simple heap pointers.</p>
<hr />
<h2 id="5-trackkind-rust-types-as-memory-roles">5. <code>TrackKind</code>: Rust Types as Memory Roles</h2>
<p><code>TrackKind</code> classifies tracked values into memory roles:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> HeapOwner</span><span style="color: #9399B2;"> {</span><span> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Container</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Value</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> StackOwner</span><span style="color: #9399B2;"> {</span><span> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span> heap_ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre><div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
graph TB
V[Vec<T>] --> H[HeapOwner]
S[String] --> H
B[Box<T>] --> H
HM[HashMap<K,V>] --> C[Container]
BT[BTreeMap<K,V>] --> C
VD[VecDeque<T>] --> C
I[i32 / bool / simple value] --> Val[Value]
A[Arc<T>] --> SO[StackOwner]
R[Rc<T>] --> SO
H --> HP[Real Heap Pointer]
C --> Meta[Metadata Only]
Val --> NoHeap[No Heap Allocation]
SO --> SP[Stack Ptr + Heap Ptr]
</div>
<p>This classification prevents <code>memscope-rs</code> from treating every Rust value as a fake heap allocation.</p>
<hr />
<h2 id="6-built-in-type-modeling">6. Built-in Type Modeling</h2>
<p>For <code>Vec<T></code>, the implementation uses the data pointer and capacity-based size estimate:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">impl</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #F9E2AF;font-style: italic;"> Trackable</span><span style="color: #CBA6F7;"> for</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> track_kind</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">HeapOwner</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">as_ptr</span><span style="color: #9399B2;">()</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> size</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">capacity</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">mem</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">size_of</span><span style="color: #94E2D5;">::</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_type_name</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> &</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #F9E2AF;font-style: italic;"> str</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "Vec<T>"</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_size_estimate</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">mem</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">size_of</span><span style="color: #94E2D5;">::</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">T</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">capacity</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>For <code>HashMap<K, V></code>, the implementation is intentionally more conservative:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">impl</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">K</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> V</span><span style="color: #89DCEB;">></span><span style="color: #F9E2AF;font-style: italic;"> Trackable</span><span style="color: #CBA6F7;"> for</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">collections</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">HashMap</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">K</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> V</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> track_kind</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> TrackKind</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Container</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_type_name</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> &</span><span style="color: #9399B2;">'</span><span style="color: #89B4FA;font-style: italic;">static</span><span style="color: #F9E2AF;font-style: italic;"> str</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "HashMap<K, V>"</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_size_estimate</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">mem</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">size_of</span><span style="color: #94E2D5;">::</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">K</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> V</span><span style="color: #9399B2;">)</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;"> *</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">capacity</span><span style="color: #9399B2;">()</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> get_data_ptr</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">&</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #CBA6F7;">usize</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> None</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This avoids exposing unstable internal container pointers as if they were simple user-owned heap buffers.</p>
<hr />
<h2 id="7-user-defined-types">7. User-Defined Types</h2>
<p>Standard library support is only half the story. Real applications use custom structs and enums.</p>
<p><code>memscope-rs</code> provides a derive macro for custom types:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">use</span><span style="color: #F9E2AF;"> memscope_rs</span><span style="color: #94E2D5;">::</span><span style="color: #9399B2;">{</span><span>track</span><span style="color: #9399B2;">,</span><span> tracker</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> Trackable</span><span style="color: #9399B2;">};</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">use</span><span style="color: #F9E2AF;"> memscope_derive</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Trackable</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;">#</span><span style="color: #9399B2;">[</span><span style="color: #F9E2AF;font-style: italic;">derive</span><span style="color: #9399B2;">(</span><span style="color: #F9E2AF;font-style: italic;">Trackable</span><span style="color: #9399B2;">)]</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;">struct</span><span style="color: #F9E2AF;font-style: italic;"> UserProfile</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> id</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> tags</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">String</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">fn</span><span style="color: #89B4FA;font-style: italic;"> main</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> tracker</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> tracker!</span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> profile</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> UserProfile</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> id</span><span style="color: #94E2D5;">:</span><span style="color: #FAB387;"> 1</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> name</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> "Alice"</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> tags</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> vec!</span><span style="color: #9399B2;">[</span><span style="color: #A6E3A1;">"rust"</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">(),</span><span style="color: #A6E3A1;"> "memory"</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">()],</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> track!</span><span style="color: #9399B2;">(</span><span>tracker</span><span style="color: #9399B2;">,</span><span> profile</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The derive macro estimates size by walking fields that implement <code>Trackable</code>. This is not perfect memory-layout reconstruction, but it provides a practical semantic layer for application objects.</p>
<hr />
<h2 id="8-event-recording-strategy">8. Event Recording Strategy</h2>
<p>When <code>track!</code> calls <code>track_as()</code>, the tracker branches based on <code>TrackKind</code>.</p>
<p>For <code>HeapOwner</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> event</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> MemoryEvent</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">allocate</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;">,</span><span> thread_id</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span>event</span><span style="color: #94E2D5;">.</span><span>var_name </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>name</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span>event</span><span style="color: #94E2D5;">.</span><span>type_name </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>type_name</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span>event</span><span style="color: #94E2D5;">.</span><span>source_file </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>file</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span>event</span><span style="color: #94E2D5;">.</span><span>source_line </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>line</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span>event</span><span style="color: #94E2D5;">.</span><span>module_path </span><span style="color: #94E2D5;">=</span><span style="color: #F9E2AF;font-style: italic;"> Some</span><span style="color: #9399B2;">(</span><span>module_path</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">());</span></span>
<span class="giallo-l"><span style="color: #F38BA8;">self</span><span style="color: #94E2D5;">.</span><span>event_store</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">record</span><span style="color: #9399B2;">(</span><span>event</span><span style="color: #9399B2;">);</span></span></code></pre>
<p>For <code>Container</code> or <code>Value</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">let mut</span><span> event</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> MemoryEvent</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">metadata</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span> name</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> type_name</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> thread_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> var</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">get_size_estimate</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #F38BA8;">self</span><span style="color: #94E2D5;">.</span><span>event_store</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">record</span><span style="color: #9399B2;">(</span><span>event</span><span style="color: #9399B2;">);</span></span></code></pre>
<p>The important distinction is that containers and values produce metadata events, not fake heap allocation records.</p>
<hr />
<h2 id="9-benchmark-interpretation">9. Benchmark Interpretation</h2>
<p>The benchmark log shows that single-variable tracking is generally sub-microsecond for small tracked values:</p>
<table><thead><tr><th>Benchmark</th><th style="text-align: right">Approximate Time</th></tr></thead><tbody>
<tr><td><code>track_single/vec/64</code></td><td style="text-align: right">~653 ns</td></tr>
<tr><td><code>track_single/vec/1024</code></td><td style="text-align: right">~666 ns</td></tr>
<tr><td><code>track_single/vec/1048576</code></td><td style="text-align: right">~4.93 µs</td></tr>
<tr><td><code>track_multiple/variables/1000</code></td><td style="text-align: right">~669.67 µs</td></tr>
<tr><td><code>track_multiple/variables/10000</code></td><td style="text-align: right">~6.59 ms</td></tr>
</tbody></table>
<p>The honest interpretation is that absolute latency is usable for profiling, but the benchmark log also contains regressions relative to previous runs. Performance should be described as measured and evolving, not magically zero-cost.</p>
<hr />
<h2 id="10-what-this-layer-can-and-cannot-know">10. What This Layer Can and Cannot Know</h2>
<p>High-confidence data:</p>
<ul>
<li>pointer address</li>
<li>allocation size</li>
<li>allocation/deallocation event</li>
<li>timestamp</li>
<li>thread id</li>
</ul>
<p>Explicit metadata:</p>
<ul>
<li>variable name</li>
<li>type name</li>
<li>source file</li>
<li>line number</li>
<li>module path</li>
<li>semantic role through <code>TrackKind</code></li>
</ul>
<p>Not directly known:</p>
<ul>
<li>every ownership transfer</li>
<li>every borrow</li>
<li>every move</li>
<li>every variable unless explicitly tracked</li>
</ul>
<p>This distinction matters. <code>memscope-rs</code> is strongest when it clearly separates runtime facts, explicit metadata, and later inference.</p>
<hr />
<h2 id="11-summary">11. Summary</h2>
<p>The single-thread tracking layer is not the most glamorous part of <code>memscope-rs</code>, but it is the foundation.</p>
<p>It combines:</p>
<ul>
<li><code>GlobalAlloc</code> for raw runtime facts;</li>
<li><code>track!</code> for variable context;</li>
<li><code>Trackable</code> for Rust type semantics;</li>
<li><code>TrackKind</code> for memory role classification;</li>
<li><code>EventStore</code> for later analysis and export.</li>
</ul>
<p>The key design principle is simple:</p>
<blockquote>
<p>Capture what is real at runtime, then add Rust semantics explicitly and conservatively.</p>
</blockquote>
Unsafe and FFI Memory Passport: Tracking Memory Across Rust's Trust Boundary
2026-05-20T00:00:00+00:00
2026-05-20T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/memscope-rs/unsafe-ffi-memory-passport/
<h1 id="unsafe-and-ffi-memory-passport-tracking-memory-across-rust-s-trust-boundary">Unsafe and FFI Memory Passport: Tracking Memory Across Rust’s Trust Boundary</h1>
<p>Rust is safe until you cross the boundary.</p>
<p>That boundary may be:</p>
<ul>
<li>an <code>unsafe</code> block;</li>
<li>a raw pointer;</li>
<li><code>std::alloc::alloc</code>;</li>
<li>C <code>malloc/free</code>;</li>
<li>a pointer returned by an external library;</li>
<li>an ownership contract that exists outside Rust’s type system.</li>
</ul>
<p><code>memscope-rs</code> does not claim to prove unsafe code correct. Instead, it records lifecycle evidence for memory that crosses Rust’s trust boundary.</p>
<p>The concept is called a memory passport.</p>
<hr />
<h2 id="1-why-unsafe-and-ffi-need-a-separate-model">1. Why Unsafe and FFI Need a Separate Model</h2>
<p>Allocator-level tracking can tell us:</p>
<ul>
<li>pointer;</li>
<li>size;</li>
<li>allocation event;</li>
<li>deallocation event.</li>
</ul>
<p>For unsafe and FFI memory, that is not enough. We also want to know:</p>
<ul>
<li>who created this memory?</li>
<li>was it handed to foreign code?</li>
<li>was it freed by Rust or by foreign code?</li>
<li>is it still in foreign custody?</li>
<li>does its lifecycle look suspicious?</li>
</ul>
<div class="mermaid-direct" data-series="memscope-rs" data-name="graph.mmd">
flowchart TD
A[Safe Rust] --> B[Unsafe Block]
B --> C[Raw Pointer]
C --> D[FFI Boundary]
D --> E[C malloc / External Memory]
C --> P[Memory Passport]
E --> P
P --> O[Origin]
P --> L[Lifecycle Events]
P --> V[Validity Status]
P --> S[Safety Violations]
P --> R[JSON / Dashboard]
</div>
<hr />
<h2 id="2-what-is-a-memory-passport">2. What Is a Memory Passport?</h2>
<p>A memory passport is a runtime identity record for a memory allocation.</p>
<p>The core structure contains:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub struct</span><span style="color: #F9E2AF;font-style: italic;"> MemoryPassport</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> passport_id</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> allocation_ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> size_bytes</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> type_name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> var_name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> status_at_shutdown</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> PassportStatus</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> lifecycle_events</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Vec</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">PassportEvent</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> created_at</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> updated_at</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u64</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> pub</span><span> metadata</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> HashMap</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">String</span><span style="color: #9399B2;">,</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is not just an allocation record. It is a lifecycle document.</p>
<hr />
<h2 id="3-allocation-source">3. Allocation Source</h2>
<p>The unsafe/FFI tracker models where memory came from:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> AllocationSource</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> RustSafe</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> UnsafeRust</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> unsafe_block_location</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> call_stack</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> CallStackRef</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> risk_assessment</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> RiskAssessment</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> FfiC</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> resolved_function</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> ResolvedFfiFunction</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> call_stack</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> CallStackRef</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> libc_hook_info</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> LibCHookInfo</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> CrossBoundary</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> from</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Box</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">AllocationSource</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> to</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Box</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">AllocationSource</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> transfer_timestamp</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> u128</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> transfer_metadata</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> TransferMetadata</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This allows <code>memscope-rs</code> to distinguish normal Rust allocation from unsafe Rust allocation, C allocation, and cross-boundary transfer.</p>
<hr />
<h2 id="4-safety-violation-signals">4. Safety Violation Signals</h2>
<p>The tracker records suspicious patterns as safety violation signals:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> SafetyViolation</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> DoubleFree</span><span style="color: #9399B2;"> {</span><span style="color: #94E2D5;"> ...</span><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> InvalidFree</span><span style="color: #9399B2;"> {</span><span style="color: #94E2D5;"> ...</span><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> PotentialLeak</span><span style="color: #9399B2;"> {</span><span style="color: #94E2D5;"> ...</span><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> CrossBoundaryRisk</span><span style="color: #9399B2;"> {</span><span style="color: #94E2D5;"> ...</span><span style="color: #9399B2;"> },</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>These are runtime signals, not formal proofs.</p>
<p>The careful wording is:</p>
<ul>
<li>double free candidate;</li>
<li>invalid free signal;</li>
<li>potential leak;</li>
<li>cross-boundary risk.</li>
</ul>
<p>Not every suspicious runtime signal is automatically a proven bug, but each one is worth investigation.</p>
<hr />
<h2 id="5-creating-and-updating-passports">5. Creating and Updating Passports</h2>
<p>Creating a passport inserts an initial lifecycle event:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub fn</span><span style="color: #89B4FA;font-style: italic;"> create_passport</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> &</span><span style="color: #F38BA8;">self</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> allocation_ptr</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> size_bytes</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> initial_context</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> String</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> type_name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">String</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #EBA0AC;"> var_name</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> Option</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">String</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -></span><span style="color: #F9E2AF;font-style: italic;"> TrackingResult</span><span style="color: #89DCEB;"><</span><span style="color: #F9E2AF;font-style: italic;">String</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> passport_id</span><span style="color: #94E2D5;"> =</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">generate_passport_id</span><span style="color: #9399B2;">(</span><span>allocation_ptr</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> initial_event</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> PassportEvent</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> event_type</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> PassportEventType</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">AllocatedInRust</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> timestamp</span><span style="color: #94E2D5;">:</span><span> current_time</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> context</span><span style="color: #94E2D5;">:</span><span> initial_context</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> call_stack</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">capture_call_stack</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> metadata</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> HashMap</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> sequence_number</span><span style="color: #94E2D5;">:</span><span style="color: #F38BA8;"> self</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">get_next_event_sequence</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> passport</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> MemoryPassport</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span> passport_id</span><span style="color: #94E2D5;">:</span><span> passport_id</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">clone</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span> allocation_ptr</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> size_bytes</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> type_name</span><span style="color: #94E2D5;">:</span><span> type_name</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">unwrap_or_else</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">||</span><span style="color: #A6E3A1;"> "-"</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">()),</span></span>
<span class="giallo-l"><span> var_name</span><span style="color: #94E2D5;">:</span><span> var_name</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">unwrap_or_else</span><span style="color: #9399B2;">(</span><span style="color: #94E2D5;">||</span><span style="color: #A6E3A1;"> "-"</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">()),</span></span>
<span class="giallo-l"><span> status_at_shutdown</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> PassportStatus</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;font-style: italic;">Unknown</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> lifecycle_events</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> vec!</span><span style="color: #9399B2;">[</span><span>initial_event</span><span style="color: #9399B2;">],</span></span>
<span class="giallo-l"><span> created_at</span><span style="color: #94E2D5;">:</span><span> current_time</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> updated_at</span><span style="color: #94E2D5;">:</span><span> current_time</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> metadata</span><span style="color: #94E2D5;">:</span><span style="color: #F9E2AF;font-style: italic;"> HashMap</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Ok</span><span style="color: #9399B2;">(</span><span>passport_id</span><span style="color: #9399B2;">)</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>Later events include:</p>
<ul>
<li><code>HandoverToFfi</code></li>
<li><code>FreedByForeign</code></li>
<li><code>ReclaimedByRust</code></li>
<li><code>BoundaryAccess</code></li>
<li><code>OwnershipTransfer</code></li>
<li><code>ValidationCheck</code></li>
<li><code>CorruptionDetected</code></li>
</ul>
<hr />
<h2 id="6-shutdown-status">6. Shutdown Status</h2>
<p>At shutdown or export time, a passport can be summarized with a final status:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">pub enum</span><span style="color: #F9E2AF;font-style: italic;"> PassportStatus</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> FreedByRust</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> HandoverToFfi</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> FreedByForeign</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> ReclaimedByRust</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> InForeignCustody</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;font-style: italic;"> Unknown</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This status helps summarize lifecycle state:</p>
<ul>
<li>was the memory freed by Rust?</li>
<li>was it handed to FFI?</li>
<li>did foreign code free it?</li>
<li>is it still in foreign custody?</li>
<li>is the status unknown?</li>
</ul>
<hr />
<h2 id="7-example-unsafe-allocation">7. Example: Unsafe Allocation</h2>
<p>The showcase example tracks manually allocated unsafe Rust memory:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">unsafe</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> layout</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;font-style: italic;"> Layout</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">new</span><span style="color: #94E2D5;">::</span><span style="color: #89DCEB;"><</span><span style="color: #9399B2;">[</span><span style="color: #CBA6F7;">i32</span><span style="color: #9399B2;">;</span><span style="color: #FAB387;"> 64</span><span style="color: #9399B2;">]</span><span style="color: #89DCEB;">></span><span style="color: #9399B2;">();</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> ptr</span><span style="color: #94E2D5;"> =</span><span style="color: #89B4FA;font-style: italic;"> alloc</span><span style="color: #9399B2;">(</span><span>layout</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> if</span><span style="color: #94E2D5;"> !</span><span>ptr</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">is_null</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span> tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">create_passport</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span> ptr</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> layout</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">size</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"unsafe_vec_</span><span style="color: #9399B2;">{}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> i</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> )</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> let</span><span> slice</span><span style="color: #94E2D5;"> =</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">slice</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">from_raw_parts_mut</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #CBA6F7;"> as</span><span style="color: #94E2D5;"> *</span><span style="color: #CBA6F7;">mut i32</span><span style="color: #9399B2;">,</span><span style="color: #FAB387;"> 64</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> for</span><span style="color: #9399B2;"> (</span><span>j</span><span style="color: #9399B2;">,</span><span> item</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> in</span><span> slice</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">iter_mut</span><span style="color: #9399B2;">()</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">enumerate</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span style="color: #94E2D5;"> *</span><span>item</span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> (</span><span>i</span><span style="color: #94E2D5;"> *</span><span style="color: #FAB387;"> 100</span><span style="color: #94E2D5;"> +</span><span> j</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> as i32</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> dealloc</span><span style="color: #9399B2;">(</span><span>ptr</span><span style="color: #9399B2;">,</span><span> layout</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>The passport gives this memory an identity before it is manually written and released.</p>
<hr />
<h2 id="8-example-ffi-allocation">8. Example: FFI Allocation</h2>
<p>The same example also tracks C allocation:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #CBA6F7;">extern</span><span style="color: #A6E3A1;"> "C"</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> malloc</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;">size</span><span style="color: #94E2D5;">:</span><span style="color: #CBA6F7;"> usize</span><span style="color: #9399B2;">)</span><span style="color: #94E2D5;"> -> *</span><span style="color: #CBA6F7;">mut</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">ffi</span><span style="color: #94E2D5;">::</span><span style="color: #EBA0AC;">c_void</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> fn</span><span style="color: #89B4FA;font-style: italic;"> free</span><span style="color: #9399B2;">(</span><span style="color: #EBA0AC;">ptr</span><span style="color: #94E2D5;">: *</span><span style="color: #CBA6F7;">mut</span><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">ffi</span><span style="color: #94E2D5;">::</span><span style="color: #EBA0AC;">c_void</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">let</span><span> ffi_ptr</span><span style="color: #94E2D5;"> =</span><span style="color: #CBA6F7;"> unsafe</span><span style="color: #9399B2;"> {</span><span style="color: #89B4FA;font-style: italic;"> malloc</span><span style="color: #9399B2;">(</span><span>size</span><span style="color: #9399B2;">) };</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;">if</span><span style="color: #94E2D5;"> !</span><span>ffi_ptr</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">is_null</span><span style="color: #9399B2;">() {</span></span>
<span class="giallo-l"><span> tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">create_passport</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span> ffi_ptr</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span> size</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"ffi_alloc_</span><span style="color: #9399B2;">{}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> i</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> )</span><span style="color: #94E2D5;">?</span><span style="color: #9399B2;">;</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span> tracker</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">record_handover</span><span style="color: #9399B2;">(</span></span>
<span class="giallo-l"><span> ffi_ptr</span><span style="color: #CBA6F7;"> as usize</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "foreign_function"</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">to_string</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"ffi_call_</span><span style="color: #9399B2;">{}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> i</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> );</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #CBA6F7;"> unsafe</span><span style="color: #9399B2;"> {</span></span>
<span class="giallo-l"><span style="color: #F9E2AF;"> std</span><span style="color: #94E2D5;">::</span><span style="color: #F9E2AF;">ptr</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">write_bytes</span><span style="color: #9399B2;">(</span><span>ffi_ptr</span><span style="color: #CBA6F7;"> as</span><span style="color: #94E2D5;"> *</span><span style="color: #CBA6F7;">mut u8</span><span style="color: #9399B2;">, (</span><span style="color: #FAB387;">0x40</span><span style="color: #94E2D5;"> +</span><span> i</span><span style="color: #9399B2;">)</span><span style="color: #CBA6F7;"> as u8</span><span style="color: #9399B2;">,</span><span> size</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;"> free</span><span style="color: #9399B2;">(</span><span>ffi_ptr</span><span style="color: #9399B2;">);</span></span>
<span class="giallo-l"><span style="color: #9399B2;"> }</span></span>
<span class="giallo-l"><span style="color: #9399B2;">}</span></span></code></pre>
<p>This is a typical trust-boundary case: Rust sees a raw <code>*mut c_void</code>, but the allocation and release semantics are foreign.</p>
<hr />
<h2 id="9-export-pipeline">9. Export Pipeline</h2>
<p>The export pipeline writes passport-related JSON files:</p>
<ul>
<li><code>memory_passports.json</code></li>
<li><code>leak_detection.json</code></li>
<li><code>unsafe_ffi.json</code></li>
</ul>
<p>The memory passport export includes fields such as:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="rust"><span class="giallo-l"><span style="color: #F9E2AF;">serde_json</span><span style="color: #94E2D5;">::</span><span style="color: #89B4FA;font-style: italic;">json!</span><span style="color: #9399B2;">({</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "passport_id"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span>passport_id</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "allocation_ptr"</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"0x</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:x</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> p</span><span style="color: #94E2D5;">.</span><span>allocation_ptr</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "size_bytes"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span>size_bytes</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "created_at"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span>created_at</span><span style="color: #9399B2;">,</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "lifecycle_events"</span><span style="color: #94E2D5;">:</span><span> p</span><span style="color: #94E2D5;">.</span><span>lifecycle_events</span><span style="color: #94E2D5;">.</span><span style="color: #89B4FA;font-style: italic;">len</span><span style="color: #9399B2;">(),</span></span>
<span class="giallo-l"><span style="color: #A6E3A1;"> "status"</span><span style="color: #94E2D5;">:</span><span style="color: #89B4FA;font-style: italic;"> format!</span><span style="color: #9399B2;">(</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">{</span><span style="color: #A6E3A1;">:?</span><span style="color: #9399B2;">}</span><span style="color: #A6E3A1;">"</span><span style="color: #9399B2;">,</span><span> p</span><span style="color: #94E2D5;">.</span><span>status_at_shutdown</span><span style="color: #9399B2;">),</span></span>
<span class="giallo-l"><span style="color: #9399B2;">})</span></span></code></pre>
<p>The unsafe/FFI export filters passports with FFI-related final states such as:</p>
<ul>
<li><code>HandoverToFfi</code></li>
<li><code>InForeignCustody</code></li>
<li><code>FreedByForeign</code></li>
</ul>
<hr />
<h2 id="10-what-this-can-and-cannot-prove">10. What This Can and Cannot Prove</h2>
<p>It can observe:</p>
<ul>
<li>pointer creation;</li>
<li>allocation size;</li>
<li>lifecycle events;</li>
<li>handover to FFI;</li>
<li>foreign free records;</li>
<li>suspicious invalid or repeated frees;</li>
<li>final passport status.</li>
</ul>
<p>It cannot prove:</p>
<ul>
<li>all unsafe code is correct;</li>
<li>foreign code obeyed its ownership contract;</li>
<li>every external pointer has a complete origin;</li>
<li>every suspicious signal is definitely a bug.</li>
</ul>
<p>This is runtime observability, not formal verification.</p>
<hr />
<h2 id="11-summary">11. Summary</h2>
<p>The memory passport model is useful because it turns unsafe and FFI memory into something auditable.</p>
<p>It records:</p>
<ul>
<li>origin;</li>
<li>lifecycle events;</li>
<li>custody transitions;</li>
<li>final status;</li>
<li>safety violation signals.</li>
</ul>
<p>The most honest description is:</p>
<blockquote>
<p><code>memscope-rs</code> does not make unsafe code safe. It makes unsafe and FFI memory lifecycles easier to observe and audit.</p>
</blockquote>
Red/Blue Team Testing Guide
2025-03-01T00:00:00+00:00
2025-03-01T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/08-log-read/
<h1 id="red-blue-team-testing-guide">Red/Blue Team Testing Guide</h1>
<p><strong>Version</strong>: v0.1.9
<strong>Purpose</strong>: Validate OmniScope's detection rate (red team) and false positive rate (blue team) against the <code>corpus/</code> test suite.</p>
<h2 id="quick-start">Quick Start</h2>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">make</span><span style="color: #A6E3A1;"> red-team</span><span style="color: #9399B2;font-style: italic;"> # Adversarial: does the tool detect known bugs?</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">make</span><span style="color: #A6E3A1;"> blue-team</span><span style="color: #9399B2;font-style: italic;"> # Defensive: does the tool avoid false alarms?</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">make</span><span style="color: #A6E3A1;"> corpus-test</span><span style="color: #9399B2;font-style: italic;"> # Run both</span></span></code></pre>
<hr />
<h2 id="part-1-red-team-adversarial-detection-test">Part 1: Red Team — Adversarial Detection Test</h2>
<h3 id="what-it-does">What It Does</h3>
<p>Runs OmniScope against crafted test files in <code>corpus/red_team_test/</code> that contain <strong>known bugs</strong> (memory leaks, double frees, use-after-free, FFI boundary violations). Measures <strong>recall</strong> — how many injected bugs the tool actually catches.</p>
<h3 id="run-log-real-output">Run Log (Real Output)</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>╔════════════════════════════════════════════════════════════════╗</span></span>
<span class="giallo-l"><span>║ RED TEAM TEST ║</span></span>
<span class="giallo-l"><span>║ Adversarial: detect known bugs in crafted test cases ║</span></span>
<span class="giallo-l"><span>╚════════════════════════════════════════════════════════════════╝</span></span>
<span class="giallo-l"><span> ✅ red_team_bugs.ll 25 issues</span></span>
<span class="giallo-l"><span> ✅ ffi_boundary_bugs.ll 18 issues</span></span>
<span class="giallo-l"><span> ✅ cross_lang_free_bugs.ll 12 issues</span></span>
<span class="giallo-l"><span> ✅ cross_lang_free_complete.ll 15 issues</span></span>
<span class="giallo-l"><span> ✅ subtle_ffi_bugs.ll 8 issues</span></span>
<span class="giallo-l"><span> ✅ python_c_api_bugs.ll 10 issues</span></span>
<span class="giallo-l"><span> ✅ posix_ffi_bugs.ll 6 issues</span></span>
<span class="giallo-l"><span>────────────────────────────────────────────────────────</span></span>
<span class="giallo-l"><span> Red Team: 7 files, 94 total issues detected</span></span>
<span class="giallo-l"><span> ✅ Detection threshold met</span></span></code></pre><h3 id="line-by-line-interpretation">Line-by-Line Interpretation</h3>
<h4 id="header">Header</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>╔════════════════════════════════════════════════════════════════╗</span></span>
<span class="giallo-l"><span>║ RED TEAM TEST ║</span></span>
<span class="giallo-l"><span>╚════════════════════════════════════════════════════════════════╝</span></span></code></pre>
<p>This is the test banner. The Makefile target <code>red-team</code> (defined in <code>Makefile:640</code>) triggers <code>build</code> first, then iterates over <code>RED_IR_FILES</code>.</p>
<h4 id="per-file-result">Per-File Result</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span> ✅ red_team_bugs.ll 25 issues</span></span></code></pre><table><thead><tr><th>Field</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>✅</code></td><td>File produced ≥1 issue (pass)</td></tr>
<tr><td><code>❌</code></td><td>File produced 0 issues (miss — potential regression)</td></tr>
<tr><td><code>red_team_bugs.ll</code></td><td>Test file name</td></tr>
<tr><td><code>25 issues</code></td><td>Total issues detected by OmniScope</td></tr>
</tbody></table>
<p><strong>How to locate the source</strong>: Each <code>.ll</code> file is compiled from a <code>.c</code> file in the same directory. For example:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>corpus/red_team_test/red_team_bugs.c ← Source code with injected bugs</span></span>
<span class="giallo-l"><span>corpus/red_team_test/red_team_bugs.ll ← Compiled LLVM IR (what OmniScope analyzes)</span></span></code></pre><h4 id="summary-line">Summary Line</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span> Red Team: 7 files, 94 total issues detected</span></span></code></pre>
<ul>
<li><code>7 files</code> — how many <code>.ll</code> files existed and were analyzed</li>
<li><code>94 total issues</code> — sum of all issues across all files</li>
</ul>
<h4 id="threshold-check">Threshold Check</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span> ✅ Detection threshold met</span></span></code></pre>
<p>If <code>total_issues < 10</code>, the test reports <code>⚠️ LOW detection count — investigate regressions</code>. This is a coarse sanity check, not a precise benchmark. For precise detection rates, use <code>make benchmark</code>.</p>
<h3 id="drilling-into-a-single-file">Drilling Into a Single File</h3>
<p>To see the full OmniScope output for one test file:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">./zig-out/bin/OmniScope</span><span style="color: #A6E3A1;"> corpus/red_team_test/red_team_bugs_O0.ll</span><span style="color: #94E2D5;"> 2>&1</span></span></code></pre><h4 id="startup-phase">Startup Phase</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: [INFO] === OmniScope IR Analysis ===</span></span>
<span class="giallo-l"><span>info: [INFO] File: corpus/red_team_test/red_team_bugs_O0.ll</span></span>
<span class="giallo-l"><span>info: [INFO] Loaded: 35 functions</span></span></code></pre><table><thead><tr><th>Line</th><th>What It Means</th></tr></thead><tbody>
<tr><td><code>=== OmniScope IR Analysis ===</code></td><td>Analysis session started</td></tr>
<tr><td><code>File: ...</code></td><td>Input LLVM IR file path</td></tr>
<tr><td><code>Loaded: 35 functions</code></td><td>Number of functions parsed from the IR</td></tr>
</tbody></table>
<h4 id="language-detection">Language Detection</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: [INFO] LANG-DETECT: module language = c, confidence = 100.0%, method = sampling</span></span></code></pre><table><thead><tr><th>Field</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>module language = c</code></td><td>Dominant language in the module</td></tr>
<tr><td><code>confidence = 100.0%</code></td><td>How certain (sampling counts function name patterns)</td></tr>
<tr><td><code>method = sampling</code></td><td><code>sampling</code> = statistical; <code>personality</code> = DWARF debug info</td></tr>
</tbody></table>
<p>The <code>sampling</code> method iterates all function names and counts language-specific patterns (<code>_ZN</code> for C++/Rust, <code>_R</code> for Rust v0, <code>Go.</code> for Go, etc.).</p>
<h4 id="pre-pass-scanners">Pre-Pass Scanners</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: [INFO] MallocCheck: Analyzed functions, found 9 unchecked allocations</span></span>
<span class="giallo-l"><span>info: [INFO] IntegerOverflow: Analyzed functions, found 0 potential overflows</span></span>
<span class="giallo-l"><span>info: [INFO] BufferOverflow: No buffer overflow issues detected</span></span>
<span class="giallo-l"><span>info: [INFO] ReturnCheck: Analyzed functions, found 1 unchecked return values</span></span>
<span class="giallo-l"><span>info: [INFO] RustFfiFilter: analyzed 17 funcs, 0 findings (0 stack escapes)</span></span></code></pre><table><thead><tr><th>Scanner</th><th>What It Checks</th></tr></thead><tbody>
<tr><td><code>MallocCheck</code></td><td><code>malloc()</code> return value not null-checked</td></tr>
<tr><td><code>IntegerOverflow</code></td><td>Arithmetic ops that may overflow</td></tr>
<tr><td><code>BufferOverflow</code></td><td>Array index out of bounds</td></tr>
<tr><td><code>ReturnCheck</code></td><td>Return values of FFI calls not checked</td></tr>
<tr><td><code>RustFfiFilter</code></td><td>Rust functions without FFI relevance (skipped)</td></tr>
</tbody></table>
<h4 id="cross-language-edge-extraction">Cross-Language Edge Extraction</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: [INFO] CallGraph: extracted 15 cross-language edges</span></span>
<span class="giallo-l"><span>info: [INFO] CallGraph: built semantics CallGraph with 35 nodes, 63 edges for BFS traversal</span></span></code></pre><table><thead><tr><th>Field</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>15 cross-language edges</code></td><td>Calls where caller and callee are in different languages</td></tr>
<tr><td><code>35 nodes, 63 edges</code></td><td>Call graph size for BFS reachability analysis</td></tr>
</tbody></table>
<p>Each edge records <code>(caller_name, callee_name, caller_lang, callee_lang)</code>. An edge is "cross-language" when <code>caller_lang != callee_lang</code>.</p>
<h4 id="danger-surface-pass">Danger Surface Pass</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: [INFO] [P1-1] DangerSurfacePass: 15 FFI, 0 allocs, 0 funcs | Phase1=0ms (args=0 rets=0 alias_traces=0) Phase2=0ms (cross_lang_free=0)</span></span></code></pre><table><thead><tr><th>Field</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>15 FFI</code></td><td>Number of FFI boundary nodes in MemoryGraph</td></tr>
<tr><td><code>Phase1</code></td><td>Time spent tracing call args/rets through FFI boundaries</td></tr>
<tr><td><code>Phase2</code></td><td>Time spent scanning for cross-language free violations</td></tr>
<tr><td><code>cross_lang_free=0</code></td><td>Cross-language alloc/free mismatches found</td></tr>
</tbody></table>
<h4 id="pointer-ownership-analysis">Pointer Ownership Analysis</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: [INFO] PointerOwnership: Source 1 (MemoryGraph) — 59 unfreed + 23 freed = 82 total nodes</span></span>
<span class="giallo-l"><span>info: [INFO] PointerOwnership: Source 2 (GlobalAllocTracker) — 9 records, 0 freed</span></span>
<span class="giallo-l"><span>info: [INFO] PointerOwnership: Source 3 (IR scan) added 32 frees — total now 32</span></span>
<span class="giallo-l"><span>info: [INFO] PointerOwnership: Pre-populated from MemoryGraph + GlobalAllocTracker + IR-scan — 59 allocs, 32 frees</span></span></code></pre>
<p>This is the <strong>three-source data fusion</strong>:</p>
<table><thead><tr><th>Source</th><th>What It Provides</th></tr></thead><tbody>
<tr><td>Source 1: MemoryGraph</td><td>Allocation sites + freed status from upstream passes</td></tr>
<tr><td>Source 2: GlobalAllocTracker</td><td>Supplementary free tracking</td></tr>
<tr><td>Source 3: IR scan</td><td>Direct LLVM IR scan for <code>free</code>/<code>dealloc</code> call instructions (fallback)</td></tr>
</tbody></table>
<p><strong>How to interpret</strong>: <code>59 unfreed</code> means 59 allocations were not matched with a corresponding free. These are potential memory leaks. The tool then applies filters (RAII, Meyers singleton, ref-counted containers) to reduce false positives.</p>
<h4 id="vulnerability-lines">Vulnerability Lines</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: [ERROR] VULNERABILITY OMI-001 [medium] [Confidence: MEDIUM]</span></span>
<span class="giallo-l"><span>info: [ERROR] Type: tainted_path_to_sink</span></span>
<span class="giallo-l"><span>info: [ERROR] Reason: Untrusted data flows to sensitive sink without validation</span></span>
<span class="giallo-l"><span>info: [ERROR] Path:</span></span>
<span class="giallo-l"><span>info: [ERROR] [Sink] printf()</span></span>
<span class="giallo-l"><span>info: [ERROR] [Source] main() - initial taint source</span></span></code></pre><table><thead><tr><th>Field</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>OMI-001</code></td><td>Auto-incremented issue ID (unique per analysis run)</td></tr>
<tr><td><code>[medium]</code></td><td>Severity: <code>critical</code> > <code>high</code> > <code>medium</code> > <code>low</code></td></tr>
<tr><td><code>[Confidence: MEDIUM]</code></td><td><code>HIGH</code> / <code>MEDIUM</code> / <code>LOW</code></td></tr>
<tr><td><code>Type: tainted_path_to_sink</code></td><td>Issue category (20+ types)</td></tr>
<tr><td><code>Reason:</code></td><td>Human-readable explanation of the problem</td></tr>
<tr><td><code>Path:</code></td><td>Data flow path from source to sink</td></tr>
</tbody></table>
<p><strong>How to locate the problem in YOUR code</strong>:</p>
<p><strong>Step 1: Identify the function.</strong> The output always includes a function name (mangled). For example:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: [ERROR] in _Z37bug_cpp_05_unique_ptr_callback_escapev</span></span></code></pre>
<p>Demangle it:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># C++ (Itanium ABI)</span></span>
<span class="giallo-l"><span style="color: #F38BA8;font-style: italic;">echo</span><span style="color: #A6E3A1;"> "_Z37bug_cpp_05_unique_ptr_callback_escapev"</span><span style="color: #94E2D5;"> |</span><span style="color: #89B4FA;font-style: italic;"> c++filt</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># → bug_cpp_05_unique_ptr_callback_escape()</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># Rust (v0)</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">rustfilt</span><span style="color: #A6E3A1;"> "_ZN4core3ptr13drop_in_place17h1234E"</span></span>
<span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># → core::ptr::drop_in_place<Type></span></span></code></pre>
<p>Then <code>grep</code> for the demangled name in your source:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">grep</span><span style="color: #A6E3A1;"> -rn "bug_cpp_05_unique_ptr_callback_escape" src/</span></span></code></pre>
<p><strong>Step 2: Use debug info for exact line.</strong> If you compiled with <code>-g</code>, the JSON output includes <code>file</code> and <code>line</code>:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">./zig-out/bin/OmniScope</span><span style="color: #A6E3A1;"> target.ll --json</span><span style="color: #94E2D5;"> |</span><span style="color: #89B4FA;font-style: italic;"> jq</span><span style="color: #A6E3A1;"> '.issues[] | {function: .location.function, file: .location.file, line: .location.line}'</span></span></code></pre>
<p>Without <code>-g</code>, you only get the function name. <strong>Always compile with <code>-g</code> for actionable reports.</strong></p>
<p><strong>Step 3: Understand the issue type.</strong> Each type points to a specific source pattern:</p>
<table><thead><tr><th>Issue Type</th><th>What to look for in YOUR code</th></tr></thead><tbody>
<tr><td><code>memory_leak</code></td><td><code>malloc</code>/<code>new</code>/<code>Box::new</code> without matching <code>free</code>/<code>delete</code>/<code>drop</code></td></tr>
<tr><td><code>use_after_free</code></td><td>Pointer used after <code>free()</code> or <code>Box::from_raw()</code></td></tr>
<tr><td><code>double_free</code></td><td>Same pointer freed twice</td></tr>
<tr><td><code>cross_lang_free_mismatch</code></td><td>Allocated in Rust (<code>Box::into_raw</code>), freed in C (<code>free</code>) — or vice versa</td></tr>
<tr><td><code>borrow_escape</code></td><td>Stack pointer or <code>&mut</code> passed to FFI function that outlives the scope</td></tr>
<tr><td><code>tainted_path_to_sink</code></td><td>User input reaches <code>system()</code>/<code>exec()</code>/<code>printf()</code> without validation</td></tr>
<tr><td><code>null_dereference</code></td><td>Pointer used without null check after <code>malloc</code> or FFI call</td></tr>
<tr><td><code>buffer_overflow</code></td><td>Array access with unchecked index</td></tr>
<tr><td><code>ffi_unsafe_call</code></td><td>Calling an FFI function with wrong pointer type or lifetime</td></tr>
<tr><td><code>format_string</code></td><td>Non-literal format string in <code>printf</code>/<code>sprintf</code></td></tr>
</tbody></table>
<p><strong>Step 4: Use <code>--sarif</code> for IDE integration.</strong> SARIF output maps directly to source files and opens in VS Code / GitHub Code Scanning with inline annotations.</p>
<h4 id="performance-profile">Performance Profile</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: Operation Calls Total (ms) Avg (us) Max (us)</span></span>
<span class="giallo-l"><span>info: --------------------------------------------------------------------------------</span></span>
<span class="giallo-l"><span>info: init 1 7.77 7770.96 7770.96</span></span>
<span class="giallo-l"><span>info: analysis 1 5.89 5888.63 5888.63</span></span>
<span class="giallo-l"><span>info: detect 2 3.75 1877.17 1909.58</span></span>
<span class="giallo-l"><span>info: total 1 7.78 7775.04 7775.04</span></span></code></pre><table><thead><tr><th>Phase</th><th>What Happens</th></tr></thead><tbody>
<tr><td><code>init</code></td><td>Load module, initialize data structures</td></tr>
<tr><td><code>analysis</code></td><td>Main function traversal + ownership tracking</td></tr>
<tr><td><code>detect</code></td><td>Violation detection (leaks, double-free, UAF)</td></tr>
<tr><td><code>total</code></td><td>End-to-end wall clock time</td></tr>
</tbody></table>
<h4 id="final-summary">Final Summary</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>info: [INFO] Analysis complete</span></span>
<span class="giallo-l"><span>info: [INFO] Functions processed: 35</span></span>
<span class="giallo-l"><span>info: [INFO] Facts generated: 54</span></span>
<span class="giallo-l"><span>info: [INFO] Time: 24ms</span></span>
<span class="giallo-l"><span>info: [INFO] Issues detected: 25</span></span></code></pre><table><thead><tr><th>Field</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>Functions processed</code></td><td>Number of functions analyzed (after zone/noise filtering)</td></tr>
<tr><td><code>Facts generated</code></td><td>Number of facts emitted to the fact store (alias, taint, ownership)</td></tr>
<tr><td><code>Time</code></td><td>Total analysis wall clock time</td></tr>
<tr><td><code>Issues detected</code></td><td>Total issues found (this is what the red team counts)</td></tr>
</tbody></table>
<hr />
<h2 id="part-2-blue-team-false-positive-audit">Part 2: Blue Team — False Positive Audit</h2>
<h3 id="what-it-does-1">What It Does</h3>
<p>Runs OmniScope against the <code>corpus/{small,medium,ffi-dense}</code> test files and compares detected issue counts against expected in-scope thresholds. Measures <strong>precision</strong> — whether the tool over-reports.</p>
<h3 id="run-log-real-output-1">Run Log (Real Output)</h3>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>╔════════════════════════════════════════════════════════════════╗</span></span>
<span class="giallo-l"><span>║ BLUE TEAM TEST ║</span></span>
<span class="giallo-l"><span>║ Defensive: false positive audit on corpus ║</span></span>
<span class="giallo-l"><span>╚════════════════════════════════════════════════════════════════╝</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>── small/ (expected ≤13 in-scope issues)</span></span>
<span class="giallo-l"><span> ✅ small/: 11 issues (expected ≤13, ok)</span></span>
<span class="giallo-l"><span>── medium/ (expected ≤20 in-scope issues)</span></span>
<span class="giallo-l"><span> ✅ medium/: 16 issues (expected ≤20, ok)</span></span>
<span class="giallo-l"><span>── ffi-dense/ (expected ≤26 in-scope issues)</span></span>
<span class="giallo-l"><span> ✅ ffi-dense/: 28 issues (expected ≤26, OVER)</span></span>
<span class="giallo-l"></span>
<span class="giallo-l"><span>────────────────────────────────────────────────────────</span></span>
<span class="giallo-l"><span> Blue Team: 2 passed, 1 failed</span></span>
<span class="giallo-l"><span> ⚠️ False positive regression detected</span></span></code></pre><h3 id="line-by-line-interpretation-1">Line-by-Line Interpretation</h3>
<h4 id="directory-header">Directory Header</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span>── small/ (expected ≤13 in-scope issues)</span></span></code></pre>
<p>This tells you which corpus directory is being tested and the expected upper bound. The bounds come from <code>corpus/EXPECTED_RESULTS.md</code>:</p>
<table><thead><tr><th>Directory</th><th>Expected In-Scope</th><th>Source</th></tr></thead><tbody>
<tr><td><code>small/</code></td><td>13 issues (4 files × ~3 issues each)</td><td><code>EXPECTED_RESULTS.md:198</code></td></tr>
<tr><td><code>medium/</code></td><td>20 issues (1 file, 14 in-scope + 6 out-of-scope)</td><td><code>EXPECTED_RESULTS.md:202</code></td></tr>
<tr><td><code>ffi-dense/</code></td><td>26 issues (4 files, 4+6+6+6 in-scope)</td><td><code>EXPECTED_RESULTS.md:204-207</code></td></tr>
</tbody></table>
<h4 id="pass-fail-result">Pass/Fail Result</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span> ✅ small/: 11 issues (expected ≤13, ok)</span></span></code></pre><table><thead><tr><th>Symbol</th><th>Meaning</th></tr></thead><tbody>
<tr><td><code>✅</code></td><td>Issue count is within expected bound</td></tr>
<tr><td><code>❌</code></td><td>Issue count exceeds expected bound (possible false positive regression)</td></tr>
</tbody></table>
<p>The threshold has a <strong>margin of ~50%</strong> above expected to account for legitimate detections that may vary across versions. The point is to catch <strong>gross inflation</strong>, not minor fluctuations.</p>
<h4 id="summary">Summary</h4>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="plain"><span class="giallo-l"><span> Blue Team: 2 passed, 1 failed</span></span>
<span class="giallo-l"><span> ⚠️ False positive regression detected</span></span></code></pre>
<p>If any directory fails, the overall blue team result is a warning. Investigate by running OmniScope on individual files:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">./zig-out/bin/OmniScope</span><span style="color: #A6E3A1;"> corpus/ffi-dense/output/sqlite_binding.ll</span><span style="color: #94E2D5;"> 2>&1</span></span></code></pre>
<p>Compare the issue list against <code>corpus/EXPECTED_RESULTS.md</code> to identify which issues are real and which are false positives.</p>
<hr />
<h2 id="part-3-contributor-reference-omniscope-internals">Part 3: Contributor Reference — OmniScope Internals</h2>
<blockquote>
<p><strong>Note</strong>: This section is for OmniScope developers/contributors. If you're a <strong>user</strong> trying to find issues in your own code, see the <a href="https://Timwood0x10.github.io/myblog/log/OmniScope/08-log-read/#vulnerability-lines">Vulnerability Lines</a> section above.</p>
</blockquote>
<h3 id="where-each-log-line-originates-omniscope-source">Where Each Log Line Originates (OmniScope Source)</h3>
<table><thead><tr><th>Log Pattern</th><th>OmniScope Source File</th><th>Function</th></tr></thead><tbody>
<tr><td><code>=== OmniScope IR Analysis ===</code></td><td><code>src/main.zig</code></td><td><code>runSingleFileAnalysis</code></td></tr>
<tr><td><code>LANG-DETECT: module language</code></td><td><code>src/semantics/language_detector.zig</code></td><td><code>detectModuleLanguage</code></td></tr>
<tr><td><code>MallocCheck: Analyzed functions</code></td><td><code>src/pass/analysis/malloc_check.zig</code></td><td><code>run</code></td></tr>
<tr><td><code>IntegerOverflow: Analyzed functions</code></td><td><code>src/pass/analysis/issue/integer_overflow.zig</code></td><td><code>run</code></td></tr>
<tr><td><code>CallGraph: extracted N cross-language edges</code></td><td><code>src/pass/analysis/call_graph.zig</code></td><td><code>extractCrossLangEdges</code></td></tr>
<tr><td><code>DangerSurfacePass: N FFI</code></td><td><code>src/pass/analysis/danger_surface.zig</code></td><td><code>run</code></td></tr>
<tr><td><code>FFITypeMismatch: analyzed N calls</code></td><td><code>src/pass/analysis/ffi_type_mismatch.zig</code></td><td><code>run</code></td></tr>
<tr><td><code>PointerOwnership: Source 1</code></td><td><code>src/pass/analysis/pointer_ownership.zig</code></td><td><code>run</code> (line 265)</td></tr>
<tr><td><code>VULNERABILITY OMI-NNN</code></td><td><code>src/diag/issue.zig</code></td><td><code>Issue.init</code></td></tr>
<tr><td><code>Issues detected: N</code></td><td><code>src/main.zig</code></td><td><code>emitOutput</code></td></tr>
<tr><td><code>GlobalAllocTracker: N memory leaks</code></td><td><code>src/semantics/global_alloc_tracker.zig</code></td><td><code>confirmLeaks</code></td></tr>
</tbody></table>
<h3 id="issue-type-detection-pass-omniscope-internals">Issue Type → Detection Pass (OmniScope Internals)</h3>
<table><thead><tr><th>Issue Type</th><th>Detection Pass</th></tr></thead><tbody>
<tr><td><code>memory_leak</code></td><td><code>pointer_ownership.zig:detectViolations</code></td></tr>
<tr><td><code>cross_lang_free_mismatch</code></td><td><code>cpp_fp_reduction.zig:detectCrossLangAllocMismatch</code></td></tr>
<tr><td><code>use_after_free</code></td><td><code>cpp_fp_reduction.zig:detectUseAfterFree</code></td></tr>
<tr><td><code>double_free</code></td><td><code>cpp_fp_reduction.zig:detectDoubleFree</code></td></tr>
<tr><td><code>null_dereference</code></td><td><code>pointer_ownership.zig:detectNullDereferences</code></td></tr>
<tr><td><code>borrow_escape</code></td><td><code>pointer_ownership.zig:detectAsPtrBorrowEscape</code></td></tr>
<tr><td><code>tainted_path_to_sink</code></td><td><code>taint.zig</code></td></tr>
<tr><td><code>ffi_unsafe_call</code></td><td><code>ffi_boundary.zig:checkCallForFFI</code></td></tr>
<tr><td><code>buffer_overflow</code></td><td><code>buffer_overflow.zig</code></td></tr>
<tr><td><code>integer_overflow</code></td><td><code>issue/integer_overflow.zig</code></td></tr>
<tr><td><code>format_string</code></td><td><code>format_string.zig</code></td></tr>
</tbody></table>
<h3 id="corpus-file-source-code-mapping">Corpus File → Source Code Mapping</h3>
<table><thead><tr><th>IR File</th><th>Source File</th><th>Bugs Injected</th></tr></thead><tbody>
<tr><td><code>red_team_bugs_O0.ll</code></td><td><code>red_team_bugs.c</code></td><td>Memory leak, UAF, double-free, null deref, format string</td></tr>
<tr><td><code>ffi_boundary_bugs.ll</code></td><td><code>ffi_boundary_bugs.c</code></td><td>FFI boundary violations, stack escape</td></tr>
<tr><td><code>cross_lang_free_bugs.ll</code></td><td><code>cross_lang_free_bugs.c</code></td><td>Rust→C, C→C++ free mismatches</td></tr>
<tr><td><code>subtle_ffi_bugs.ll</code></td><td><code>subtle_ffi_bugs.c</code></td><td>Subtle FFI patterns (realloc, partial cleanup)</td></tr>
<tr><td><code>python_c_api_bugs.ll</code></td><td><code>python_c_api_bugs.c</code></td><td>Python C API misuse (Py_DECREF, refcount)</td></tr>
<tr><td><code>posix_ffi_bugs.ll</code></td><td><code>posix_ffi_bugs.c</code></td><td>POSIX API misuse (FILE*, fd leak)</td></tr>
<tr><td><code>cpp_ffi_simple.ll</code></td><td><code>cpp_ffi_simple.cpp</code></td><td>C++ new/delete mismatch, RAII escape</td></tr>
<tr><td><code>boundary_test.ll</code></td><td><code>boundary_test.c</code></td><td>Null deref at FFI boundary, circular ownership</td></tr>
<tr><td><code>stress_patterns.ll</code></td><td><code>stress_patterns.c</code></td><td>70 bugs: alloc leaks, cross-lang mismatches, chains</td></tr>
<tr><td><code>sqlite_binding.ll</code></td><td><code>sqlite_binding.c</code></td><td>SQLite resource leaks, dangling pointers</td></tr>
<tr><td><code>openssl_wrapper.ll</code></td><td><code>openssl_wrapper.c</code></td><td>OpenSSL ctx/bio/key leaks</td></tr>
<tr><td><code>zlib_binding.ll</code></td><td><code>zlib_binding.c</code></td><td>zlib stream leaks, double-free, UAF</td></tr>
</tbody></table>
OmniScope Design Concept
2025-01-01T00:00:00+00:00
2025-01-01T00:00:00+00:00
Unknown
https://Timwood0x10.github.io/myblog/log/OmniScope/000-design/
<h1 id="omniscope-design-concept">OmniScope Design Concept</h1>
<p>Hi, previously, I only gave a brief overview of this project. Now, I'll break down the architecture and share my design insights.</p>
<p>This project aims to solve memory leak issues in cross-language calls. Unlike general-purpose security detection tools, my design focuses 90% of my effort on analyzing unsafe/ffi code, and only 10% (and perhaps even less) on general security detection.</p>
<h2 id="why-zig">Why zig?</h2>
<p>Initially, I didn't know what to do, choosing between Rust and C++. However, Rust's compilation speed was too slow, and although it has an excellent memory management model, analyzing unsafe code might require a lot of unsafe code, which deterred me. As for C++, I'm sorry, I'm not very good at it. So I switched to Zig, and this was my first attempt at creating a small tool using Zig.</p>
<h2 id="why-ir">Why IR?</h2>
<p>I've been thinking that many FIF implementations aren't limited to Rust ←-> C, but also include C ←> C++, and Go ← → C. If I were to adapt for each language, it would be a catastrophic workload. So, I realized that the commonality among these languages is that they all use LLVM as their compiler backend. Therefore, I started researching (yes, I'm not a very competent developer; I don't even understand compilers very well). I dissected the compiler and realized that the Instructions Repository (IR) contains a large amount of information about the program; this might be my breakthrough point.</p>
<p>Then I started designing the architecture. Besides researching on my own, I consulted senior development engineers and various AIs (GPT, CC). I started by creating a simple demo, for example, zig ←-> C. So I looked at the zig source code to find out how zig and C interact… and did all sorts of other work, also taking various compiler notes (but that's another story). Then the first version of the code was runnable. I also designed various bugs in the project, namely red team vs. blue team battles, to test the TP and FP of this demo. The first run results were pretty good, at least not too bad, which was a good start for me.</p>
<h2 id="how-to-abstract">How to abstract?</h2>
<p>After successfully running the Zig←->C demo, my confidence soared, and I switched to Rust←->C. Little did I know this would be the beginning of a nightmare. The Rust compiler generated a massive amount of intermediate results, storing them all in the Instructions Redirect (IR). The project's TP remained consistently low, below 20%, and I even considered giving up at one point.</p>
<p>For a long time afterward, I didn't want to touch this project until I thought of something: since so many intermediate products are generated, why don't I just ignore them, assume they are trustworthy, and only analyze the content in the user-defined program?</p>
<p>Then, I excitedly opened the Rust source code and, with the help of AI, read the Rust compiler source code. The goal was to find what the compiler produced, and use that as a safe zone (this is how Java, Python, etc., were designed later).</p>
<h2 id="why-memory-diagram">Why Memory diagram?</h2>
<p>With this idea in mind, I started working on the project. Zone filtering was successful, but the TP (True Positive Rate) was still low. After analysis, I realized the issue was with the program's context awareness. I consulted the AI, which suggested adding a whitelist, a suggestion I initially agreed to. However, after implementing it, I discovered that maintaining a very large whitelist was necessary, which wasn't what I wanted. So I went back to my previous notes…</p>
<p>Since there's a deficiency in context awareness, I might as well build a memory graph and a call graph during IR scanning, recording the complete call stack, and then analyze the graph. I'll only focus on the FHI part; I don't care about the other parts.</p>
<p>I got right to it, and the results were surprisingly good. My confidence soared, and I started challenging existing open-source projects, relentlessly adding various red team and blue team tests.</p>
<p>From the initial sqlite3 to various Rust-based cryptographic libraries and game libraries, all the actual analysis data is documented in the project documentation.</p>
<p>When memory-safe languages meet legacy C code, the safety net usually tears at the FFI boundary. OmniScope is built to continuous-stitch this gap.</p>
<p>This is the entire development process of the project. I hope this tool can be helpful to you. It may not be perfect, but your feedback is appreciated. Thank you.</p>