Agent Communication — Agent IPC Primitives and the Syscall Tool Layer

Note: This article is grounded in the actual code (internal/agentipc — bus.go / primitives.go / policy.go / deadletter.go / collaboration_observer.go — and internal/agentsyscall — syscall.go / plan.go), the dedicated Agent-communication article in the docs series. Agents are peer cognitive processes (A ≡ B ≡ C); communication runs over a peer-mesh message bus, no Leader relay required.

1. Why Agent Communication Matters

In a multi-agent system agents must exchange messages — "I found X" / "help me verify Y" / "your conclusion conflicts with mine" (the archetypal collaboration phrases listed in internal/agentipc/doc.go). This is ARES's Kernel IPC pillar (P4), and the third of the three context layers (Task Shared / Agent Private / IPC Messages) — the first two live in internal/fabric/agent; this layer is carried by this package.

Let's first separate duties: task dispatch is the Scheduler's (Task Fabric) job; agent communication is IPC's job. They are not the same thing:

Core principle: agents express intent (Send / Request / Delegate / Handoff / Subscribe); the Kernel guarantees delivery. Communication is a "collaboration channel", not a "task execution channel".

2. The Native Message Unit: Message

Every IPC message uses Message from internal/agentipc/bus.go (design Section 13: Context layer 3 — IPC Messages):

type Message struct {
    ID            string    // unique message ID generated by the bus
    From          string    // sender agent ID
    To            string    // target agent ID ("" = broadcast to subscribers)
    Topic         string    // message subject (e.g. "verify-conclusion" / "handoff-task")
    CorrelationID string    // correlation id pairing a reply with its request
    Payload       any       // message body
    At            time.Time // send timestamp
}

The paired delivery function and sentinel errors (Handler, ErrAgentNotRegistered, ErrNoHandler, ErrTimeout, ErrInvalidMessage) are defined in the same package.

3. The Six Primitives: Bus

The core object is Bus (internal/agentipc/bus.go): a peer message bus mapping agent IDs to handlers. It maintains handlers, subscribers (topic → subscriber list), and pending / pendingErr (per-correlation reply channels / stashed errors). The full primitive set:

Note: doc.go frames the set as Send / Request / Delegate / Handoff / Subscribe (+ Reply); the implementation adds Broadcast / Unsubscribe as supplementary entry points, listed all together here.

graph TD
    A[Initiator Agent] -->|Send / Broadcast| B[Bus]
    A -->|Request / Delegate / Handoff| B
    B -->|Send: sync handler call, no wait| C1[Target handler]
    B -->|Request: allocate corrID + reply channel| C2[Target handler]
    B -->|Broadcast: fan out by topic| C3[Subscriber handlers]
    C2 -->|Reply same corrID| A
    C1 -->|result vs failure| D[DeadLetterStore / CollaborationObserver]
    C3 -->|delivery count| A

Send vs Request: Two Collaboration Stances

• Send(ctx, from, to, topic, payload) error — fire-and-forget. The target handler is invoked synchronously in the caller's goroutine; a handler error is surfaced but no reply channel is set up — "fire and done", no paired reply.
• Request(ctx, from, to, topic, payload, timeout) (*Message, error) — synchronous request/reply. The bus allocates a correlation ID and registers a pending reply channel; the target handler must call Reply with the same correlation ID to complete the request. Timeout has a sane default: a timeout <= 0 falls back to defaultRequestTimeout = 30 * time.Second (primitives.go, B16 — no indefinite blocking). The request runs in a managed goroutine with a recover boundary — if the handler panics, it is contained as one ErrHandlerPanic, failing exactly that single request rather than the whole process.

4. Delegate and Handoff

• Delegate(ctx, delegator, to, topic, payload, timeout) — forwards a request to another agent on the caller's behalf. The original requester's correlation ID is preserved end-to-end so the reply chains back. Semantics: "I can't handle this — let me ask someone who can".
• Handoff(ctx, from, to, taskID, contextSnapshot, timeout) — transfers a task's ownership from a yielding agent to an accepting agent. Payload is {task_id, context, artifacts} on the constant topic handoff-task; the receiver returns an acceptance reply. Key point: this is a peer-to-peer task-transfer primitive — it does NOT go through the Scheduler.

5. Publish/Subscribe: Subscribe / Broadcast / Unsubscribe

func (b *Bus) Subscribe(agentID, topic string) error   // idempotent: duplicates de-duplicated
func (b *Bus) Unsubscribe(agentID, topic string)
func (b *Bus) Broadcast(ctx context.Context, from, topic string, payload any) int

Broadcast fans one message out to every subscriber's handler for a topic; a handler error does not stop the rest, and the successful-delivery count is returned. Subscribe de-duplicates the same agent on the same topic (B16).

6. Observability: DeadLetterStore and CollaborationObserver

IPC is not just communication — it is one input to the evolution loop (N-11 / GAP-3 closure):

• DeadLetterStore (deadletter.go): a bounded FIFO recording failed deliveries (ErrAgentNotRegistered / ErrTimeout / handler errors). Default capacity 1024 (capacity <= 0 falls back). Record (oldest evicted), Snapshot, Count serve diagnosis and redelivery. Both Send and Request record failures.
• CollaborationObserver (collaboration_observer.go): each collaboration attempt with an observable receipt emits one feedback.CollaborationOutcome. Send and Request (and the Delegate / Handoff built on them) are observed; Broadcast is deliberately NOT observed (no single target to attribute the outcome to). The observer is invoked on the initiator's path, so it MUST NOT block — and it is called outside the bus lock.

graph LR
    S[Initiator] -->|Send / Request failure| DL[DeadLetterStore bounded FIFO]
    S -->|Send / Request receipt| OBS[CollaborationObserver]
    OBS --> FB[feedback source: collaboration]
    DL -->|Snapshot / redelivery| OPS[diagnosis / ops tooling]

7. The Syscall Layer: Kernel and Tools

internal/agentsyscall wraps IPC/collaboration capability into LLM-callable tools (W2 design: the real LLM executor decides whether to split work; the Kernel validates and enforces — "Agent decides. Kernel enforces."). BindTools(binder, kernel) registers four tools:

ToolSchemas() returns the LLM-facing schemas for these tools and injects them into resources.Registry, alongside the built-in tools in the Chat API.

The Kernel's AskAgentFn is the key injection point (syscall.go):

type AskAgentFn func(ctx context.Context, from, to, topic string, payload any) error

// WithAskAgent injects the collaboration primitive; SetAskAgent can replace it during
// serve assembly (after setupPeerRegistry). Production wire-up:
// aresrecovery.EvolutionAwareIPC.Send → agentipc Bus.

ask_agent is the ACT half of the collaboration channel (Step Y.2-ACT): it turns "which agent to ask" into an agent-visible, changeable decision. Kernel.AskAgent requires a non-empty target and a wired primitive (fail-loud if absent — never a silent no-op). Note its semantics are a fire-and-forget send: AskAgentResult.Accepted means "the request was handed to the collaboration primitive", not "an answer is in hand".

graph LR
    LLM[LLM Agent] -->|ask_agent tool call| Bind[BindTools binding]
    Bind --> K[Kernel.AskAgent]
    K -->|AskAgentFn: ipc.Send| BUS[agentipc Bus.Send]
    BUS --> H[Target agent handler]
    H -->|receipt| OBS[CollaborationObserver]
    OBS --> FB[collaboration feedback source]

8. Single-Track Dispatch: the dual-track machinery is retired

ExecutionPolicy / PolicyFlag / shadow mode / Mismatches() and the whole facade in policy.go (Dispatcher / KernelDispatcher / NewPath / SetNewPath) are gone — internal/agentipc no longer carries any "how a task is dispatched" abstraction. HTTP submits tasks to the Task Fabric directly (submitPeerTask / POST /api/graphs) and the kernelScheduler is the sole executor (Schedule→Acquire→RunQuantum). The kernel-bridge dispatch layer that only tests ever invoked (wireKernelDispatcher / enableKernelExecution / submitFabricTask) was removed along with it.

9. Summary

Design line: agents express intent, the Kernel guarantees delivery, and the evolution loop stays observable. The communication primitives are the peer collaboration layer; task dispatch is the scheduling execution layer — orthogonal. And ask_agent turns a "collaboration intent" into a real, injectable, observable, evolvable tool, wiring the IPC from a library primitive into the agent's cognition loop.