Skip to content

Production Network Transport: Design History

Status: historical and superseded. The work this document scoped was cancelled by the sibling-network-surfaces decision in ROADMAP.md (2026-07): std.Io.net is the literal same-code socket seam, typed endpoints are experimental message-modeling tools, and Marionette will not ship its own production socket bus. This file is retained as design history because its analysis (framing, pooling, reconnect, TigerBeetle's MessageBus) informed that decision. The experimental adapters and their private socket runtime were removed in 0.6. Nothing below is planned work. Sections below retain the terminology and proposed phases of that historical design; descriptions of files or APIs are not statements about the current tree. The current API direction lives in docs/network-api.md.

This document was a forward-looking plan for the target shape of Marionette's production-side Endpoint(Message), had it grown from a same-process FIFO into a real cross-process transport (formerly roadmap item 15, "App-facing typed network composition: production transport").

The current simulation network is documented in docs/network.md; production code uses host std.Io.net. This file covers only the removed, counterfactual production-side architecture.

Why this is a real engineering project

Marionette's parity claim is "the same code runs in production and under the simulator." Today that claim is true only inside a single OS process: a sender's bytes reach a receiver because both share an in-memory queue. The moment a service uses Marionette across hosts, parity breaks.

Closing that gap requires a production transport that:

  • accepts a topology of peer addresses at startup,
  • frames messages on the wire so torn reads cannot cross a message boundary,
  • pools and bounds the memory used for in-flight messages,
  • manages outbound connections with reconnect and backoff,
  • exposes the same Endpoint(Message) vtable that simulation already exposes,
  • preserves the application code path so production-shaped code never branches on whether the network is real.

This is not a small library. TigerBeetle's MessageBus is roughly a thousand lines of Zig dedicated to exactly this problem, and that count excludes their IO backend.

Reference: TigerBeetle's MessageBus

The closest fully-realized example in the Zig ecosystem. Studied in detail in docs/tigerbeetle-lessons.md lesson 15. Quick recap of the shape:

  • One MessageBusType(IO) per process, parametric on an IO backend so fuzz tests can swap a deterministic IO.
  • Identity is asymmetric: ProcessID = union { replica: u8, client: u128 }.
  • Topology declared at init as a list of network addresses (the Zig 0.16 equivalent is []const std.Io.net.IpAddress).
  • Wire format: u32 size, header checksum, body checksum, VSR header.
  • Pool of refcounted messages preallocated at startup; sender obtains via bus.get_message, fills, calls send_message_to_replica or send_message_to_client.
  • Send drops silently on full per-peer queue or no connection. Caller retries.
  • Receive is callback-driven: bus invokes on_messages_callback(bus, buffer), application iterates the buffer and either consumes or suspends each message.
  • Connection management is internal: lazy outbound connect, exponential backoff with seeded jittered delay, async close that waits for in-flight recv/send to complete.
  • VOPR uses a separate, simpler testing/cluster/message_bus.zig that bypasses sockets and routes through an in-memory packet simulator.

Marionette's seam choice

There are actually two seams hiding in this question, and they have different answers.

Public-API seam (settled). The user-visible Endpoint(Message) type will not be parametric on an IO backend. The vtable already gives the sim/prod swap; adding Endpoint(Message, IO) would push the IO choice into every function signature that takes an endpoint and leak library internals into application code. The same Endpoint(Message) type is satisfied by two impls: a simulation impl backed by the deterministic packet bus, and a production impl backed by sockets. Production transport work would have gone behind the endpoint vtable in src/network/endpoint.zig, with a private production runtime. No type signatures would have changed in user-facing code.

Implementer-internal seam (deferred, not foreclosed). Whether the production-side bus implementation is itself parametric on an internal IO type is a separate question. TigerBeetle's MessageBusType(IO) is not primarily for swapping sim and prod (VOPR uses a different MessageBus entirely); it is for message_bus_fuzz, which exercises the real production MessageBus code against a deterministic test IO that simulates partial reads, EOF mid-frame, EAGAIN, and similar IO-edge behaviors. That coverage is not optional in a correctness-focused project: framing, recv-buffer reassembly, and connection state-machine code only ever runs against real sockets without it.

The vtable seam does not deliver this coverage. Once code is on the prod side of Endpoint(Message)'s vtable, syscalls are wired in directly. The pragmatic plan is to leave room for an internal IO seam without committing to one yet:

  • Sub-tasks 15a (framing) and 15b (buffer pool) are already independent of IO and are fuzzable directly.
  • The socket transport, connection-management, reconnect, and queueing steps should structure their IO calls behind a small internal abstraction, even if the only impl is the real one. That keeps a future deterministic IO impl cheap to add.
  • A decision on whether to actually ship a deterministic IO + bus-impl fuzzer is deferred until 15d lands and we know what the IO surface looks like in practice. At that point we either commit to internal parameterization or drop the option.

The user-visible API is unaffected by this question either way.

Why this differs from std.testing.io

std.Io is a host-I/O execution surface. It is useful for production adapters and for ordinary unit tests that should use Zig's testing backend instead of a real process-global backend. Marionette already uses this shape for RealDisk: production setup receives std.Io, and RealDisk routes host file operations through it.

That is not the same thing as deterministic simulation. SimDisk does not use std.testing.io to fake crashes, torn writes, latency, or corrupt sectors. It owns an in-memory disk model, routes every fault decision through World, and exposes only the app-facing Disk capability to service code. std.testing.io helps test the production disk adapter; SimDisk is the simulator.

The network plan follows the same split:

  • Endpoint(Message) is the app-facing capability shared by sim and prod.
  • TypedRuntime(Message) is the cheap deterministic simulator bus owned by World.
  • The future production bus should use an internal IO backend for sockets.
  • A fake/deterministic IO backend may later test the production bus itself, especially partial reads, reconnects, EOF mid-frame, and queue behavior.

Normal cluster simulation should not route through a fake socket stack by default. VOPR makes the same tradeoff: the production MessageBus can be fuzzed with a deterministic IO seam, while full-cluster simulation uses a simpler in-memory message bus.

Target architecture

Wire format

Length-prefixed framing with header and body checksums:

+------------+--------------------+-----------------+--------+--------+----------+--------+
|  size:u32  | checksum_header:u128 | checksum_body:u128 | from:u16 | to:u16 | reserved:u32 | payload |
+------------+--------------------+-----------------+--------+--------+----------+--------+

Constraints:

  • size is the total frame size including header, in bytes. Receivers reject frames whose declared size exceeds a configured maximum.
  • Header checksum covers everything from from through the payload length; body checksum covers the payload bytes. Two checksums let us reject torn headers without reading the body.
  • from and to are NodeId values matching the topology config.
  • reserved accommodates a future framing version. v0 sets it to zero.

Checksum function: SipHash-128 with a per-process seed is the working default. The framing is extension-shaped so the choice can change without breaking the surface API.

Payload encoding is the user's responsibility. For fixed-shape Zig structs, std.mem.toBytes is sufficient. Variable-length payloads require a user-supplied encode/decode pair; this is deferred until a real example demands it.

Topology declaration

The item-15 target production setup uses a topology config already accepted by the current in-process stub:

const endpoint = try production.endpoint(Message, .{
    .self = 1,
    .peers = &.{
        .{ .id = 0, .address = "127.0.0.1:4240" },
        .{ .id = 1, .address = "127.0.0.1:4241" },
        .{ .id = 2, .address = "127.0.0.1:4242" },
    },
    .listen = "0.0.0.0:4241",
});

Simulation does not need this; World.simulate(.{ .network = .{ .nodes = N } }) declares topology implicitly. The experimental production adapters once accepted a peer-table shape here; those adapters no longer exist.

This will be a deliberate divergence between production and simulation setup: production declares peer addresses and listener state, while simulation declares a node count. The returned endpoint shape is the same. Application code that holds an Endpoint(Message) is unaware of how its peers were declared.

Why endpoints stay above listen/connect/read/write

Marionette simulates the app's distributed effect boundary, not the host socket syscall API. App code should own a node-scoped endpoint and exchange messages; the production bus can then decide whether those messages are delivered by an in-memory simulator, a fake production IO backend, or real sockets.

Putting listen/connect/read/write directly in app code would make every app own framing, buffering, partial reads, reconnects, and stream lifetimes. It would also make the simulator mimic a byte stream instead of the thing Marionette needs to control: message delivery, drops, delay, partitions, and process identity. The stream seam still exists, but it is internal to the production transport where it can be tested with fake IO.

Identity model

Single NodeId = u16 namespace, matching the simulator. Marionette will not adopt TigerBeetle's asymmetric replica/client identity model at the bus layer. Users who need a replica/client distinction encode it in their Payload.

Rationale: TigerBeetle conflates client UUIDs and replica indices because VSR requires it. Marionette is a generic library and should not constrain payload shape. If a future use case demands UUID-shaped clients on the bus, we will add it as a separate primitive rather than retrofit it.

Buffer pool

Preallocated, refcounted message pool. Pool size is a config field; the default is computed from the topology and per-peer queue depths. Pool exhaustion returns error.PoolExhausted from send; this is one of the few conditions that propagates as a hard error rather than dropping silently, because it represents a configuration bug, not a transient condition.

User-facing API for v1 is copy semantics: endpoint.send(to, message) copies the payload into a pooled buffer. Zero-copy primitives such as acquire/ commitSend are deferred until a workload justifies them.

Connection management

Internal to the production impl. Outbound: lazy connect on first send to a peer; persistent connections; exponential backoff with seeded jittered delay on failure. The jitter seed comes from the local NodeId so that multiple replicas reconnecting to a flapping peer naturally desynchronize.

Inbound: a single listener bound to options.listen accepts connections. Peer identity is inferred from the first valid frame's source field. The application cannot choose that source per send; it is fixed by the endpoint's configured self identity.

Async close: when a connection terminates, pending recv and send completions must drain before the socket file descriptor is released. This prevents TOCTOU races and dropped data on graceful shutdown.

Backpressure

Send side: bounded per-peer queue. Default capacity TBD; configured via options. Full queue drops the message and emits a trace event; the caller does not receive an error. This matches TigerBeetle. The application owns retries.

Receive side: bounded per-connection recv buffer. When the buffer is full, the kernel read on that connection is paused until receive consumes a message. The pull-shaped receive API gives application backpressure for free.

Marionette will not implement TigerBeetle's per-message suspend_message semantics in v1. The pull model plus bounded recv buffer is simpler and preserves the existing API. If a workload demands fine-grained per-message deferral, we add it then.

Send semantics

try endpoint.send(to, message);

Returns !void. Possible errors:

  • error.PoolExhausted: the message pool is full. Configuration bug; this is not a transient condition.
  • error.InvalidNode: to is not in the topology. Caller bug.

Does not return errors for transient conditions:

  • Peer is unreachable: drop silently, emit network.drop reason=peer_down trace event. Caller retries at the application layer.
  • Per-peer queue is full: drop silently, emit network.drop reason=queue_full trace event. Caller retries.

This converges sim and prod on the same surface. The current sim error.EventQueueFull becomes a silent drop with a trace event in both impls. The reconciliation lands as a small follow-up to the simulation network code.

Receive semantics

while (try endpoint.receive()) |envelope| { ... }

Returns !?Envelope(Message). The production impl fills pooled recv buffers from sockets and hands the next available message for this endpoint's node to the caller. When no message is currently deliverable, the production impl yields control to the IO backend (i.e. blocks on a poll) until one arrives or until a configured deadline expires; the deadline is exposed via a future receiveWithin(duration) variant if needed.

For v1, the socket-backed receive blocks indefinitely until a message is available or the endpoint is closed. The current in-process production stub is non-blocking and returns null when no queued message is addressed to the endpoint. Callers that need an explicit production poll API use a future trySend / tryRecv pattern, deferred until a real example asks for it.

IO backend

Target: std.Io once it stabilizes in Zig 0.16 or later. Until then, a thin internal abstraction with one impl per platform: io_uring on Linux, kqueue on Darwin, IOCP on Windows.

The internal IO type is not part of the public API. Users see only Endpoint(Message). The decision to migrate to std.Io happens when std.Io exposes the primitives the production impl needs (async accept, recv, send, close); until then, the internal abstraction is whatever works.

The internal IO surface should be small and well-defined for a second reason beyond std.Io migration: structured this way, the production bus implementation can later be made parametric on the internal IO type so a deterministic IO impl can drive bus-implementation fuzz tests (TigerBeetle's message_bus_fuzz pattern). This is not committed work for v1, but the cost of structuring the IO calls cleanly now is small and preserves the option. See "Marionette's seam choice" above for the full reasoning.

Implementation order

Each item ships on its own. Cross-process parity is the done-signal.

  1. Wire format and framing primitive. Encode/decode helpers, checksum, roundtrip tests. No sockets yet. Lives in src/network/frame.zig.
  2. Buffer pool primitive. Refcounted, preallocated pool. Pool exhaustion returns hard error. Lives in src/message_pool.zig. Used by both sim and prod once integrated.
  3. Topology config and Production.endpoint(Message, opts). Historical prototype, removed in 0.6. Production endpoint accepts peers and self id. It still returns the same in-process FIFO behavior as today; the topology API change is the gate.
  4. ByteEndpoint bridge. Started. ByteEndpoint exists for sim and production setup with explicit ownership semantics: send copies borrowed bytes, sendMessage transfers an acquired buffer, and receive returns a releasable byte message. This gives future Zig network/RPC libraries a Marionette backend target without forcing them to use typed value-message endpoints directly; encoding above the byte surface is the library's own job (the codec/transport wrappers were removed in 0.6).
  5. Internal network IO seam. Historical prototype, removed in 0.6. It covered listen/connect/read/write/close/time plus exact transfer helpers and a fake backend for short I/O and close behavior.
  6. Single-peer end-to-end socket transport. Historical prototype, removed in 0.6. It reached framed loopback sockets but not reconnect, background receive, or multi-peer connection management.
  7. Production-bus fake IO tests. Add a small deterministic/fake IO backend for the production bus, focused on partial frame reads/writes, EOF mid-frame, reconnect timing, and close discipline. This tests production bus machinery without replacing normal World simulation.
  8. Multi-peer with internal connection management. Lazy outbound connect, inbound listener, peer-type resolution from frames.
  9. Reconnect with seeded jittered backoff. Connection drop and recovery tested end to end. Jitter seed comes from the local NodeId.
  10. Bounded send and recv queues with silent-drop semantics. Sim reconciles to the same drop semantics as part of this step.
  11. Cross-process parity test. The replicated-register example runs on N OS processes. Same source, same scenario, real network. This is the done-signal that closes roadmap item 15.

Steps 1 and 2 are independent of the rest and can be picked up in parallel. Steps 3-11 are sequential, except that the fake IO tests can grow incrementally alongside steps 6-9 once the internal seam exists.

Open questions

These are deferred until implementation forces an answer. Recording them so they do not get rediscussed.

  • Checksum function specifics: SipHash-128 is the working default but not yet settled.
  • Pool sizing: is the default computed from topology automatically, or must the user configure it explicitly? TigerBeetle computes from topology; we may need to until users complain.
  • Listener lifetime: bound to Production.deinit, or explicit bind/ shutdown? Current plan is implicit lifetime; revisit if it bites.
  • Variable-length payloads: deferred until a real example demands them.
  • Multiple buses on one process (RPC plus gossip): out of scope here; tracked separately under the named-bus question in docs/network-api.md.

Deliberate non-goals

  • General-purpose RPC: no request/response correlation, no service discovery, no client libraries.
  • TLS, real DNS, arbitrary std.Io.net compatibility.
  • Stream or datagram primitives as part of this production endpoint transport. A separate deterministic std.Io.net stream subset is tracked in docs/std-io-direction.md and the roadmap; it is a simulator surface, not the production Endpoint(Message) transport described here.
  • A drop-in replacement for Tokio, libuv, or any general-purpose async runtime.

If a user needs any of the above, they should use a production host std.Io directly and not use Endpoint(Message) for that workload.

Deviations from TigerBeetle

For posterity, where this design diverges from TigerBeetle and why:

  • One user-visible seam, not two. Marionette's vtable substitutes for the TigerBeetle equivalent of "different MessageBus per use case" (separate VOPR bus vs production bus). Justification: keeps the public type uniform across sim and prod. Note this is narrower than "one seam total": TigerBeetle's parametric MessageBusType(IO) exists primarily to let message_bus_fuzz exercise the production bus against a deterministic IO. Marionette has not adopted that internal seam yet but has not foreclosed on it; see "Marionette's seam choice" above.
  • No bus-level replica/client distinction. Single NodeId namespace. Justification: Marionette is generic, not VSR-specific.
  • Pull-shaped receive (receive). TigerBeetle is callback-shaped. Justification: pull is simpler in the single-threaded simulator; the production impl buffers internally and the user pulls.
  • Sim and prod converge on silent-drop send semantics. TigerBeetle does silent-drop on the prod side; Marionette's sim currently returns error.EventQueueFull. We will converge on the TigerBeetle behavior in both impls.