Roda-Ledger

Roda-Ledger is a high-performance, crash-safe financial ledger engine. It handles the hard parts of financial infrastructure — strict transaction ordering, atomicity, durability, and crash recovery — so you can focus on your business logic.

It is built for engineers who need a ledger they can trust. Not a general-purpose database stretched to fit financial use cases. Not a message queue masquerading as a transaction log. A purpose-built ledger engine, designed from the ground up for correctness and throughput.

It is also designed to get out of your way. Common operations — deposits, withdrawals, transfers — work out of the box. Custom multi-account logic is expressed through registered WASM functions without touching the engine. How much durability you need is your call, per submission: fire and forget for maximum throughput, or wait for disk confirmation when it matters. Run it as a Docker service in minutes, or embed it as a library. The ledger adapts to your use case, not the other way around.

5.7 million transactions per second. Sub-microsecond latency. WAL-backed durability. Zero torn writes.

Quick Start

Get a ledger running in under a minute.

1. Start the server

docker run -p 50051:50051 -v $(pwd)/data:/app/data tislib/roda-ledger:latest

That’s it. The ledger is running, persisting data to ./data, and listening for gRPC connections on port 50051.

2. Submit a deposit

grpcurl -plaintext -d '{"deposit": {"account": 1, "amount": "1000", "user_ref": "123"}}' \
  localhost:50051 roda.ledger.v1.Ledger/SubmitOperation

3. Check the balance

grpcurl -plaintext -d '{"account_id": 1}' \
  localhost:50051 roda.ledger.v1.Ledger/GetBalance

You should see a balance of 1000 for account 1.

Stop the container and start it again — the balance is still there. That is the durability guarantee in action.

Where to Go Next

Roda-Ledger’s documentation is organized into four levels. Start at the top and go as deep as your use case requires.

If you want to understand what roda-ledger guarantees and why, start with Concepts.

If you want to integrate it into your system, start with API.

Concepts

This document establishes the mental model and vocabulary that all other roda-ledger documentation builds on. Read it before anything else. No implementation details are discussed here — only what the system is, what it guarantees, and what it does not do.

What is a Ledger

A ledger is an append-only record of financial facts. In roda-ledger, transactions are the source of truth. Balances are derived state — they are computed by applying all transactions that have touched an account, in order. A balance is never stored as a primary fact; it is always the result of replaying history.

This has a concrete implication: if you lose your balances but keep your transactions, you lose nothing. If you lose your transactions, no balance snapshot can recover them. The log is the ledger.

Accounts and Balances

An account is an identifier — a u64 that represents a participant in the ledger. Accounts are not created explicitly; they come into existence the first time a transaction references them.

A balance is the current state of an account, represented as an i64. Balances can be positive or negative. By default, roda-ledger protects accounts from going below zero — a transaction that would produce a negative balance is rejected. This protection can be intentionally bypassed via WASM-defined functions that explicitly check get_balance themselves (for example, to model overdraft accounts or internal system accounts).

Account 0 is the system account. It serves as the source and sink for all money entering or leaving the ledger. A Deposit produces a debit on the user account (balance goes up) and a credit on account 0; a Withdrawal produces a credit on the user account (balance goes down) and a debit on account 0. The convention used throughout the engine and the WASM host API is debit adds, credit subtracts — see the WASM Runtime guide for the host-call contract. This is not a stylistic choice — it is required by the zero-sum invariant described below.

Operations and Transactions

These two terms are distinct and the distinction matters.

An operation is the client’s intent. roda-ledger provides four operation types:

• Deposit — debits the user account (balance up), credits account 0 (the source)
• Withdrawal — credits the user account (balance down), debits account 0 (the sink)
• Transfer — moves an amount from one account to another (credit on the sender, debit on the receiver). If from == to, the operation is a no-op — sequenced and recorded but emits no TxEntry records.
• Function — invokes a previously registered WebAssembly module by name, with up to eight i64 parameters. The module produces credits and debits via host calls and runs atomically as a single transaction. This is the programmable ledger entry point and the only extension surface: any multi-account, multi-step logic that does not fit the named types is expressed as a registered function.

Every operation carries a user_ref — a caller-supplied value that serves two purposes: an idempotency key to prevent duplicate processing, and a reference stored alongside the transaction for correlation or auditing. Idempotency is described in detail below.

The client submits operations. It never writes transactions directly.

A transaction is the ledger’s immutable record of what happened. When an operation is accepted and executed, roda-ledger creates a transaction — a permanent, ordered fact in the log. Transactions are never modified, never deleted, and never reordered. The caller can query a transaction directly to check its status or inspect its details.

When querying a transaction, the caller sees entries — the individual credits and debits the Transactor produced from the original operation — not the operation itself. A Transfer becomes two entries: a credit on the sender and a debit on the receiver (recall: credit subtracts, debit adds). A Function becomes one entry per credit / debit host call it issued. The operation is the intent; the entries are the facts.

The client thinks in operations. The ledger thinks in transactions. The Transactor translates between the two.

Atomicity

A transaction is the atomicity boundary for failure. All steps within a single transaction either all succeed or all fail — if any condition fails at any point during execution (a balance check, a custom rule, or the zero-sum invariant), the entire transaction is rolled back, no balance is updated, and the caller is notified. The Transactor — the single deterministic writer at the heart of roda-ledger — enforces this for every operation type, including registered WebAssembly functions: a function that traps, returns a non-zero status, or emits an unbalanced credit/debit set is rolled back exactly the same way a failed Transfer is.

No torn writes. Each individual internal credit or debit is a single atomic store. A concurrent reader will never observe a partially-written balance for any single account. Every balance a reader sees is always a complete, valid value.

Intermediate state is visible across accounts. Within a multi-step operation — a Transfer, or a Function issuing several credit / debit host calls — the Transactor applies each step in sequence. A concurrent reader can observe some of these stores before others have landed — for example, seeing a Credit applied to one account before the corresponding Debit on another has landed. This is by design. The last_tx_id returned by get_balance is the signal that a transaction is fully settled: it is only updated after all steps in the transaction are complete. If a caller needs to know that a specific transaction is fully reflected across all accounts, they should wait for last_tx_id to advance past their transaction ID, or use submit_wait(snapshot) for a linearizable read.

The Zero-Sum Invariant

Every transaction must net to zero. The sum of all credits in a transaction must equal the sum of all debits. roda-ledger enforces this invariant on every transaction — a transaction that violates it is rejected before any balance is updated.

Because every individual transaction nets to zero, the sum of all balances across all accounts in the ledger is always zero. This is not a configuration option — it is an emergent property of the per-transaction invariant. Money is never created or destroyed inside the ledger. It only moves.

This is why account 0 exists. When money enters the system (a deposit), it must come from somewhere — account 0 is credited (its balance goes down). When money leaves the system (a withdrawal), it must go somewhere — account 0 is debited (its balance goes up). The ledger always balances.

Execution Model

The Transactor is a single, deterministic writer. There is no concurrency within transaction execution — one transaction at a time, in strict order. This is not a limitation. It is the source of correctness.

Because only one transaction executes at a time, there are no races, no conflicts, and no need for locks or conflict resolution. The Transactor maintains a hot in-memory cache of all account balances. Every write path — every operation submitted to the ledger — executes against this cache, which always reflects the latest committed state. The write path is always linearizable: no transaction ever makes a decision based on stale data.

The throughput of roda-ledger does not come from parallelizing execution. It comes from the pipeline that surrounds the Transactor, which allows sequencing, durability, and snapshotting to proceed concurrently with each other, without blocking the Transactor.

Staged Guarantees

A transaction does not go from submitted to done in a single step. It moves through a pipeline of stages, and each stage adds a specific guarantee. The caller chooses which stage to wait for.

Stage 1 — Sequencer The transaction receives a unique, monotonic ID and a permanent position in the global order. From this point, its place in history is fixed.

Stage 2 — Transactor The transaction is executed. Business logic runs, balances are updated in the Transactor’s in-memory cache, and the result (committed or rejected) is known. After this stage, the transaction has happened — but only in memory.

Stage 3 — WAL Storer The transaction is written to the Write-Ahead Log on disk. After this stage, the transaction survives a process crash or power loss. This is the durability boundary.

Stage 4 — Snapshotter The updated balances are written to the snapshot, making them visible to readers via get_balance. After this stage, reads reflect this transaction and all transactions before it.

The gap between Stage 3 and Stage 4 is typically tens to hundreds of nanoseconds — the Snapshotter runs continuously as part of the pipeline and is not a slow checkpoint process.

The significant gap is between Stage 2 and Stage 3. WAL writes are batched dynamically: the WAL Writer drains everything currently available, writes it, and only then yields to fdatasync (driven by an independent WAL Committer thread). Throughput and latency of this stage are bounded by sequential disk write speed.

Consistency Model

roda-ledger offers two consistency levels, and the caller controls which one they get.

Serializability — always, on the write path Every transaction is executed by the single Transactor against the latest in-memory state. Execution is strictly ordered. This gives serializability by default — the result of any set of concurrent submissions is equivalent to some serial execution.

Linearizability — on the read path, opt-in By default, get_balance reads from the snapshot, which may be a few nanoseconds behind the latest committed transaction. The response includes last_tx_id — the ID of the most recent transaction reflected in the returned balance. The caller can use this to reason about freshness without blocking.

If the caller needs a linearizable read — a guarantee that the balance reflects all transactions up to and including a specific one — they submit with WaitLevel::OnSnapshot (gRPC: WAIT_LEVEL_SNAPSHOT). This blocks until the Snapshotter has processed the transaction, after which get_balance is guaranteed to reflect it and everything before it.

To summarize:

Idempotency and the Active Window

Duplicate protection is always on — it cannot be disabled. When user_ref > 0, the ledger guarantees that submitting the same user_ref twice within the active window produces a single committed transaction. The second submission is recorded as a duplicate, linked to the original, and rejected with DUPLICATE — the caller always recovers the original tx_id.

The active window is the range of the last N transactions, where N equals the configured segment transaction count (transaction_count_per_segment). This is the same boundary that controls WAL segment rotation. Because the deduplication cache uses a flip-flop mechanism (active + previous), the effective coverage is N to 2N transactions — at least one full segment worth, at most two.

The window is defined by transaction count, not wall-clock time. This makes idempotency guarantees deterministic regardless of throughput or load spikes — the same number of transactions always provides the same protection, whether they arrive in one second or one hour.

user_ref = 0 opts out of the idempotency check for that individual transaction. This is the mechanism, not a global toggle — there is no configuration to disable deduplication system-wide.

Flexibility

roda-ledger is designed around two dimensions of flexibility that set it apart from opinionated ledger systems.

Programmable ledger via WebAssembly. Beyond the built-in Deposit, Withdrawal, and Transfer operation types, the Function operation type is the single extension point for arbitrary multi-account logic. The caller uploads a compiled WebAssembly module, registers it under a name (a durable operation in its own right — see below), and from then on invokes it by name with up to eight i64 parameters. The runtime exposes only three host calls — credit, debit, get_balance — so the module can move value and inspect balances but cannot break out of the ledger’s invariants. This covers any financial logic that does not fit the named types: split payments, fee deductions, multi-leg settlements, or domain-specific balance rules. New transaction types can be added at runtime without recompiling or redeploying the engine, while keeping the same correctness, durability, and audit guarantees as the built-in operations.

Choose your guarantee level. The staged pipeline exposes a dial between performance and consistency. The caller chooses how much to wait per submission:

• SubmitOperation (gRPC) / submit (Rust) — returns a transaction ID immediately, waits for nothing. Maximum throughput.
• WaitLevel::Computed — waits for execution. The caller knows whether the transaction committed or was rejected.
• WaitLevel::Committed — waits for durability. The transaction is on disk and survives a crash.
• WaitLevel::OnSnapshot — waits for the balance to be visible. Guarantees linearizable reads via get_balance.

The caller makes this choice per submission based on what their use case requires. See the API for the exact gRPC and Rust call shapes.

Atomic Function Execution

A Function operation is, from the ledger’s perspective, indistinguishable from any other transaction once committed. From the perspective of the programmer who wrote the function, the contract is precise and worth stating explicitly:

One execution = one transaction. A single call to the function’s exported execute symbol corresponds to exactly one transaction. The function may issue any number of credit / debit calls during that execution; together they form one atomic unit.

All-or-nothing. The Transactor accumulates every credit and debit the function emits in an isolated host-side context. Nothing is applied to the live balance cache until the function has fully returned and its credits balance its debits. If the function returns a non-zero status, traps, or violates the zero-sum invariant, the entire batch of host calls is discarded and no balance changes — exactly as if a built-in operation had failed mid-step.

No partial visibility. Because the function runs on the single Transactor thread, intermediate state from a partially-executed function is never visible to readers, other functions, or other transactions. Other transactions see the function’s effects all at once, at the same moment they see the last_tx_id advance past it.

Reusable, named, versioned. A Function is registered once and invoked many times by name. Every register or override bumps the version by one; the CRC32C of the executing binary is recorded in every transaction it produces, so any past entry can be traced back to the exact code that wrote it.

Durable registration. RegisterFunction is itself transactional. The call only returns after the binary is on disk, a FunctionRegistered record is committed to the WAL, and the handler is loaded into the live runtime. A subsequent Operation::Function is therefore guaranteed to see the new version. After a crash, recovery rebuilds the registry from a paired function snapshot plus the FunctionRegistered records that follow it — the live runtime always matches what the WAL says it should be.

Determinism by construction. The host API does not expose clocks, randomness, file or network I/O, threads, or atomics. Every legal function is a pure mapping from (params, observed balances) → (status, credits, debits). This is what makes the runtime safe for future Raft replication: the leader executes the function, and followers apply the WAL entries it produced without ever re-running the WASM code.

In short, the Function operation gives the caller a way to express new transaction shapes — fee splits, multi-leg settlements, conditional transfers, accounting templates — that behave as if they had been built into the engine from day one.

What Roda-Ledger Is Not

Understanding the boundaries is as important as understanding the capabilities.

Not a general-purpose database. There is no query language, no secondary indexes, no ad-hoc reads beyond balance lookup by account ID or transaction ID.

Not an authorization layer. roda-ledger does not authenticate callers or enforce access control. This is the responsibility of the layer above it.

Not a distributed system today. roda-ledger is a single-node engine. There is no replication, no leader election, no multi-node coordination. This is a current limitation, not a design principle — Raft-based multi-node replication is planned.

What it does have today. A gRPC interface and a Docker image. roda-ledger is not only an embedded library — it can run as a standalone service.

Planned additions. Raft-based multi-node replication, mTLS authentication, and per-function CPU / memory metering for the WASM runtime.

API

This document covers the full API surface of roda-ledger. Every operation is shown in both interfaces — gRPC for standalone server usage and Rust library for embedded usage. The concepts are identical; only the syntax differs.

For a full understanding of what operations, transactions, and wait levels mean, read Concepts first.

Setup

gRPC Server

Start the server with Docker:

docker run -p 50051:50051 -v $(pwd)/data:/app/data tislib/roda-ledger:latest

To customize the server, mount a TOML config file:

docker run -p 50051:50051 \
  -v $(pwd)/data:/app/data \
  -v $(pwd)/config.toml:/app/config.toml \
  tislib/roda-ledger:latest

config.toml — full reference:

[server]
host                  = "0.0.0.0"
port                  = 50051
max_connections       = 1000
max_message_size_bytes = 4194304  # 4MB

[ledger]
max_accounts    = 1000000
wait_strategy   = "balanced"   # low_latency | balanced | low_cpu

[ledger.storage]
data_dir                       = "/data"
transaction_count_per_segment  = 10000000  # 10M transactions per segment
snapshot_frequency             = 2         # take a snapshot every N sealed WAL segments

wait_strategy options:

Rust Library

Add to Cargo.toml:

[dependencies]
roda-ledger = { git = "https://github.com/tislib/roda-ledger" }

Initialize and start the ledger:

#![allow(unused)]
fn main() {
use roda_ledger::ledger::{Ledger, LedgerConfig};
use roda_ledger::storage::StorageConfig;

let config = LedgerConfig {
    max_accounts: 1_000_000,
    wait_strategy: WaitStrategy::Balanced,
    storage: StorageConfig {
        data_dir: "./data".to_string(),
        transaction_count_per_segment: 10_000_000,
        snapshot_frequency: 2,
        temporary: false,
    },
    ..LedgerConfig::default()
};

let mut ledger = Ledger::new(config);
ledger.start()?;
}

The ledger recovers its state automatically on start — it finds the latest snapshot, loads it, and replays any WAL segments that followed. No manual recovery step is needed.

Submitting Operations

Fire and Forget

Submit an operation and get a transaction_id back immediately. The operation is sequenced but not yet executed.

gRPC:

# Deposit
grpcurl -plaintext -d '{
  "deposit": {"account": 1, "amount": "1000", "user_ref": "42"}
}' localhost:50051 roda.ledger.v1.Ledger/SubmitOperation

# Withdrawal
grpcurl -plaintext -d '{
  "withdrawal": {"account": 1, "amount": "500", "user_ref": "43"}
}' localhost:50051 roda.ledger.v1.Ledger/SubmitOperation

# Transfer
grpcurl -plaintext -d '{
  "transfer": {"from": 1, "to": 2, "amount": "300", "user_ref": "44"}
}' localhost:50051 roda.ledger.v1.Ledger/SubmitOperation

Response:

{"transactionId": "42"}

Rust library:

#![allow(unused)]
fn main() {
use roda_ledger::transaction::Operation;

let tx_id = ledger.submit(Operation::Deposit {
    account: 1,
    amount: 1000,
    user_ref: 42,
});

let tx_id = ledger.submit(Operation::Withdrawal {
    account: 1,
    amount: 500,
    user_ref: 43,
});

let tx_id = ledger.submit(Operation::Transfer {
    from: 1,
    to: 2,
    amount: 300,
    user_ref: 44,
});
}

Operation Types

Deposit — credits a user account. Account 0 (the system account) is debited automatically to satisfy the zero-sum invariant. Deposits never fail due to insufficient funds.

Withdrawal — debits a user account. Fails with INSUFFICIENT_FUNDS if the account balance is below the requested amount. Account 0 is credited automatically.

Transfer — moves funds between two accounts atomically. Fails with INSUFFICIENT_FUNDS if the source account balance is insufficient. If from == to, the operation succeeds immediately as a no-op.

Function — invokes a registered WASM function by name with up to 8 i64 parameters. See WASM Runtime for the full guide on writing, registering, versioning, and invoking functions.

The user_ref Field

Every operation carries a user_ref — a uint64 supplied by the caller. It serves two purposes:

• Idempotency key — when user_ref > 0, if the same user_ref appears within the active window — the deduplication cache uses a flip-flop (active + previous) keyed off transaction_count_per_segment, giving an effective window of N to 2N transactions — the second submission is detected as a duplicate. It is sequenced and recorded, but linked to the original via a TxLinkRecord { kind: DUPLICATE } rather than re-executed. This prevents double-processing on client retries. Deduplication is always on and cannot be disabled. Pass user_ref = 0 to opt out of the idempotency check for individual transactions.
• Correlation reference — stored in the WAL alongside the transaction. Use it to link to your own database record — an order ID, payment ID, or any external reference.

Custom Operations (WASM Function)

Arbitrary multi-account, multi-step atomic logic is expressed as a registered WASM function and invoked via Operation::Function. The function can call the ledger.credit / ledger.debit / ledger.get_balance host imports; the zero-sum invariant is enforced after it returns, identically to built-in operations.

See the WASM Runtime guide for the full story: writing functions in Rust / AssemblyScript / WAT, registering them over gRPC or the Rust library, versioning, recovery, and performance numbers.

Submit and Wait

Block until the operation reaches a specific pipeline stage. Returns the full result including whether the transaction was rejected and why.

gRPC:

grpcurl -plaintext -d '{
  "deposit": {"account": 1, "amount": "1000", "user_ref": "46"},
  "wait_level": "WAIT_LEVEL_SNAPSHOT"
}' localhost:50051 roda.ledger.v1.Ledger/SubmitAndWait

Response:

{"transactionId": "43", "failReason": 0}

Rust library:

#![allow(unused)]
fn main() {
use roda_ledger::transaction::WaitLevel;

let result = ledger.submit_and_wait(
    Operation::Deposit { account: 1, amount: 1000, user_ref: 46 },
    WaitLevel::OnSnapshot,
);

if result.fail_reason.is_none() {
    println!("committed: tx {}", result.tx_id);
}
}

Wait levels:

WAIT_LEVEL_COMPUTED — The Transactor has executed the operation. Validation has run. If the operation violated any constraint (insufficient funds, zero-sum violation, etc.), fail_reason is set and the transaction is permanently rejected. If fail_reason = 0, the transaction is accepted and balance changes are live in memory — but not yet durable. A crash at this point would lose the transaction.

WAIT_LEVEL_COMMITTED — The WAL Storer has flushed the transaction to disk. Durability is guaranteed. The transaction will survive a crash and be replayed on restart. Balance changes are not yet visible via get_balance.

WAIT_LEVEL_SNAPSHOT — The Snapshotter has applied the transaction to the balance cache. get_balance now reflects this transaction and all transactions before it. This is the linearizable read guarantee — any get_balance call after this point will see a consistent, fully settled balance.

To get a transaction ID without waiting for anything, use SubmitOperation / submit instead.

Batch Operations

Submit multiple operations in a single round trip. Each operation is independent — failure of one does not affect others. There is no atomicity guarantee across operations in a batch.

gRPC:

grpcurl -plaintext -d '{
  "operations": [
    {"deposit":  {"account": 1, "amount": "1000", "user_ref": "50"}},
    {"deposit":  {"account": 2, "amount": "2000", "user_ref": "51"}},
    {"transfer": {"from": 1, "to": 2, "amount": "500", "user_ref": "52"}}
  ]
}' localhost:50051 roda.ledger.v1.Ledger/SubmitBatch

# With wait:
grpcurl -plaintext -d '{
  "operations": [
    {"deposit": {"account": 1, "amount": "1000", "user_ref": "53"},
     "wait_level": "WAIT_LEVEL_COMMITTED"}
  ]
}' localhost:50051 roda.ledger.v1.Ledger/SubmitBatchAndWait

Rust library:

#![allow(unused)]
fn main() {
let ops = vec![
    Operation::Deposit { account: 1, amount: 1000, user_ref: 50 },
    Operation::Deposit { account: 2, amount: 2000, user_ref: 51 },
    Operation::Transfer { from: 1, to: 2, amount: 500, user_ref: 52 },
];

let results = ledger.submit_batch_and_wait(ops, WaitLevel::Committed);

for result in results {
    println!("tx {}: {:?}", result.tx_id, result.fail_reason);
}
}

Operations within a batch are sequenced in the order provided. However, other clients’ transactions may be interleaved between them — there is no reservation of a contiguous block in the global order.

Prefer batch submission over repeated single calls for high-throughput ingestion — it reduces round trips significantly.

Transaction Status

Poll the pipeline stage of a transaction by ID.

gRPC:

# Single
grpcurl -plaintext -d '{"transaction_id": "42"}' \
  localhost:50051 roda.ledger.v1.Ledger/GetTransactionStatus

# Batch
grpcurl -plaintext -d '{"transaction_ids": ["42", "43", "44"]}' \
  localhost:50051 roda.ledger.v1.Ledger/GetTransactionStatuses

Rust library:

#![allow(unused)]
fn main() {
let status = ledger.get_transaction_status(tx_id);
}

Transaction status values:

Rejection codes (fail_reason):

Reading Balances

Single Account

gRPC:

grpcurl -plaintext -d '{"account_id": 1}' \
  localhost:50051 roda.ledger.v1.Ledger/GetBalance

Response:

{"balance": "1500", "lastSnapshotTxId": "42"}

Rust library:

#![allow(unused)]
fn main() {
let balance = ledger.get_balance(1);
// balance.balance: i64
// balance.last_snapshot_tx_id: u64
}

Multiple Accounts

gRPC:

grpcurl -plaintext -d '{"account_ids": [1, 2, 3]}' \
  localhost:50051 roda.ledger.v1.Ledger/GetBalances

Rust library:

#![allow(unused)]
fn main() {
// call get_balance per account
}

Linearizable Reads

get_balance reflects all transactions that have reached ON_SNAPSHOT. By default this lags behind committed transactions by nanoseconds — almost always negligible.

If you need a guaranteed linearizable read after a specific write — the balance must reflect your transaction and everything before it — you have two options:

Option 1 — wait at submit time:

# gRPC
grpcurl -plaintext -d '{
  "deposit": {"account": 1, "amount": "1000", "user_ref": "55"},
  "wait_level": "WAIT_LEVEL_SNAPSHOT"
}' localhost:50051 roda.ledger.v1.Ledger/SubmitAndWait
# then call GetBalance — guaranteed to reflect your transaction

#![allow(unused)]
fn main() {
// Library
ledger.submit_and_wait(op, WaitLevel::OnSnapshot);
let balance = ledger.get_balance(1); // linearizable
}

Option 2 — poll last_snapshot_tx_id:

#![allow(unused)]
fn main() {
let tx_id = ledger.submit(op);
loop {
let balance = ledger.get_balance(1);
if balance.last_snapshot_tx_id >= tx_id {
// balance is now linearizable with respect to your transaction
break;
}
}
}

Pipeline Index

Observe the current progress of each pipeline stage. Useful for health checks, lag monitoring, and determining when a bulk load is complete.

gRPC:

grpcurl -plaintext -d '{}' \
  localhost:50051 roda.ledger.v1.Ledger/GetPipelineIndex

Response:

{
  "computedIndex": "10500",
  "committedIndex": "10498",
  "snapshotIndex": "10496"
}

Rust library:

#![allow(unused)]
fn main() {
let computed   = ledger.last_compute_id();   // processed by Transactor
let committed  = ledger.last_commit_id();  // flushed to WAL
let snapshotted = ledger.last_snapshot_id();  // applied to balance cache
}

Common patterns:

• Health check — all three indexes should be advancing. A stalled index indicates a stuck pipeline stage.
• Commit lag — computed_index - committed_index. High lag means WAL writes are falling behind.
• Snapshot lag — committed_index - snapshot_index. Should be near zero under normal load.
• Bulk load completion — submit all operations, then poll until snapshot_index >= last_tx_id.

Transaction Details

Retrieve the full record of a transaction — its entries and any links to related transactions.

gRPC:

grpcurl -plaintext -d '{"tx_id": "42"}' \
  localhost:50051 roda.ledger.v1.Ledger/GetTransaction

Response:

{
  "txId": "42",
  "entries": [
    {"accountId": "1", "amount": "1000", "kind": "DEBIT",  "computedBalance": "1000"},
    {"accountId": "0", "amount": "1000", "kind": "CREDIT", "computedBalance": "-1000"}
  ],
  "links": []
}

Each entry shows the account, amount, direction (CREDIT or DEBIT), and the balance of that account immediately after the entry was applied.

Links connect related transactions:

Account History

Retrieve the transaction history for an account, newest first, with pagination.

gRPC:

# First page
grpcurl -plaintext -d '{"account_id": 1, "from_tx_id": 0, "limit": 20}' \
  localhost:50051 roda.ledger.v1.Ledger/GetAccountHistory

# Next page — pass next_tx_id from previous response
grpcurl -plaintext -d '{"account_id": 1, "from_tx_id": "38", "limit": 20}' \
  localhost:50051 roda.ledger.v1.Ledger/GetAccountHistory

Response:

{
  "entries": [
    {"accountId": "1", "amount": "500", "kind": "CREDIT", "computedBalance": "500"},
    ...
  ],
  "nextTxId": "38"
}

next_tx_id = 0 means there are no more entries. Pass from_tx_id = 0 to start from the latest transaction. Default limit is 20, maximum is 1000.

Use cases: account statements, balance verification, dispute resolution, reconciliation.

Library Extras

These methods are available in the Rust library only and have no gRPC equivalent.

wait_for_transaction(tx_id) — blocks until the given transaction reaches ON_SNAPSHOT. Times out after 10 seconds.

wait_for_transaction_until(tx_id, duration) — same as above with a custom timeout.

wait_for_transaction_level(tx_id, level) — blocks until the given transaction reaches the specified WaitLevel. Returns immediately on rejection regardless of wait level.

wait_for_pass() — blocks until the last submitted transaction reaches ON_SNAPSHOT. Useful after a bulk load to confirm everything is visible.

get_rejected_count() — returns the total number of rejected transactions since startup. Useful for metrics and health monitoring.

query(request) / query_block(request) — low-level access to the Snapshot stage query queue. query is non-blocking; query_block blocks until the Snapshot stage processes the request. Used for advanced read patterns not covered by get_balance.

Architecture

This document explains the design of roda-ledger — what each part does, why it exists, and the reasoning behind key tradeoffs.

If you haven’t already, read Concepts first — it defines the terminology and guarantees this document builds on.

Design philosophy

roda-ledger is built around two problems that pull in opposite directions: correctness and throughput.

Correctness requires strict ordering and a single source of truth — concurrent writers create races, conflicts, and partial states that are catastrophic in a financial system. Throughput requires parallelism — sequential processing leaves hardware idle.

The solution is a staged pipeline. Each stage does one job on its own thread. The Transactor’s record stream flows to the read stages through a single lock-free transaction ring; the submit and query paths use lock-free SPSC queues. There is no shared mutable state between stages.

The key insight: parallelism around execution, not within it. The Transactor executes one transaction at a time — this is the source of all correctness guarantees. But while the Transactor executes transaction N, the WAL is persisting N-1, and the Snapshotter is making N-2 visible to readers. All stages run concurrently without ever coordinating.

Design principles

Three named principles recur throughout the design and are referenced by name in code and ADRs. P1 and P2 were introduced in ADR-022; P3 in ADR-023.

P1 — WAL entry types are concrete and self-defining. Each WAL entry type does exactly one concrete thing, and its named fields fully and unambiguously define it: no nested discriminators, no op-codes reinterpreting fields, no overloaded or union fields. A replayer, auditor, or human understands a record from its kind and fields alone. Every 40-byte WAL record follows it — including the programmable-state KvEntry (ADR-023), which records a plain key→value mutation; which in-memory store it feeds is a transactor concern that never reaches the log.

P2 — Balances are eventually consistent on the read side. The online read view (the Snapshotter’s balance cache) may lag and need not be strictly ordered or point-in-time. This prioritises the snapshot write path and permits relaxed memory ordering and client-side caching. The durable WAL and sealed snapshot files remain the strongly-consistent source of truth.

P3 — The WAL and Snapshot are self-sufficient; derived computation lives in the consumer, not the Ledger. The durable WAL is self-explanatory by design, and together with the Snapshot it carries enough to understand and surface any detail about the data with no external context — P1 carried from the single record to the whole. The Ledger therefore serves complete, typed reads of stored state — balances, entries, history, programmable state — but builds no custom, analytical, or aggregate queries: there is no plan and no need for an in-engine query language or programmable read path under the current design. Any later analytical or derived view is the consuming system’s job — it reads the self-describing WAL/Snapshot and computes downstream. Holding computation outside the Ledger is what preserves the single-writer correctness and throughput this document is built on.

Pipeline overview

A transaction moves through four stages, each adding a guarantee:

The caller chooses which stage to wait for — see API for wait level details.

Transaction logical view

Every transaction occupies a fixed position in a globally ordered sequence. Three pipeline indexes divide that sequence into zones at any given moment:

• compute_index — the furthest transaction the Transactor has executed. Everything to its right is accepted, in memory, serializable. Not yet durable.
• commit_index — the furthest transaction flushed to disk. Everything to its right is durable and crash-safe. The Commit Gap between compute_index and commit_index represents transactions that would be lost on a crash — bounded by the WAL’s batch flush cycle (~100s µs).
• snapshot_index — the furthest transaction visible to get_balance. Everything to its right is readable and linearizable. The Balance Gap between commit_index and snapshot_index is typically nanoseconds.

Two invariants hold at all times:

1. All transactions are monotonically ordered — no gaps, no reordering.
2. If transaction N is committed, all transactions before N are also committed. The indexes only move forward.

These invariants are what make per-call wait levels meaningful. When you call submit_wait(wal), you are waiting for commit_index to pass your transaction ID — and you know with certainty that everything before it is also durable.

The Sequencer

The Sequencer is the entry point. It assigns a unique, monotonically increasing ID to every transaction and fixes its position in the global order. This decision is permanent — no later stage can change it.

The Sequencer has no dedicated thread. It runs on the caller’s thread at submit time — nothing more than an atomic increment and a queue push.

The Transactor

The Transactor is a single-threaded, deterministic execution engine. It processes one transaction at a time, in strict sequence order.

Why single-threaded? A single writer eliminates races and conflicts entirely. Correctness is guaranteed by construction, not by coordination.

For each transaction the Transactor: checks deduplication by user_ref, executes the operation (producing credit/debit entries and updating the balance cache live), and writes those entries directly into the transaction ring’s uncommitted region — slots it owns exclusively that no reader can see yet. It verifies the zero-sum invariant by walking its own uncommitted slots, folds a running CRC over the entries as they are written, then assembles the transaction’s metadata — its commit record — and publishes the whole transaction atomically by advancing the ring’s write frontier. Failed transactions — duplicates, execution errors, invariant violations — discard the uncommitted entries and publish a lone trailing metadata record, so the audit trail stays complete and readers only ever observe whole, validated transactions.

The balance cache always reflects the latest state. This is why the write path is always linearizable — the Transactor never makes a decision based on stale data.

For built-in operations (Deposit, Withdrawal, Transfer) the Transactor runs native Rust code. For an Operation::Function, the Transactor delegates to the embedded WASM Runtime described below. Either path produces the same TxEntry records, is bounded by the same zero-sum check, and uses the same rollback machinery — there is no second code path for durability or recovery.

The WASM Runtime — programmable execution inside the Transactor

The WASM Runtime is not a separate pipeline stage. It is a component inside the Transactor that handles Operation::Function the same way native code handles Operation::Transfer. This placement is deliberate: a function execution must inherit every guarantee the Transactor already provides — single-writer ordering, zero-sum enforcement, atomic rollback, deduplication — without any new coordination.

Where it sits in the pipeline

Sequencer → Transactor ─┬─ native execution path  ─┐
                        │                          ├─→ TxEntry stream → WAL → Snapshotter
                        └─ WasmRuntime.invoke()   ─┘

A Function operation flows through Sequencer → WAL → Snapshotter exactly like any other operation. The only difference is how the Transactor produces its entries: by calling into a sandboxed WebAssembly handler that issues credit / debit host calls, instead of running a hard-coded Rust branch.

Components

• WasmRuntime — one per ledger, wrapping a single shared wasmtime::Engine and Linker. Lives behind an Arc<WasmRuntime> and is cheap to clone.
• Per-Transactor caller cache — a HashMap<name, CachedHandler> owned by the Transactor thread. No locks on the hot path; the cache stores the update_seq of the registry it was verified against and revalidates lazily when a registration changes.
• Shared registry — a small lock-protected map of name → (version, crc32c, compiled module). Only touched on registration, unregistration, and cache miss / revalidation — never on every transaction.
• Host-side execution context — a per-Transactor TransactorState that captures the host calls a function emits during a single execute() invocation. Credits and debits accumulate here, not in WASM-visible globals, so no state leaks across calls.

Atomic execution flow

For an Operation::Function { name, params, user_ref } the Transactor:

1. Performs the standard user_ref deduplication check.
2. Resolves name against the caller cache, falling back to the shared registry on miss; verifies update_seq to detect a concurrent registration.
3. Opens an isolated host-side execution context — credits and debits emitted from now on are recorded against this transaction only.
4. Calls TypedFunc::call(execute, params) on the cached handler. The function may issue any number of credit / debit / get_balance host calls; each is captured into the execution context.
5. On any trap, host error, or non-zero return code: discards every captured credit/debit, marks the transaction failed with the appropriate status (the user-defined u8 for 128..=255, a standard reason otherwise), and writes a TxMetadata record so auditors can see exactly which function and which return code rejected the transaction.
6. On a 0 return: verifies sum(credits) == sum(debits). Failure here yields ZERO_SUM_VIOLATION and the same rollback. Success commits the captured entries to the live balance cache and emits them as ordinary TxEntry records.
7. Tags the resulting TxMetadata with b"fnw\n" ++ crc32c[0..4] so every committed transaction permanently identifies the exact binary that produced it.

Stages 5–7 are the atomicity boundary. Nothing is published to the transaction ring until the function has either fully succeeded and balanced, or been fully rolled back. The WAL stage and the Snapshotter that follow it are unchanged — they cannot tell a function-produced entry stream from a native one.

Why this fits the pipeline

• Single-writer correctness. A function executes on the Transactor thread, in strict sequence, with the same in-memory balance cache native operations use. Two functions cannot race; a function cannot race with a native operation.
• No new persistence path. Function-produced entries are normal TxEntry records. WAL segmentation, sealing, snapshotting, and replay treat them identically to entries from a Transfer.
• No new failure mode. A WASM trap, an unbalanced credit/debit set, or a domain-specific reject all use the existing transaction-rollback machinery.
• Determinism for replication. The host API is intentionally narrow: no clocks, no randomness, no I/O, no atomics, no threads. The leader executes a function; any future Raft follower can apply the resulting WAL entries directly without re-running the WASM code.

Registration as a first-class WAL event

Registering or unregistering a function is itself a durable transaction in the pipeline:

1. The submitted binary is validated (signature, size, exports) before any disk write.
2. The binary is written atomically (tmp + rename) to data/functions/{name}_v{N}.wasm. Unregister writes a 0-byte file under the next version — the audit trail is preserved.
3. A FunctionRegistered WAL record (name, version, crc32c) is committed to the active segment.
4. The handler is compiled and installed in the live WasmRuntime.
5. Only after all four steps does RegisterFunction return.

Periodic snapshots emit a paired function_snapshot_{N}.bin next to the balance snapshot_{N}.bin, on the same snapshot_frequency trigger. Recovery loads the function snapshot first, replays FunctionRegistered records from the WAL after it (crc32c != 0 → load, crc32c == 0 → unload), and only then resumes processing transactions. A missing or corrupted binary aborts startup — the runtime never serves traffic against a registry that diverges from the WAL.

Performance shape

• The hot path adds one HashMap::get, one TypedFunc::call, and one host-import crossing per credit / debit.
• Compiled modules are cached per (name, crc); instantiation cost is paid once per registration, not per call.
• Cache invalidation is per-name — registering foo does not evict the cached entry for bar.
• At the Transactor level, WASM execution adds ~200 ns per operation over the native path. At the pipeline level, this is invisible: the WAL fdatasync cost dominates and end-to-end TPS for Function and Deposit are within a percent of each other.

Detailed ABI, host API, validation rules, and recovery semantics live in WASM Runtime and ADR-014.

The WAL

The WAL stage makes transactions durable. A transaction executed by the Transactor exists only in memory — the WAL writes it to disk before it can be considered safe.

The WAL runs as two concurrent threads communicating through shared atomics:

• WAL Writer — reads published records from the transaction ring, writes them to the active segment file, advances last_written_tx_id, and rotates segments when full.
• WAL Committer — runs independently, calls fdatasync whenever last_written_tx_id > last_committed_tx_id, then advances last_committed_tx_id (which the pipeline exposes as commit_index).

The Writer never blocks waiting for fdatasync — it continues writing while the Committer syncs. The WAL forwards nothing: the Snapshotter reads the same ring independently and gates itself on commit_index, so a transaction becomes visible to readers only after it is durable. This decoupling is why the WAL sustains high write throughput despite the inherent latency of fdatasync (~100s µs, disk-bound).

The WAL is segmented — divided into files based on transaction count (transaction_count_per_segment). When the transaction count in the active segment reaches the configured limit, the Writer rotates to a new one. Segment files are dynamically sized on disk — a segment with many complex (multi-entry) transactions will be larger than one with simple deposits — but the transaction count per segment is always fixed and predictable. Sealed segments are complete, consistent units used for recovery.

Segment lifecycle

A segment moves through three states:

ACTIVE → CLOSED: transaction count threshold hit → WAL Writer fdatasyncs, renames wal.bin to wal_NNNNNN.bin, opens new wal.bin.

CLOSED → SEALED: Seal process picks it up on a timer, computes CRC32, writes .crc and .seal sidecar files.

Segment anatomy

Every WAL record is exactly 40 bytes — fixed size, no variable-length scanning. This makes recovery fast and deterministic. A transaction is a contiguous group of records: its entries followed by a single trailing TxMetadata that acts as the commit record — a transaction exists exactly when its metadata has been written. Segments are a pure physical partition of one logical stream, so a transaction may span a segment boundary (its entries in one segment, the trailing metadata in the next). See ADR-020.

Seal process

The Seal process runs on a configurable timer. For each unsealed segment it: loads the WAL data, seals it (CRC + sidecar files), builds on-disk tx and account indexes, updates its local balance buffer, and conditionally writes a snapshot file.

Snapshots are not written on every seal — only every snapshot_frequency segments (default: 2). The snapshot captures all non-zero balances at that point and is the base for recovery.

Active Window, Idempotency, and Hot Indexes

The active window is the central sizing concept in roda-ledger. It is defined as the last N transactions where N = transaction_count_per_segment. Three subsystems are derived from this single value:

Segment rotation

WAL segment rotation triggers when the transaction count in the active segment reaches transaction_count_per_segment. This makes the active window predictable — operators always know exactly how many transactions fit in the active segment, regardless of transaction complexity or entry counts.

Idempotency (deduplication)

The Transactor maintains a flip-flop deduplication cache — two HashMaps (active and previous) mapping user_ref → tx_id. When tx_id crosses a segment boundary, the maps flip: active becomes previous, and a new active begins. This gives an effective deduplication window of N to 2N transactions.

The flip is driven by transaction ID, not wall-clock time. Under a burst of 1M transactions per second the window covers the same number of transactions as under 100 transactions per second. Idempotency guarantees are deterministic.

Deduplication is always on — there is no config toggle. user_ref = 0 skips the check on a per-transaction basis, but the cache is always active.

Hot indexes

The Snapshotter maintains two circular indexes for serving GetTransaction and GetAccountHistory queries on recent data, hot index is keeping last N transactions where N is active window size.

The Snapshotter

The Snapshotter applies committed entries to the readable state — the balance cache, transaction index, and account history index.

Why a separate stage? The Transactor’s balance cache is write-side truth, private and optimized for writes. The Snapshotter maintains read-side truth. Separating them means readers never block writers.

The Snapshotter draws from two independent sources:

• The transaction ring — it reads published records, buffering a transaction’s entries until the trailing metadata arrives and its tx_id is at or below commit_index (the durability gate). Only then does it apply the entries to the indexes and balance cache together, advance snapshot_index, and — as the ring’s sole releaser — reclaim the consumed slots. Readers therefore never see a partially-applied or not-yet-durable transaction.
• A query-only queue — GetTransaction / GetAccountHistory requests execute inline against current state and call the response callback.

The Snapshotter does no computation — balances are pre-computed by the Transactor and stored in entries. The gap between COMMITTED and ON_SNAPSHOT is typically nanoseconds.

The transaction ring

The Transactor, WAL, and Snapshotter share a single lock-free transaction ring — the one transport for the record stream, replacing the former transactor→WAL and WAL→snapshot queues (ADR-019).

• One producer. The Transactor appends records into ring slots and publishes them. It also builds each transaction in place in the uncommitted region, so the ring doubles as its per-transaction scratch space — no separate staging buffer.
• Independent copy-out readers. The WAL and Snapshotter each track their own absolute cursor and copy records out at their own pace. A slot may be reused once released, so readers copy rather than borrow.
• One releaser, durability-gated. The Snapshotter is the only party that advances the reclamation point, and only up to the durability watermark. A slot is reused only after the record it held is durable on disk and consumed by every reader — making “never overwrite an undurable record” a structural property of the transport.

Positions are absolute, monotonically increasing indices (mapped to physical slots on access), so progress and ordering stay unambiguous across wraps. Because a transaction is published atomically, the ring capacity (ring_size) must be at least the largest possible single transaction.

Inter-stage communication

The record stream rides the transaction ring described above; the submit and query paths use lock-free SPSC queues (sequencer → transactor, and the Snapshotter’s query queue) — one producer, one consumer, no locks. Each stage owns its data completely.

Backpressure propagates naturally. The Snapshotter releases ring slots only up to the durability watermark, so if durability or indexing stalls, the ring fills and the Transactor stops finding free slots; that stalls the Transactor, which fills the sequencer → transactor queue, which eventually stalls submit(). No explicit flow control is needed, and the producer never blocks inside the ring — it retries under the shared wait strategy.

The wait strategy under backpressure is controlled by the wait_strategy config field (see API → Setup): low_latency (spin forever), balanced (spin → yield → park), low_cpu (park quickly).

Recovery

On startup, roda-ledger restores state automatically before accepting transactions:

1. Find the latest snapshot — gives all balances at a known transaction ID
2. Load those balances
3. Replay WAL segments that postdate the snapshot
4. Resume from the exact point where the ledger left off

Because every committed transaction is in the WAL, and the WAL is always replayed on recovery, a committed transaction is never lost. Replay buffers a transaction’s entries until its trailing metadata closes and validates the group — carrying the buffer across segment seams for a transaction that spans a boundary — while a torn tail of the active segment is truncated back to its last metadata boundary. Recovery time is bounded by snapshot frequency — more frequent snapshots mean less WAL to replay.

Design boundaries

Single node. All pipeline stages run on one machine. Raft-based multi-node replication is planned — the segmented, append-only WAL is a natural fit for log replication.

Pre-allocated account space. Balances are stored in a structure sized to max_accounts. O(1) reads and writes always, but memory is committed at startup — a capacity planning decision.

Disk-bound durability throughput. fdatasync latency determines COMMITTED throughput. On NVMe this is hundreds of µs per batch. COMPUTED throughput (in-memory only) reaches millions of transactions per second.

Single-writer ceiling. The Transactor is the upper bound on execution throughput. In practice the WAL is almost always the bottleneck — but on fast enough storage, the single-threaded Transactor becomes the limit.

WASM Runtime — Programmable Ledger

roda-ledger lets you extend the transaction set at runtime with sandboxed WebAssembly functions. A registered function becomes a first-class Operation::Function: it is sequenced, executed atomically by the Transactor, and produces normal TxEntry records in the WAL alongside every other transaction.

This document covers everything about writing, registering, invoking, and operating programmable functions: ABI, lifecycle, storage layout, recovery guarantees, and gRPC surface.

The design is specified in ADR-014.

Table of contents

• When to use it
• Function ABI
• Host API
• Return values & rollback
• Writing a function
  • Rust
  • AssemblyScript
  • Hand-written WAT
• Registering a function
  • gRPC
  • Rust library
• Invoking a function
• Listing & unregistering
• Versioning
• Storage layout on disk
• Durability & recovery
• Determinism & isolation
• Performance
• Limits & validation
• Troubleshooting

When to use it

Use a function when the built-in operations aren’t enough:

• Fee splits — deduct a fee to one account, forward the net to another, in one atomic transaction.
• Multi-leg settlements — move value between three or more accounts with custom rules (e.g. escrow release + platform cut + counterparty payout).
• Conditional logic — decline the transaction when a business rule fails (insufficient collateral, failed KYC flag, overdraft policy).
• Accounting templates — reusable “compliance check”, “tax withholding”, “loyalty accrual” rules shared across submitters.

If your operation is expressible as one of Deposit, Withdrawal, or Transfer, use those — they are faster and need no registration step.

Function ABI

Every registered function must export one symbol named execute with a fixed signature:

execute(i64, i64, i64, i64, i64, i64, i64, i64) -> i32

• Exactly 8 i64 positional parameters. Unused slots are conventionally passed as 0.
• Exactly 1 i32 return value, interpreted as a u8 status code:
  • 0 — success (the transaction’s host-side credits / debits are committed);
  • 1..=127 — standard failure reasons (defined by the ledger, full rollback);
  • 128..=255 — user-defined custom reasons (full rollback).

The fixed arity eliminates any need for linear memory pointer / length passing: account ids, amounts, rates, flags, timestamps all fit into 8 i64 slots.

Host API

A function can only call three host imports. All three live in the ledger module:

(import "ledger" "credit"      (func (param i64 i64)))
(import "ledger" "debit"       (func (param i64 i64)))
(import "ledger" "get_balance" (func (param i64) (result i64)))

Convention in roda-ledger: credit subtracts, debit adds. A Transfer { from, to, amount } is credit(from, amount) + debit(to, amount) — credits move value out of an account, debits move value in. Same convention applies to WASM functions.

Zero-sum invariant. The ledger verifies sum(credits) == sum(debits) after the function returns. If the function emits an unbalanced set of credits / debits, the transaction is rejected and rolled back with Status::ZERO_SUM_VIOLATION.

Overdraft policy. Host credits / debits never fail individually, and the ledger does not enforce negative-balance checks inside functions. If you need overdraft protection, call get_balance and return a non-zero status to trigger rollback:

#![allow(unused)]
fn main() {
if get_balance(sender) < amount as i64 {
    return 1; // INSUFFICIENT_FUNDS
}
}

No randomness, no wall clock, no I/O, no WASM threads, no atomics. The ABI is deliberately small so every execution is deterministic.

Return values & rollback

The execute return value is the transaction’s final status:

• 0 — the host-side credits and debits become part of the WAL transaction; the zero-sum invariant is verified; if it holds, the transaction commits.
• any non-zero value — the transaction is fully rolled back: every credit / debit the function applied is reversed in the same way a failed built-in operation would be. The TxMetadata record is still written, tagged with the function’s CRC32C, and its status field carries the returned u8 so auditors can distinguish unsuccessful named operations by exact reason.

Host failures (linking, instantiation, traps) surface as Status::INVALID_OPERATION (numeric 5).

Standard status values used by the ledger:

Returning 128..=255 is how functions report domain-specific errors without polluting the standard status namespace.

Writing a function

Rust

# Cargo.toml
[lib]
crate-type = ["cdylib"]

[profile.release]
opt-level = "z"
lto = true
strip = true

#![allow(unused)]
fn main() {
// src/lib.rs
#[link(wasm_import_module = "ledger")]
unsafe extern "C" {
    fn credit(account_id: u64, amount: u64);
    fn debit(account_id: u64, amount: u64);
    fn get_balance(account_id: u64) -> i64;
}

/// Transfer `amount` from `from` to `to`, taking a `fee_bps` (basis
/// points) fee into `fee_acct`.
///
/// Params:
///   param0: sender
///   param1: receiver
///   param2: fee_acct
///   param3: amount
///   param4: fee_bps (e.g. 50 = 0.5%)
#[unsafe(no_mangle)]
pub extern "C" fn execute(
    param0: i64, param1: i64, param2: i64, param3: i64,
    param4: i64, _p5: i64, _p6: i64, _p7: i64,
) -> i32 {
    let sender   = param0 as u64;
    let receiver = param1 as u64;
    let fee_acct = param2 as u64;
    let amount   = param3 as u64;
    let fee_bps  = param4 as u64;
    let fee      = amount * fee_bps / 10_000;

    unsafe {
        if get_balance(sender) < (amount + fee) as i64 {
            return 1; // INSUFFICIENT_FUNDS
        }
        credit(sender, amount + fee);
        debit(receiver, amount);
        debit(fee_acct, fee);
    }
    0
}
}

Compile:

cargo build --target wasm32-unknown-unknown --release
# produces target/wasm32-unknown-unknown/release/<crate>.wasm

AssemblyScript

// assembly/index.ts
@external("ledger", "credit")
declare function credit(account_id: u64, amount: u64): void;

@external("ledger", "debit")
declare function debit(account_id: u64, amount: u64): void;

@external("ledger", "get_balance")
declare function get_balance(account_id: u64): i64;

export function execute(
    param0: i64, param1: i64, param2: i64, param3: i64,
    param4: i64, _p5: i64, _p6: i64, _p7: i64,
): i32 {
    const sender   = <u64>param0;
    const receiver = <u64>param1;
    const feeAcct  = <u64>param2;
    const amount   = <u64>param3;
    const feeBps   = <u64>param4;
    const fee      = amount * feeBps / 10_000;

    if (get_balance(<i64>sender) < <i64>(amount + fee)) {
        return 1; // INSUFFICIENT_FUNDS
    }
    credit(sender, amount + fee);
    debit(receiver, amount);
    debit(feeAcct, fee);
    return 0;
}

Compile:

asc assembly/index.ts -o build/function.wasm --optimize --runtime stub

Hand-written WAT

Useful for minimal-size tests and examples. Every WAT snippet in the test suite follows this template:

(module
  (import "ledger" "credit" (func $credit (param i64 i64)))
  (import "ledger" "debit"  (func $debit  (param i64 i64)))
  (import "ledger" "get_balance" (func $get_balance (param i64) (result i64)))

  (func (export "execute")
    (param i64 i64 i64 i64 i64 i64 i64 i64) (result i32)

    ;; credit(param0, param1)
    local.get 0 local.get 1 call $credit

    ;; debit(param2, param1)
    local.get 2 local.get 1 call $debit

    i32.const 0))

Compile via the wat crate (already a dependency in test / bench paths):

#![allow(unused)]
fn main() {
let bytes = wat::parse_str(WAT)?;
}

Registering a function

Registration is atomic and durable: it returns only after the WAL record has been committed and the handler is loaded into the live runtime. The next Operation::Function { name } is guaranteed to see it.

gRPC

rpc RegisterFunction(RegisterFunctionRequest) returns (RegisterFunctionResponse);
rpc UnregisterFunction(UnregisterFunctionRequest) returns (UnregisterFunctionResponse);
rpc ListFunctions(ListFunctionsRequest)         returns (ListFunctionsResponse);

message RegisterFunctionRequest {
    string name              = 1; // snake_case, max 32 bytes
    bytes  binary            = 2; // WASM binary, max 4 MB
    bool   override_existing = 3; // false → ALREADY_EXISTS if name is taken
}

message RegisterFunctionResponse {
    uint32 version = 1; // monotonic per name, starts at 1
    uint32 crc32c  = 2; // CRC32C of the binary
}

Example with grpcurl:

grpcurl -d "$(jq -n --arg bin "$(base64 -w0 function.wasm)" \
  '{name:"fee_transfer", binary:$bin, override_existing:false}')" \
  -plaintext localhost:50051 \
  roda.ledger.v1.Ledger/RegisterFunction

Rust library

#![allow(unused)]
fn main() {
use roda_ledger::client::LedgerClient;

let client = LedgerClient::connect("127.0.0.1:50051".parse()?).await?;

let binary = std::fs::read("function.wasm")?;
let (version, crc) = client
    .register_function("fee_transfer", &binary, /* override_existing = */ false)
    .await?;

println!("registered fee_transfer v{} (crc={:08x})", version, crc);
}

Or in embedded mode (one process, no gRPC):

#![allow(unused)]
fn main() {
use roda_ledger::ledger::{Ledger, LedgerConfig};

let mut ledger = Ledger::new(LedgerConfig::temp());
ledger.start()?;

let binary = std::fs::read("function.wasm")?;
let (version, crc) = ledger.register_function("fee_transfer", &binary, false)?;
}

Re-registering the same name without override_existing = true returns AlreadyExists. With the flag set, the new binary becomes version = previous + 1 and replaces the old one in the live registry atomically.

Invoking a function

A registered function is invoked as Operation::Function { name, params, user_ref }:

gRPC

message Function {
    string         name     = 1; // max 32 bytes
    repeated int64 params   = 2; // 0..=8 values; short lists are zero-padded
    uint64         user_ref = 3; // idempotency / dedup key, same as other ops
}

grpcurl -d '{
    "function": {
        "name": "fee_transfer",
        "params": [101, 202, 999, 10000, 50],
        "user_ref": 123456
    },
    "wait_level": "COMMITTED"
}' -plaintext localhost:50051 \
    roda.ledger.v1.Ledger/SubmitAndWait

Rust library

#![allow(unused)]
fn main() {
use roda_ledger::transaction::{Operation, WaitLevel};

let result = ledger.submit_and_wait(
    Operation::Function {
        name: "fee_transfer".into(),
        params: [101, 202, 999, 10_000, 50, 0, 0, 0],
        user_ref: 123_456,
    },
    WaitLevel::Committed,
);

if result.fail_reason.is_failure() {
    eprintln!("fee_transfer failed: {:?}", result.fail_reason);
}
}

gRPC client (Rust)

#![allow(unused)]
fn main() {
use roda_ledger::client::LedgerClient;
use roda_ledger::grpc::proto::WaitLevel;

let result = client
    .submit_function_and_wait(
        "fee_transfer",
        [101, 202, 999, 10_000, 50, 0, 0, 0],
        123_456, // user_ref
        WaitLevel::Committed,
    )
    .await?;
}

Every Operation::Function produces a normal transaction in the WAL with a TxMetadata.tag of the form:

b"fnw\n"  ++  crc32c[0..4]     (8 bytes total)

roda-ctl unpack renders it as:

{"type": "TxMetadata", "tx_id": 441001, "tag": "fnw\n4a2f1c3d", ...}

The CRC32C identifies the exact binary that executed — cross-reference it with the FunctionRegistered WAL record or with a ListFunctions response to resolve the name / version.

Listing & unregistering

List — returns every currently-loaded handler:

#![allow(unused)]
fn main() {
for info in client.list_functions().await? {
    println!("{} v{} (crc={:08x})", info.name, info.version, info.crc32c);
}
}

Unregister — writes an empty file under the next version and a FunctionRegistered WAL record with crc32c = 0. The call blocks until the handler is gone from the live registry; any Operation::Function { name } submitted after it returns will fail with INVALID_OPERATION.

#![allow(unused)]
fn main() {
let unregistered_version = client.unregister_function("fee_transfer").await?;
}

Unregister is durable: the 0-byte-file + crc32c = 0 pair survives restarts and crashes, and is replayed during recovery.

Versioning

• Every register or override bumps the version by +1, starting at 1.
• Version counter is stored in the live registry (sourced from FunctionRegistered WAL records + the function snapshot). The on-disk functions/ directory is reference data only.
• A Function operation always uses the current latest version — there is no pinning to a specific version at submit time.
• The CRC32C embedded in TxMetadata.tag identifies which version actually ran, so auditors can always tell which binary produced any given set of entries.

Storage layout on disk

data/
├── wal.bin                            # active WAL segment
├── wal_000001.bin                     # sealed segment (+ .crc, .seal, indexes)
├── wal_000002.bin
├── snapshot_000002.bin                # balance snapshot
├── snapshot_000002.crc
├── function_snapshot_000002.bin       # function-registry snapshot (same trigger)
├── function_snapshot_000002.crc
└── functions/
    ├── fee_transfer_v1.wasm           # binary under its version
    ├── fee_transfer_v2.wasm           # replaced version (override)
    └── removed_fn_v3.wasm             # 0 bytes — unregister marker

Function binaries are written atomically (temp file + rename). Unregister truncates the file to 0 bytes — it is not deleted, preserving the audit trail.

The function snapshot is emitted on the same snapshot_frequency trigger as the balance snapshot, so recovery always finds a paired snapshot_{N}.bin + function_snapshot_{N}.bin at the same segment boundary.

Durability & recovery

Registration is durable before register_function returns. The call blocks until:

1. The binary is written atomically to functions/{name}_v{N}.wasm.
2. A FunctionRegistered WAL record is committed to the active segment.
3. The live WasmRuntime has compiled and installed the handler.

Recovery on clean restart or crash proceeds as follows:

1. Load the latest function_snapshot_{N}.bin. For each record with crc32c != 0, read functions/{name}_v{version}.wasm and compile it into the runtime.
2. Replay every FunctionRegistered WAL record in segments after the snapshot:
   • crc32c != 0 → load the referenced version (replaces any older handler).
   • crc32c == 0 → unload the handler.
3. Resume normal operation.

A failure to read a binary or install a handler during recovery is non-recoverable: the server aborts startup rather than continue with a registry that diverges from the WAL. The CRC32C embedded in every FunctionRegistered record lets recovery detect silent disk corruption before anything transactional runs.

Determinism & isolation

The runtime is intentionally narrow:

• No randomness. No host API exposes a PRNG.
• No wall clock. Functions do not see time; tag timestamps are decided by the host.
• No I/O, no network, no filesystem. Only the 3 ledger host calls are available.
• No threads, no atomics. Functions run single-threaded.
• No persistent memory. A function has no state that survives between invocations: every call runs against a fresh wasmtime instance (internally cached by the engine for speed; no durable state is carried).
• Sandboxed. A trap or infinite loop cannot corrupt ledger state — the transaction is rolled back and the handler is left intact for subsequent calls.

This is the property we need for future Raft replication: the leader executes the WASM function, and followers apply the resulting entries directly — no re-execution, no divergence.

Performance

The runtime is tuned for low per-call overhead on the hot path.

• Per-call cost: one HashMap::get on the per-Transactor caller cache, one TypedFunc::call, two host imports per credit / debit crossing.
• Cache invalidation is per-name: registering foo does not evict the cached entry for bar. Each cached entry stores the update_seq it was verified at; the next lookup either short-circuits (seq unchanged) or does a shared-registry read to reconcile just that one name.
• One wasmtime Engine per ledger (shared via Arc<WasmRuntime>).
• One Linker with host imports wired exactly once at ledger startup.
• One Store per Transactor, long-lived across calls. Function state is carried in host TransactorState, not in WASM-visible globals.
• Instantiation happens once per (name, crc) pair on first lookup; the resulting TypedFunc is cached and reused.

Empirical numbers from the current build (Apple M-series, release build):

The per-op overhead at the Transactor level is ~200 ns — pure host-crossing cost. At the pipeline level (gRPC → WAL commit → response) it is invisible: the WAL commit path dominates.

Run the comparison yourself:

cargo bench --bench transaction_runner_bench       # native
cargo bench --bench transaction_runner_bench_wasm  # WASM
cargo run --release --bin load      -- --wait --duration 30
cargo run --release --bin load_wasm -- --wait --duration 30

Limits & validation

Validation runs at register_function before any disk write:

A binary that fails any of these returns InvalidArgument from gRPC or io::ErrorKind::InvalidData / InvalidInput from the Rust API. Nothing is written to disk and no WAL record is produced.

Troubleshooting

ALREADY_EXISTS on RegisterFunction A function by this name is already loaded. Pass override_existing = true to replace.

Operations return INVALID_OPERATION (status 5)

• The function is not registered, or was unregistered.
• The function trapped (out-of-bounds memory, stack overflow, unreachable, division by zero).
• The WASM binary parsed but failed instantiation. Check tonic / ledger logs around the call.

Operations return ZERO_SUM_VIOLATION (status 3) The function’s total credits did not match total debits. Every credit(a, N) needs a matching debit(b, N) somewhere else in the same invocation (or vice versa).

Unexpected balances after a restart Inspect both snapshots at the last sealed segment id:

ls data/function_snapshot_*.bin
roda-ctl unpack data/wal_000NNN.bin

You should see a FunctionRegistered record for every function that should be loaded and a matching snapshot file at a later or equal segment id.

Snapshot: read_function(name vN) failed during startup The WAL references a binary that is missing from data/functions/. This is non-recoverable. Either restore the binary from backup, or manually remove the stale FunctionRegistered record from the WAL using roda-ctl (at your own risk).

See also

• ADR-014 — WASM Function Registry and Function Operation Execution
• Architecture — for context on the Transactor / WAL / Snapshot pipeline.
• API — full API reference for the ledger service.

Internal

Single source of truth. Each § states one complete invariant or decision in its own paragraph. Ambiguous behaviour is resolved by what this document says; where an ADR exists for a decision it is cited as [ADR-NNNN].

Stages:

1. The Ledger — §1..§16. The in-memory engine: structures, threads, queues, indexes, and the sequence in which records flow through the pipeline.
2. The Storage — §17..§25. The on-disk file format: segment file layout, WAL record byte layout, sidecar files, snapshot file format, function binary storage, on-disk index file format, durable cluster state (term and vote logs), and ctl tools.
3. The Cluster — §26..§39. The network-facing surface (gRPC, client, proto) and Raft replication.

Stage 1 specifies the mechanism by which the ledger reads and writes records; Stage 2 specifies the on-disk shape of those records; Stage 3 specifies how independent ledgers replicate to one another over a network.

§1 Domain model

§1.1 Accounts and the system account

An account is identified by a u64. Accounts are not created explicitly; they come into existence the first time a transaction references them. The system account is SYSTEM_ACCOUNT_ID = 0 and serves as the source/sink for every deposit and withdrawal — it is required by the zero-sum invariant (§4.11), not a convention. The system account’s balance is never tracked or queried.

§1.2 Balances

A balance is an i64. Balances are derived state: the source of truth is the ordered transaction log, and a balance is the result of applying every entry that has touched the account. Arithmetic uses saturating add/sub so a malformed operation cannot overflow the account silently. Default protection prevents balances from going negative (§4.9), and only WASM functions may explicitly bypass this by inspecting get_balance themselves.

§1.3 Operation vs. transaction

These are two distinct things and the distinction matters. An operation is the caller’s intent (Deposit, Withdrawal, Transfer, Function). A transaction is the immutable record produced by executing an operation — permanent, ordered, and never modified. The client submits operations; the ledger writes transactions; the Transactor (§4) is the bridge. When a caller queries a transaction it sees the entries (the credits/debits the Transactor emitted), not the original operation.

§1.4 The four operation types

Roda-ledger accepts exactly four operation types, all carrying a user_ref idempotency key (§5):

• Deposit { account, amount } — credits account, debits the system account.
• Withdrawal { account, amount } — debits account, credits the system account.
• Transfer { from, to, amount } — debits from, credits to.
• Function { name, params, user_ref } — invokes a previously registered WASM module by name (§6).

Function is the single extension surface; any multi-account or domain-specific logic that does not fit the named types is expressed as a registered function.

§1.5 Function operation parameters

Function parameters are a fixed [i64; 8]. Slots a caller does not need are zero-padded. The arity is locked at the type level so the host ABI (§6.7) and WAL/transaction representation never require variable-length parameter blocks.

§1.6 WAL record format invariants

Every WAL record is exactly 40 bytes — a hard invariant for every record type. The first byte of every record is its entry_type discriminant, which makes the WAL stream [record][record][record]… self-describing without a separate length-prefix or framing layer. Records are laid out for zero-copy serialization: the bytes on the wire are the bytes in memory. A transaction is a group of follower records (TxEntry/TxLink) terminated by a trailing TxMetadata — its commit record (§18.3, [ADR-0020]).

§2 Pipeline structure

§2.1 Stages and guarantees

The pipeline has four execution stages plus one independent maintenance stage. A transaction moves through Sequencer → Transactor → WAL → Snapshotter, and each step adds exactly one guarantee:

Seal (§11) runs in parallel and is responsible for finalising closed segments, producing on-disk indexes, and writing snapshot files for recovery.

§2.2 Status is derived, not stored

A TransactionStatus is computed at query time from the pipeline’s progress indexes — never persisted as a record. A transaction is Computed iff compute_index ≥ tx_id, Committed iff commit_index ≥ tx_id, OnSnapshot iff snapshot_index ≥ tx_id. Failed transactions are recognised through the rejection registry (§4.17). This avoids a per-transaction state machine on disk and makes status a pure function of the pipeline.

§2.3 Pipeline ownership and context slicing

The Pipeline is the sole owner of the transaction ring (§13.4), both inter-stage queues, every progress index, the shutdown flag, and the shared wait strategy. Stages never own this state directly — each receives a typed context that exposes only the slice it may legally read or publish. The wrapping is purely for type-level encapsulation; there is no runtime cost.

§2.4 Inter-stage transport

The record stream rides a single lock-free transaction ring (§13.4): the Transactor is the sole producer; the WAL and Snapshotter are independent copy-out readers; the Snapshotter is the sole releaser. The former transactor → wal and wal → snapshot queues are gone — both stages now read the same ring ([ADR-0019]).

Two lock-free, fixed-capacity, single-producer / single-consumer queues remain, for the paths that are not the record stream:

• sequencer → transactor — submitted operations awaiting execution.
• snapshot query — GetTransaction / GetAccountHistory requests (§8.4).

Queue capacity is fixed at construction from queue_size (§15.2); ring capacity from ring_size (§15.10). A full queue or a full ring is the only backpressure signal; producers spin/yield until space exists. There is no other flow-control mechanism.

§2.5 Global progress indexes

The pipeline holds five monotonic atomic counters and one shutdown flag:

• sequencer_index — next tx_id to be handed out.
• compute_index — last tx_id executed by the Transactor.
• commit_index — last tx_id durably written by the WAL.
• snapshot_index — last tx_id reflected in the Snapshotter.
• seal_index — last segment id sealed.
• running — global shutdown flag.

The indexes are padded to separate cache lines so that a publication on one does not invalidate the cache line of an adjacent index. They are strictly monotonic and never go backwards. The mapping commit_index ≥ N ⇒ every transaction ≤ N is committed (no committed gaps) is what makes per-call wait levels (§14) meaningful: a single threshold check on a single counter resolves any wait.

§2.6 Backpressure and shutdown

Backpressure is implicit and propagates through ring and queue saturation: a slow Snapshotter stops releasing ring slots (it releases only up to the durability watermark, §13.5), so the ring fills and the Transactor stops finding free slots; the stalled Transactor fills the sequencer→transactor queue, which stalls submit(). There is no rate limiter, leaky bucket, or admission control. Shutdown is the inverse signal: Pipeline::shutdown() clears running; every stage’s idle loop checks the flag and exits. The shutdown signal does not need to be observed immediately — the queues drain to a safe state regardless of the precise moment a stage notices.

§3 Sequencer

§3.1 Placement on the caller’s thread

The Sequencer has no dedicated thread. It runs synchronously inside Ledger::submit on whichever thread the caller invokes from. Its work is intentionally trivial: one atomic increment plus one queue push. Adding a thread (and the cross-thread channel that would imply) would only add latency.

§3.2 ID assignment

ID assignment is a single atomic increment of sequencer_index. submit(operation) reserves one ID and stamps the resulting Transaction. submit_batch(operations) reserves a contiguous run of len(operations) IDs in one atomic step; per-operation IDs inside the batch are start_tx_id + index. Batches are atomic at the sequencer: their IDs are guaranteed consecutive even under concurrent submitters.

§3.3 Permanence of position

Once the Sequencer assigns an ID, the transaction’s position in the global order is permanent. No later stage can reorder, reassign, or skip an ID. This is what allows downstream stages (Transactor, WAL, Snapshotter) to publish progress as a single monotonically advancing index per stage — there are no holes to fill or gaps to wait on.

§3.4 Backpressure spin/yield policy

When the sequencer→transactor queue is full, the submitter spins, yielding the thread periodically while the spin persists. There is no timeout — submit returns only after the operation is queued. Spinning (rather than parking) is preferred here because submit-side latency is the primary concern: a parked thread misses the moment space appears.

§4 Transactor — the single writer

§4.1 Single-writer rule

The Transactor is one dedicated thread that processes one transaction at a time, in strict ID order. This is the source of all correctness guarantees: there are no races, no conflicts, no need for locks or conflict resolution. Throughput does not come from parallelising execution — it comes from sequencing, durability, and snapshotting running concurrently around the Transactor. The single-writer ceiling is the upper bound on per-ledger write throughput. [ADR-0001]

§4.2 Thread-local state

The Transactor’s working state — balance cache, per-step entries buffer, position marker, current tx_id, current fail_reason — lives entirely on the Transactor thread and never crosses a thread boundary. The WASM runtime instance bound to a Transactor inherits this constraint. Single ownership means no locks are needed on the hot path; host calls (§6.7) mutate the same state through interior mutability with no contention.

§4.3 Balance cache layout

The Transactor’s balance cache is a flat array indexed directly by account_id. Lookups and updates are O(1) array accesses with no hashing, no probing, no resizing. The array is pre-allocated to max_accounts at construction and never grows. [ADR-0002]

§4.4 The max_accounts ceiling

Any operation referencing an account_id ≥ max_accounts is rejected with ACCOUNT_LIMIT_EXCEEDED before the operation can mutate any state. This is the price of the direct-indexed array layout (§4.3) — the account-ID space must be sized at startup. Resizing would require pausing the Transactor and reallocating every read-side cache (§8.3), which is not worth the complexity given how rarely a deployed ledger needs to grow its ceiling. [ADR-0002]

§4.5 Linearizability of the write path

The Transactor never reads stale balances. Because it is the sole writer and operates on the in-memory cache (§4.3), every read it performs during operation execution observes the cumulative effect of every earlier transaction. The write path is therefore linearizable by construction; the read path requires an explicit submit_wait(snapshot) (§14.4) for the same guarantee.

§4.6 In-place assembly in the ring

The Transactor writes the records it produces directly into the transaction ring’s uncommitted region — the slots above the write frontier that it owns exclusively and no reader can see (§13.4). There is no separate staging buffer: the ring is the per-transaction scratch space. As it pushes followers it accumulates the two derived metadata fields — a running CRC digest (§4.10) and a follower count (pending_items) — so finalize needs no second pass. On finalize it assembles the trailing TxMetadata once with those values and publishes the whole transaction atomically by advancing the write frontier; on failure it discards the uncommitted tail (§4.12).

§4.7 The tx_start_index marker

The Transactor records the ring index where the current transaction’s first follower lands (tx_start_index). Verify (§4.11) and rollback (§4.12) scope their work to the slots from tx_start_index to the write head — the in-flight transaction’s records — and never touch already-published ones. The CRC is folded incrementally as those records are pushed (§4.10), not re-walked from the marker.

§4.8 Built-in operation semantics

For built-in operation types the Transactor runs native Rust:

• Deposit { account, amount } — debit(SYSTEM_ACCOUNT_ID, amount), credit(account, amount).
• Withdrawal { account, amount } — checks balance(account) ≥ amount; on success, debit(account, amount), credit(SYSTEM_ACCOUNT_ID, amount); on failure, fail with INSUFFICIENT_FUNDS.
• Transfer { from, to, amount } — checks balance(from) ≥ amount; on success, debit(from, amount), credit(to, amount); on failure, fail with INSUFFICIENT_FUNDS.

Each credit / debit produces one TxEntry record with the running computed_balance field already populated.

§4.9 Default INSUFFICIENT_FUNDS protection

For built-in operations, the Transactor refuses any change that would drive an account balance negative (it checks before the debit). This is the default protection. WASM functions (§6) are the only mechanism for producing a negative balance: a function must explicitly call get_balance itself and decide to proceed. There is no configuration knob to disable the built-in check.

§4.10 Per-transaction CRC32C

The Transactor folds a running CRC32C over each follower record as it is pushed, in push order, then — when the transaction finalizes — appends the trailing TxMetadata (with its own crc32c field zeroed) into the digest and stores the result into that field. The CRC therefore covers the followers (in order) followed by the zeroed-crc metadata, and is computed incrementally rather than by re-walking the transaction after the fact. It uses CRC32C (Castagnoli) — hardware-accelerated on modern CPUs. Because the field participates in the digest while being part of the record, it is zeroed during computation; a consumer verifies by recomputing in the same order — the followers, then the zeroed metadata (§18.3, [ADR-0020]).

§4.11 Zero-sum invariant

Every transaction must net to zero: sum(credits) == sum(debits) across all TxEntry records of that transaction. The Transactor verifies this after operation execution and before any state is committed; a violation raises ZERO_SUM_VIOLATION and the transaction is rolled back (§4.12). The invariant is per-transaction — there is no cross-transaction arithmetic. Because each individual transaction nets to zero, the sum of all balances across all accounts is always zero — an emergent property, not a configuration option.

§4.12 Rollback

When verify fails or a WASM execution rejects, the Transactor reverses every credit/debit it produced for this transaction against the live balance cache and discards the uncommitted tail in the ring (rolling the write head back to tx_start_index, §4.7). It then sets fail_reason and finalizes the transaction as a lone trailing TxMetadata (sub_item_count = 0) — see §4.13. There is no leading metadata to preserve: the metadata is materialized only at finalize. The cost is O(entry_count) and is acceptable because typical entry counts are small.

§4.13 Failed transactions are journaled

A transaction that fails — whether by deduplication, an execution error, or a zero-sum violation — still produces a TxMetadata record with a non-zero fail_reason, plus any TxLink records that describe the relationship (e.g. Duplicate linking to the original). This metadata flows through the WAL and the rest of the pipeline like any other record. The audit trail is therefore complete: every submitted operation that reached the Transactor leaves a trace, success or failure. [ADR-0001]

§4.14 No torn writes per account

Every individual credit or debit is a single atomic store. A concurrent reader on the read-side balance cache (§8.3) will never observe a partially-written balance for any single account — the value is always a complete, valid i64.

§4.15 Intermediate state is visible across accounts

Within a multi-step operation (a Transfer, or a Function issuing several credit/debit calls), the Transactor applies each step in sequence. A concurrent reader can observe some of these stores before others have landed — for example, seeing a credit applied to one account before the matching debit on another has landed. This is by design. last_tx_id returned by get_balance is the signal that a transaction is fully settled across all accounts: it is updated only after all entries of a transaction have been applied (§8.5). Callers who need to see a multi-account transaction atomically should wait for last_tx_id to advance past their tx_id or use submit_wait(snapshot).

§4.16 tx_id is on state, not on the host call

The current tx_id is set on the Transactor’s thread-local state at the start of each transaction; host calls (§6.7) read it from there. It is never passed across the WASM host-call boundary as a parameter. This avoids the cost of an extra call argument and removes the risk of a function caching or forging a stale tx_id.

§4.17 Rejection registry

Rejected transactions are recorded in a concurrent map keyed by tx_id, shared with the Ledger so that status queries can resolve Error(reason) for any failed tx_id (§2.2). The map is sparse — successes are not recorded — because in steady state the overwhelming majority of transactions succeed.

§5 Deduplication and the active window

§5.1 Always on, no global toggle

Deduplication is always on. There is no configuration that disables it system-wide. The only opt-out is per-transaction (§5.4).

§5.2 Active window and flip-flop cache

The active window size equals transaction_count_per_segment — the same number that drives WAL segment rotation (§7.6). The dedup cache is two user_ref → tx_id hash maps, active and previous. When tx_id crosses the segment boundary, the maps flip: active becomes previous, the previous map’s storage is reused as the new (cleared) active. Effective dedup coverage is therefore N..2N transactions — at least one full window, at most two.

§5.3 Determinism: count-based, not time-based

The flip is driven by tx_id, not by wall-clock time. The same N transactions always provide the same protection, whether they arrive in one second or one hour. Idempotency guarantees do not degrade under load spikes and do not silently widen during quiet periods.

§5.4 user_ref = 0 opt-out

A user_ref value of 0 opts the individual transaction out of the dedup check. The cache itself remains active and continues to record every user_ref ≠ 0. There is no path that disables the cache globally short of recompiling.

§5.5 Hit handling

On a hit, the Transactor emits TxMetadata (with fail_reason = DUPLICATE) followed by TxLink { kind: Duplicate, to_tx_id: original_tx_id }. The caller therefore always recovers the original tx_id for an idempotent retry, not just a “rejected” status. [ADR-0009]

§5.6 Recovery seeding

On startup, the dedup cache is seeded by walking each WAL transaction inside the active window (§12.3). Entries outside 2 × window are dropped; entries within the window are placed into the correct half (active vs previous) based on their relative position. Post-recovery dedup behaviour is therefore identical to pre-crash — a duplicate submitted before and after a restart is rejected the same way.

§6 WASM runtime — embedded execution

§6.1 Inside the Transactor, not a separate stage

The WASM runtime is not a separate pipeline stage. It is a component inside the Transactor and executes on the Transactor thread. Operation::Function is dispatched by the Transactor like any built-in operation (§4.8); the only difference is how TxEntry records are produced — by calling into a sandboxed WASM handler that issues credit/debit host calls instead of running a hard-coded Rust branch. This placement means a function execution inherits every Transactor invariant: single-writer ordering (§4.1), zero-sum (§4.11), atomic rollback (§4.12), deduplication (§5), and CRC (§4.10) — without any new coordination.

§6.2 Pipeline-identical flow

A Function operation flows through Sequencer → Transactor → WAL → Snapshotter exactly like a Deposit. WAL segmentation, sealing, snapshotting, and replay treat function-produced entries identically to native ones. There is no second persistence path, no second recovery path, no second wait-level mechanism.

§6.3 One engine, one linker per ledger

There is one wasmtime engine and one host-import linker per ledger, shared across every Transactor. The linker carries the host imports (§6.7); the engine compiles and caches modules. Constructing either is expensive, so a ledger does it once.

§6.4 Shared registry

The shared registry maps name → (version, crc32c, compiled module). It is protected by a reader-writer lock and is touched only on register, unregister, and cache miss / revalidation — never on every transaction. The hot path (§6.5) avoids the lock entirely.

§6.5 Per-Transactor caller cache

Each Transactor holds its own local cache of name → compiled handler, tagged with a sequence number that records the registry version the entry was verified against. Resolution is:

1. Look up name in the local cache.
2. If the cached sequence matches the registry’s current value, use the cached handler directly with no lock.
3. Otherwise, take the registry read lock, fetch the current entry, instantiate, install into the local cache, stamp the entry with the new sequence.

Cache invalidation is per-name: registering or unregistering foo does not evict the cached entry for bar.

§6.6 Required export

Every registered binary must export exactly one function: execute(i64, i64, i64, i64, i64, i64, i64, i64) → i32. The eight parameters correspond to the Function operation’s [i64; 8] (§1.5). The i32 return is the function’s status: 0 is success, anything else is mapped per §6.12. [ADR-0014]

§6.7 Host imports

The host module is "ledger" and exports exactly three imports:

• ledger.credit(account_id: i64, amount: i64) → ()
• ledger.debit(account_id: i64, amount: i64) → ()
• ledger.get_balance(account_id: i64) → i64

Each call mutates the Transactor’s thread-local state (§4.2): credit and debit append a TxEntry to the per-step buffer; get_balance reads the live balance cache. The current tx_id is pulled from the state (§4.16), not passed through the call. [ADR-0014]

§6.8 Determinism rule

The host ABI is intentionally narrow: no clocks, no random, no I/O, no threads, no atomics. Every legal function is therefore a pure mapping from (params, observed balances) → (status, credits, debits). This is not a stylistic choice — it is what makes the WAL replayable on a follower without re-running the WASM code (§6.9). Adding any non-deterministic host call would make replication unsafe.

§6.9 Replication implication

Because of §6.8, a future Raft follower can apply the WAL records produced by a function on the leader directly — without instantiating the WASM module, without re-running execute, and without observing any of the function’s internal state. The leader is the only node that needs to compile and run the binary; followers see only the TxEntry/TxLink/TxMetadata stream the leader produced. This is what enables the future cluster (Stage 3).

§6.10 Atomicity of function execution

Host-call effects accumulate in an isolated execution context on the Transactor side, not in WASM-visible globals. Nothing is applied to the live balance cache until all of the following hold: execute returned 0, the captured credits and debits balance (§4.11), and no wasmtime-layer error occurred. If any of those fail, the entire batch of host calls is discarded — the same rollback machinery as a built-in operation (§4.12). Nothing is published to the transaction ring, so no reader ever sees a partial function execution.

§6.11 Wasmtime-layer failure mapping

A wasmtime layer failure — link error, instantiation error, or runtime trap — is mapped to INVALID_OPERATION = 5 and rolled back. The mapping is single-valued by design: from outside the runtime, any infrastructure-level WASM failure looks the same. The function author’s own status codes (§6.12) live in a disjoint range.

§6.12 Status code passthrough

The i32 return of execute is interpreted as follows:

• 0 → success (provided zero-sum holds; §4.11).
• 1..=7 → mapped to the corresponding standard FailReason.
• 8..=127 → reserved; treated as INVALID_OPERATION.
• 128..=255 → user-defined; passed through into TxMetadata.fail_reason unchanged.
• Anything else (negative, > 255) → treated as INVALID_OPERATION.

User-defined codes are the WASM author’s contract with their callers; the ledger never reinterprets them. [ADR-0014]

§6.13 Entry-limit enforcement

If a function emits more than 65535 credit/debit calls in a single execution, the u16 sub_item_count field of TxMetadata cannot encode the count. The Transactor fails the transaction with ENTRY_LIMIT_EXCEEDED and rolls it back. Function authors who need more than 65535 entries must split the work across multiple transactions.

§6.14 Audit tag

Every committed function-produced transaction stamps TxMetadata.tag with b"fnw\n" ++ crc32c[0..4] — the four-byte literal "fnw\n" followed by the first four bytes of the executing binary’s CRC32C. Any past transaction can therefore be traced back to the exact binary that produced it, even after registrations have changed.

§6.15 Names and versions

Function names are validated at registration: ASCII snake_case, ≤ 32 bytes, null-padded to 32 in the WAL record. Each register or override bumps the per-name version by one, starting at 1, with a ceiling of 65535. The (name, version, crc32c) triple uniquely identifies the executing binary in every transaction. [ADR-0014]

§6.16 Registration is durable

RegisterFunction only returns after all four of the following have happened: the binary has been validated, written atomically to disk, a FunctionRegistered WAL record has been committed, and the compiled handler has been installed in the live WASM runtime. A subsequent Operation::Function is therefore guaranteed to see the new version. After a crash, recovery rebuilds the registry from a paired function snapshot plus the FunctionRegistered records that follow it (§12.4) — the live runtime always matches what the WAL says it should be.

§6.17 Unregistration

Unregistration emits a FunctionRegistered record with crc32c = 0 and writes a 0-byte file under the next version on disk. The binary history is preserved as a sequence of versions; the function registry’s current state is the last record seen. The WAL is authoritative; the on-disk file is a secondary signal. [ADR-0014]

§6.18 Snapshotter applies registry events

FunctionRegistered records flow through the WAL like any other record (§7.4). The Snapshotter (§8.7) is the stage that actually loads or unloads the compiled handler in the shared WASM runtime: it reads the binary off disk on a load, and unloads on an unregister. This is what keeps the live runtime synchronised with the WAL on every node, leader or follower.

§7 WAL stage — durability orchestration

§7.1 Two threads, one role each

The WAL stage runs as two cooperating threads. The WAL Writer (wal_write) owns the active segment and all disk I/O for record writes; the WAL Committer (wal_commit) owns fdatasync calls only. Splitting the two means the Writer never blocks behind a fdatasync — it continues appending records to the next batch while the Committer is mid-sync. The hot path is dominated by fdatasync latency (~hundreds of µs on NVMe); isolating it from record writes is what lets the WAL keep up with the Transactor. [ADR-0011]

§7.2 Coordination between Writer and Committer

The two threads share two counters and one segment handle:

• last-written — the highest tx_id whose record has been written (but not necessarily synced) to the active segment file. The Writer publishes; the Committer observes.
• last-committed — the highest tx_id whose record has been fdatasync’d. The Committer publishes; the Writer observes. This is what commit_index exposes to the rest of the pipeline (§2.5) and what the wal wait level (§14.1) gates on.
• The active segment’s sync handle, swappable across rotations (§7.6) so the Committer always syncs the segment that is currently open.

There is no other coordination — no condvars, no channels.

§7.3 Writer buffer

The Writer copies records out of the ring into a bounded pending-write buffer, batched so a whole iteration’s worth of records reaches disk in one write syscall. The buffer is sized in advance — never resized — so the WAL stage’s memory footprint is fixed. It forwards records nowhere: the Snapshotter reads the ring independently (§8.4).

§7.4 Writer per-iteration loop

Each iteration the Writer copies the newly published ring records into the active segment’s pending-write buffer, performs a single buffered write syscall (one per iteration, not per record), advances last-written, and rotates the segment if the transaction-count threshold is met (§7.6). It does not gate or forward anything to the Snapshotter — durability visibility is the Snapshotter’s own concern (§7.5). As records are written and the Committer syncs them, commit_index (§2.5) advances.

§7.5 Durability rule, enforced reader-side

The durability rule still holds: the Snapshotter — and therefore get_balance, queries, and the snapshot wait level — never sees a transaction that is not yet on disk. But the WAL no longer enforces it by forwarding only durable records. Instead the Snapshotter reads the ring and gates itself: it applies a transaction only once the trailing TxMetadata’s tx_id ≤ commit_index (§8.4). The Writer therefore keeps no per-transaction “expected record count” bookkeeping; it writes whatever the ring publishes, and a transaction is recognised as complete at its trailing metadata. A transaction may span a segment boundary (§18.3, [ADR-0020]).

§7.6 Segment rotation

The Writer rotates segments on a transaction-count threshold: when the number of transactions in the active segment reaches transaction_count_per_segment. This is the same boundary that drives dedup window flips (§5.2) and hot-index sizing (§9.6). Rotation performs a synchronous fdatasync (the Writer cannot defer this one to the Committer because the next step closes the file), closes the segment, opens the next one, and hands the new sync handle to the Committer. No in-band markers are written — segment boundaries are determined by file rename and the .seal marker created later by the seal thread. Rotation is a transient cost: the pipeline pauses only for the synchronous sync and the open of the next file. [ADR-0013]

§7.7 Committer loop

Driven by the shared wait strategy (§13.2), the Committer samples last-written, calls fdatasync, and publishes the sample as last-committed; it picks up a fresh sync handle whenever rotation swaps the active file. It never parses entries, never touches the Writer’s buffer, and never inspects any queue — its single responsibility is moving last-committed forward as fast as storage allows.

§7.8 Backpressure from a slow reader

There is no outbound queue to fill. A persistently slow Snapshotter stops releasing ring slots (it releases only up to the durability watermark, §13.5), so the ring eventually fills and the Transactor — the sole producer — stalls for want of free slots (§2.6). The WAL Writer, an independent reader, is never itself blocked by the Snapshotter; the two consume the ring at their own pace.

§7.9 Failure handling

A failed write or fdatasync is a hard panic. There is no retry loop on durability failures: returning an error to a submitter that has already been told its transaction was committed would be a worse violation of the contract than a process exit. Recovery (§12) is responsible for putting the WAL back into a consistent state on restart.

§8 Snapshotter — apply path

§8.1 Apply, don’t re-execute

The Snapshotter applies committed entries to the read-side state. It does no business logic and never re-executes operations. Because every TxEntry carries computed_balance, the Snapshotter’s job is a sequence of store(account_id, computed_balance) operations plus index maintenance (§9). This is the entries-based execution model and the reason recovery and replication are cheap. [ADR-0001]

§8.2 Why a separate stage

The Transactor’s balance cache (§4.3) is write-side truth, private and optimised for writes. The Snapshotter maintains read-side truth and is optimised for concurrent readers. Separating them means readers never block writers, and the Transactor never needs to coordinate with query threads.

§8.3 Read-side balance cache

The read-side cache is a flat array of atomic balances, sized to max_accounts. The Snapshotter writes; many readers can load concurrently without locks. The array is pre-allocated at startup and never resized. [ADR-0002]

§8.4 Ring entries and the query queue

The Snapshotter draws from two independent sources, in a fixed order each iteration:

• The transaction ring — the published record stream (§13.4), buffering a transaction’s followers until the trailing TxMetadata (§8.5).
• A query-only SPSC queue — GetTransaction / GetAccountHistory requests enqueued by callers (§10).

A query is no longer interleaved with entries in one FIFO, so read-your-own- writes is not automatic from enqueue ordering; it rests on the wait levels (§14). A caller that waits for snapshot before querying is guaranteed snapshot_index ≥ tx_id, and because the loop applies a transaction’s entries before advancing snapshot_index, the subsequent query observes them. A query issued without that wait races the apply loop — the same trade-off a none-level submit makes.

§8.5 Apply algorithm

The Snapshotter walks the ring from its cursor, buffering a transaction’s TxEntry / TxLink followers. When the trailing TxMetadata arrives it checks the durability gate: if tx_id > commit_index the transaction is not yet durable, so the walk stops and resumes next iteration. Otherwise it applies the buffered followers together — updating the hot index (§9.4) and publishing each TxEntry’s computed_balance to the read-side balance cache — then advances snapshot_index (§2.5) to this tx_id and, as the ring’s sole releaser, reclaims the consumed slots (§13.5). Applying at the trailer is the only point the read-side index advances, so readers never observe a partially-applied or not-yet-durable transaction. FunctionRegistered takes the registry path of §8.7.

§8.6 last_tx_id as the freshness signal

get_balance returns the current balance plus last_tx_id — the highest tx_id whose every entry has been applied to the read-side cache. last_tx_id is the tool callers use to reason about freshness without blocking, and it is the signal that confirms multi-account transactions are fully settled (§4.15). For a linearizable read, the caller waits for last_tx_id to pass their tx_id, or uses submit_wait(snapshot) (§14.4).

§8.7 FunctionRegistered handling

FunctionRegistered records are not financial transactions and do not advance snapshot_index. On a register record, the Snapshotter reads the binary off disk and installs it in the shared WASM registry (§6.4). On an unregister record (crc32c == 0, §6.17), the Snapshotter unloads the function from the registry. Errors at either step are logged; they do not stop the apply loop. The Snapshotter is therefore the only stage that mutates the live runtime registry while the system is running — the Transactor reads through its per-Transactor caller cache (§6.5), which revalidates lazily on miss.

§8.8 Snapshotter is the bottleneck for queries

The Snapshotter owns the indexer exclusively; queries (§10) execute inline on the Snapshotter thread. Concurrent queries serialise on this thread. The work per query is a couple of array lookups and at most a chain walk bounded by limit, so the bottleneck is generous, but it exists — a workload that is dominated by query throughput should batch queries or accept that queries compete with apply for the same thread.

§9 Hot indexes — TransactionIndexer

§9.1 Three buffers, all pre-allocated

The hot transaction index is a single in-memory structure with three pre-allocated buffers, sized at construction and never resized:

• circle1 — for each tx_id, a pointer to that transaction’s entries in circle2.
• circle2 — entry storage with per-account chain links.
• account_heads — for each account, a pointer to that account’s latest entry in circle2.

A separate map keyed by tx_id holds link records (sparse — most transactions have none — so a hash map is acceptable here). All three array sizes must be powers of two. [ADR-0008]

§9.2 Why power-of-two

All three lookups reduce to “id AND mask” to find a slot, where mask is size − 1. The power-of-two requirement turns the lookup into a single bitwise operation. The cost is at most ~2× memory in the worst case (desired size just above a power of two); the alternative (modulo) costs a division on every query, which is far more expensive at ledger throughput.

§9.3 circle1 — direct-mapped tx slot table

Each circle1 slot stores a tx_id, an offset into circle2, and the entry count for that transaction. The slot index for a transaction is its tx_id masked by the circle1 size. A slot whose stored tx_id does not match the queried tx_id is evicted — overwritten by a colliding transaction that mapped to the same slot. Eviction is silent: the slot simply now belongs to a newer transaction. There is no eviction queue, no cleanup pass. Lookups detect eviction by comparing the slot’s tx_id to the queried one and falling back to disk on mismatch (§10.2).

§9.4 circle2 — entry storage and per-account chains

Each circle2 slot stores one TxEntry’s payload (tx_id, account_id, amount, kind, computed_balance) plus a previous-link — the circle2 index of the same account’s previous entry, or zero if there is no previous. circle2 is written sequentially with a write head that advances on every insert and wraps modulo circle2’s size. On insert, the indexer reads the account’s head from account_heads; if the head is for the same account, the new entry’s previous-link is set to it, otherwise the link is empty. Then the entry is written, account_heads is updated to point at the new slot, and the write head advances. Account history walks (§10.3) follow the previous-link chain backwards in time until the chain ends, an evicted slot is detected (account-id mismatch), the transaction predates the requested lower bound, or the caller’s limit is reached.

§9.5 account_heads — direct-mapped account head table

Each account_heads slot stores (account_id, circle2_idx). The slot index is the account id masked by the account_heads size. The slot stores the account id it belongs to so a colliding lookup can detect eviction the same way circle1 does: a mismatch means the head was overwritten and the chain head is gone, and the caller must fall back to disk for any history older than the current chain. The size is next_power_of_two(max_accounts), which means in practice every account has its own slot; collisions only occur if the account-id space is sparser than the table.

§9.6 Sizing rule

The three sizes are derived from transaction_count_per_segment and max_accounts:

• circle1 — one slot per transaction in the active window, rounded up to a power of two.
• circle2 — enough entry slots for the active window assuming roughly two entries per transaction (transfers), rounded up.
• account_heads — one slot per account, rounded up.

Changing the active window changes the hot indexes and the dedup window in lockstep; there are no separate knobs. [ADR-0013]

§9.7 Recovery seeding

The recovery sequence (§12) seeds the indexer directly: as it walks WAL records from the last snapshot through the active segment’s tail, each metadata, entry, and link record is replayed into the indexer through the Snapshotter’s recovery hooks. By the time the Snapshotter thread starts, the indexer is already populated for everything in the active window.

§10 Query path — hot/cold routing

§10.1 Query types

Two query types reach the Snapshotter through its query queue (§8.4):

• GetTransaction { tx_id } — return the entries (and links) of one transaction.
• GetAccountHistory { account_id, from_tx_id, limit } — return up to limit entries that touched account_id, newest first, stopping at from_tx_id (exclusive lower bound; can be omitted).

A third query type — get_balance(account_id) — is not routed through the Snapshotter. It is a direct read against the read-side cache (§8.3) and never blocks.

§10.2 GetTransaction resolution

The Snapshotter dispatches GetTransaction(tx_id) to indexer.get_transaction(tx_id), which: looks up circle1[tx_id & mask]; returns None on empty (tx_id == 0) or mismatch (eviction); otherwise reads entry_count consecutive slots starting at circle2[offset], comparing each slot’s tx_id to the queried tx_id — a mismatch means the circle2 slot was overwritten between the metadata insert and the query, and the indexer returns None. Links are fetched separately from the links map. The QueryResponse::Transaction(...) carries Option<TransactionResult>: Some on a hit, None on a miss. A None is the signal to the caller that this transaction is no longer in the hot tier and must be read from disk via the on-disk transaction index (Stage 2).

§10.3 GetAccountHistory resolution

GetAccountHistory reads account_heads[account_id & mask]; if the stored account_id does not match, returns an empty vec (account head evicted). Otherwise walks prev_link backwards through circle2, stopping on account_id mismatch (chain broken by eviction), tx_id < from_tx_id (past the requested window), limit reached, or prev_link == 0 (chain head). Results are returned newest-first. An empty vec from a query that should have had results is the caller’s signal to fall back to the on-disk account index (Stage 2).

§10.4 Hot tier vs cold tier

The hot tier is everything currently in the indexer; the cold tier is everything sealed to disk. The boundary is not a function of tx_id absolute value but of how recently each tx_id’s slot has been overwritten — a low-traffic account’s chain can survive deep into history, while a hot account’s chain may be evicted within one segment. Cold-tier reads are sealed-segment lookups (the on-disk transaction index and the on-disk account index built by Seal — §11.4). The mechanism by which the caller routes a hot miss to the cold tier is the same for both query types: the hot result returns None / empty, and the caller reissues against the cold-tier API.

§10.5 Synchronous response via callback

A query carries a one-shot callback. The convention is: caller creates a per-request channel, wraps the sender in a closure, enqueues the request, and blocks on the receiver. The Snapshotter computes the response inline (§8.8) and invokes the callback before continuing the apply loop. There is no separate response channel, no shared state for partial results, no async runtime.

§10.6 Status and balance queries

Two query paths bypass the query queue entirely:

• get_balance(account_id) — a direct atomic load on the read-side balance cache. Does not block, does not touch the Snapshotter, returns whatever balance the cache currently holds along with last_tx_id (§8.6).
• get_transaction_status(tx_id) — derived (§2.2) from the pipeline indexes plus the rejection registry (§4.17). Does not block.

Neither of these traverses the Snapshotter’s query queue; both serve very high read volumes without contention.

§11 Seal stage — segment finalisation and snapshots

§11.1 What sealing means

Sealing is the act of making a closed segment safe to recover from. The WAL Writer (§7.6) closes a segment when the transaction-count threshold is reached; the closed file on disk has no integrity-check sidecars and no built indexes. Sealing turns this into a fully recoverable artifact: it computes a file-level CRC, writes sidecar files, builds the on-disk transaction and account indexes for cold-tier queries (§10.4), and — at the configured frequency — emits a balance snapshot and a function snapshot. The exact file format of all of these is in Stage 2; the mechanism is described here.

§11.2 Independent thread, polling loop

Seal runs as one dedicated thread (seal) on a polling loop. There is no queue from the WAL or any other stage to Seal — the WAL closes segments by renaming files on disk, and Seal discovers work by listing the storage directory each iteration. The poll interval is seal_check_internal (§15.6, default 1 s). Between polls the thread sleeps; on shutdown the loop checks running and exits. Seal is not on the hot path of any submission; missing the threshold by a few seconds is harmless. The cost of the poll is one directory listing per interval.

§11.3 Per-segment seal procedure

For each closed-but-unsealed segment, Seal computes the file-level CRC, writes the .crc and .seal sidecars, builds the on-disk transaction and account indexes (§22), replays the segment into Seal’s private balance vector and function map (§11.4), and — at the snapshot frequency boundary (§11.5) — writes the paired snapshot files. The sealed segment id is then published (advances seal_index, §2.5). A failure during sidecar or index construction is logged; the segment is left unsealed and retried on the next poll.

§11.4 Why Seal owns its own balance vector and function map

Seal holds two pieces of state separate from the rest of the pipeline:

• A balance array sized to max_accounts, updated from each entry’s computed_balance as Seal walks the segment. This is the source for the balance snapshot (§11.5); it is not the read-side cache (§8.3).
• A map of name → (version, crc32c), updated on each FunctionRegistered record. Unregister records (crc32c == 0) are kept in the map so the snapshot preserves the audit trail.

Why a separate copy: Seal must snapshot the committed-and-sealed state, which lags commit_index. The Snapshotter’s read-side cache is the latest-committed state and can race ahead. A snapshot taken from either of those would either include uncommitted state or be racy. Reading off the segment, on a thread that does not care about the freshest tx, gives Seal a consistent point-in-time view at zero synchronisation cost.

§11.5 snapshot_frequency rule

When snapshot_frequency > 0 and the just-sealed segment’s id is a multiple of snapshot_frequency, Seal emits two snapshot files for that segment:

• A balance snapshot — only non-zero balances, sorted by account id for determinism.
• A function snapshot — every entry from Seal’s function map, sorted by name. Written even when the map is empty so recovery can always jump straight to the latest snapshot boundary instead of replaying WAL from segment 1. [ADR-0014]

When snapshot_frequency == 0, no snapshots are emitted and the system must replay from segment 1 on every recovery. The default is snapshot_frequency = 4.

§11.6 Pre-seal during recovery

Recovery (§12) may discover a closed-but-unsealed segment from a crash between WAL rotation and the next Seal poll. Rather than start the Seal thread early, Recover invokes the same per-segment seal procedure (§11.3) synchronously and publishes the resulting segment id. After recovery finishes, the Seal thread starts and resumes its normal poll. This keeps the Seal thread’s lifecycle simple: one thread, one loop, one shutdown — no special “first iteration” mode.

§11.7 Seal does not touch the live indexes

Seal does not write to the Snapshotter’s indexer (§9), the read-side balance cache (§8.3), or any of the dedup state (§5). Its outputs are on-disk only: sidecar files, on-disk indexes, snapshot files. The in-memory state is owned by other stages and updated by them; Seal’s job is to make the disk side recoverable. The one in-memory effect Seal has is publishing seal_index (§2.5), which ctl tools and recovery read.

§11.8 disable_seal for benchmarks

disable_seal = true (§15.7) skips starting the Seal thread entirely. This removes all sealing I/O — sidecars, indexes, snapshots — from the benchmarked path. It is only a benchmarking knob: a ledger run with disable_seal = true cannot recover from a crash beyond the active segment, and cold-tier queries (§10.4) will return empty for any sealed segment. Production must always have it false.

§12 Recovery — in-memory orchestration

§12.1 Recovery is synchronous and runs before any stage starts

Ledger::start calls into Recover before spawning Sequencer, Transactor, WAL, Snapshotter, or Seal threads. Recovery is therefore single-threaded and observes a quiescent disk; nothing else is reading or writing the data directory. Once recovery returns, the pipeline is in a state that is observationally identical to the state that existed at last_committed_tx_id immediately before the previous shutdown.

§12.2 Pre-seal of unsealed segments

The first thing Recover does is scan for closed-but-unsealed segments (§11.6). For each one it runs the per-segment seal procedure synchronously and publishes the resulting id. After this step every segment on disk except the active one is sealed.

§12.3 Snapshot load and WAL replay

Recovery loads the latest balance and function snapshots — seeding the Transactor’s balance cache (§4.3), Seal’s balance vector (§11.4), the read-side balance cache (§8.3), and the live WASM runtime — then replays sealed segments newer than the snapshot’s segment id, then replays the active segment — buffering each transaction’s followers until its trailing TxMetadata validates the group, and trusting the tail only as far as the last complete metadata boundary (Stage 2). The follower buffer carries across segment seams, so a transaction split over a boundary reassembles. Finally sequencer_index is restored to last_tx_id + 1 (§16.2). The fixed ordering is the load-bearing constraint (§12.4).

§12.4 Order requirement

The order in §12.3 is not negotiable. Loading the balance snapshot before replaying segments ensures the replay only re-applies entries that postdate the snapshot. Loading the function snapshot before replaying FunctionRegistered records ensures the runtime starts from the snapshotted set and only diffs forward. Pre-sealing (§12.2) before the snapshot search ensures the latest snapshot is always available — without it, a snapshot trigger that fired on a segment that was not yet sealed at crash time would be missed.

§12.5 Recovery seeds, not replays, the dedup cache

Dedup recovery is not a replay of every WAL record into the cache: it is a windowed seed (§5.6). Recover walks the active window’s worth of WAL records (the last 2 × transaction_count_per_segment) and inserts each (user_ref, tx_id) into the appropriate half of the flip-flop cache. Older user_refs are not re-inserted; they are outside the dedup window and would be evicted on the next flip anyway.

§12.6 Failed transactions are replayed too

A TxMetadata with fail_reason ≠ 0 represents a transaction that failed but was journaled (§4.13). Recovery still walks it and updates the indexer and dedup, but the balance vectors are not modified (failed transactions emit no TxEntry records that move balance, by construction of the rollback in §4.12). After recovery the rejected transaction’s tx_id continues to resolve through the rejection registry (§4.17), populated in the same pass.

§12.7 No partial recovery

Recovery either succeeds completely or fails the process. There is no partial-recovery mode where some segments are replayed and others skipped. A corrupt segment in the middle of the chain is fatal at startup; a corrupt tail of the active segment is the one tolerated case (Stage 2). The principle: a started ledger is a ledger that is in a known-good state.

§13 Inter-stage transport, backpressure, wait strategies

§13.1 Lock-free transport

The record stream uses a single lock-free transaction ring (§13.4); the submit and query paths use lock-free, fixed-capacity, single-producer / single-consumer queues (§2.4). Each stage owns its data completely; no shared mutable state crosses stage boundaries. Queue capacity is sized at construction from queue_size (§15.2), the ring from ring_size (§15.10).

§13.2 Wait strategies

Every stage’s idle / backpressure loop is driven by the shared wait strategy. Four variants exist:

• LowLatency — pure spin. CPU is burned unconditionally; latency is minimised. Suitable for dedicated cores.
• Balanced (default) — short spin, then yield, then park briefly. Adapts: under load it behaves like LowLatency; when idle it costs nothing.
• LowCpu — no spin, short yield, then park. For deployments where background CPU cost matters more than latency.
• Custom — explicit spin / yield / park parameters.

A single wait strategy is shared by every stage of one Pipeline; there is no per-stage override.

§13.3 Backpressure as ring and queue saturation

There is no rate limiter, leaky bucket, or admission control. Slowness anywhere propagates upstream: a stalled Snapshotter stops releasing ring slots (§13.5), filling the ring and stalling the Transactor, which fills the sequencer→transactor queue and eventually stalls submit() (§2.6). This is intentional — the ring and queues are the only place stages observe each other’s progress, and using them as the backpressure mechanism keeps the design free of explicit coordination.

§13.4 Transaction ring transport

The transaction ring is a fixed-capacity, lock-free buffer that is the sole transport for the record stream between the Transactor and the read stages ([ADR-0019]). It is single-producer, multi-reader:

• The Transactor is the only writer. It writes records into the ring’s uncommitted region (slots above the write frontier), builds each transaction in place there (§4.6), and publishes by advancing the write frontier — so readers only ever observe whole, committed transactions.
• The WAL and Snapshotter are independent copy-out readers, each tracking its own absolute cursor. They copy records out (a slot may be reused once released, so they cannot hold borrows across reclamation).
• The Snapshotter is the sole releaser (§13.5).

Positions are absolute, monotonically increasing indices mapped to physical slots on access; the capacity (ring_size, §15.10) is a power of two and must be at least the largest possible single transaction, since a transaction is published atomically and must fit in the uncommitted region. The producer never blocks inside the ring: when no free slots exist it backs off under the shared wait strategy (§13.2) and retries — backpressure is caller-driven, not hidden in the transport.

§13.5 Durability-gated reclamation

A ring slot may be reused only after the record it held is durable on disk and consumed by every reader. The Snapshotter, as sole releaser, advances the reclamation point (the release frontier) only up to what it has applied, and it applies a transaction only once its trailing TxMetadata.tx_id ≤ commit_index (§8.5). The frontiers therefore stay ordered write ≥ persisted ≥ durable ≥ applied = released. This makes “never overwrite a record that is not yet durable” a single structural property of the transport, rather than something each stage enforces separately. If durability stalls, reclamation stalls and the producer eventually blocks once the ring fills — the explicit, durability-coupled form of bounded-transport backpressure.

§14 Wait levels

§14.1 Four levels

The caller picks one of four wait levels per submission:

• none — return tx_id immediately. Maximum throughput.
• transactor — wait until compute_index ≥ tx_id (COMPUTED). The caller knows whether the transaction succeeded or was rejected.
• wal — wait until commit_index ≥ tx_id (COMMITTED). The transaction is durable on disk and survives a crash.
• snapshot — wait until snapshot_index ≥ tx_id (ON_SNAPSHOT). The effects are visible to readers via get_balance.

[ADR-0010]

§14.2 Implementation

A wait is a poll of the relevant pipeline index against the requested tx_id, driven by the shared wait strategy (§13.2). When the index passes the threshold, the wait completes and synchronises with every store that produced the advance.

§14.3 Per-call dial

Wait level is per-submission; there is no global default. The caller chooses the trade-off between latency and consistency on every call. This is the primary performance dial of the system: a high-throughput batch ingester might use none; a settlement step that must read its own balance immediately uses snapshot.

§14.4 Linearizable read recipe

The documented way to obtain a linearizable read is submit_wait(snapshot) followed by get_balance(account_id). After the wait returns, snapshot_index is past the submitted tx_id, so get_balance reflects the submitted transaction and everything before it. This is the only blocking primitive a caller needs for linearizable reads.

§15 Configuration — ledger-side knobs

§15.1 max_accounts

Pre-allocated capacity of every balance vector and read-side atomic vector. Fixed at startup; cannot grow at runtime (§4.4). Accounts referencing IDs ≥ this value are rejected with ACCOUNT_LIMIT_EXCEEDED. Sized so the balance vector fits comfortably in L2/L3 even with read-side and seal-side duplicates.

§15.2 queue_size

Capacity of the inter-stage queues. Not exposed via config.toml; an internal tuning knob. The two remaining queues — sequencer→transactor and the Snapshotter’s query queue (§2.4) — are both sized from this single value. The record-stream ring is sized separately (ring_size, §15.10).

§15.3 wait_strategy

Pipeline-mode for every stage’s idle/backpressure loop (§13.2). One value is shared across every stage of a single Pipeline.

§15.4 log_level

Not exposed via config.toml; set programmatically. The temp and bench config presets both lower this to keep test output clean.

§15.5 transaction_count_per_segment (drives multiple subsystems)

Set on StorageConfig. Controls three things at once:

• WAL segment rotation threshold (§7.6).
• Dedup active window size (§5.2).
• Hot index circle sizing (§9.6).

These are deliberately driven by one knob: changing the active window should not desynchronise any of the three.

§15.6 seal_check_internal

Interval between Seal polls (§11.2). Not exposed via config.toml. The Committer (§7.7) does not use this; it is purely the Seal thread’s poll interval and is not on the hot path.

§15.7 disable_seal

When true, the Seal thread is not started (§11.8). Not exposed via config.toml. Benchmark-only: a ledger with disable_seal = true is not crash-recoverable past the active segment.

§15.8 snapshot_frequency

Set on StorageConfig. Controls the Seal stage’s snapshot emission rule (§11.5): emit when segment_id % snapshot_frequency == 0. Smaller values mean less WAL replay on recovery (faster startup) at the cost of more snapshot I/O during operation.

§15.9 Hot-index sizes are derived

Hot-index sizes (§9.6) are derived from transaction_count_per_segment (§15.5) and max_accounts (§15.1). There are no separate knobs; changing the active window changes both the hot indexes and the dedup window in lockstep.

§15.10 ring_size

Capacity of the transaction ring (§13.4). A power of two; not exposed via config.toml, an internal tuning knob like queue_size. Because a transaction is published atomically and assembled in place, the ring must be at least as large as the biggest possible single transaction (sub_item_count ≤ u16::MAX followers plus the trailing metadata); otherwise the Transactor cannot make progress.

§16 Transaction ID space and recovery posture

§16.1 tx_id properties

tx_id is a u64. It is monotonic, globally unique per submission, strictly ordered by submission sequence, and never reused. A tx_id is a position in the global log — there are no gaps, no reorderings, and the ID itself is the source of all per-transaction identity in the WAL, the indexes, and the link records.

§16.2 Restart restores sequencer_index

On startup, after recovery (§12.3) determines last_tx_id from the WAL, sequencer_index is set to last_tx_id + 1. The next submitted operation therefore receives the lowest available unused ID, preserving monotonicity across restarts.

§16.3 In-flight IDs are not re-used

Any ID that the Sequencer assigned but the WAL did not commit before a crash is not re-used after restart. Such transactions were not durable; they are simply lost. The next submitted operation gets last_committed_tx_id + 1, leaving a (possibly empty) gap in the Sequencer’s historical assignment without producing any gap in the committed log. The committed log, by §16.4, has no gaps.

§16.4 Committed transactions are never lost

Every committed transaction is in the WAL, and the WAL is always replayed on recovery. A committed transaction therefore survives any crash — including a crash mid-fdatasync — provided the underlying storage honours its durability contract. The Snapshotter’s durability gate (§7.5, §8.5) makes this true on the read side as well: it never applies a transaction whose trailing TxMetadata.tx_id exceeds commit_index, so a transaction that was visible to a reader before the crash is by definition committed and is replayed on restart.

Stage 2 — The Storage

The previous sections describe the in-memory engine. This stage describes the on-disk shape: how records are laid out byte by byte, how files are named, how the writer and reader cooperate to keep the disk consistent across a crash, and what the ctl tooling can read and write.

§17 Storage layout and segment lifecycle

§17.1 The data directory is a flat namespace

A ledger owns one data directory. Every file belonging to that ledger lives directly in the directory; the only subdirectory is functions/ (§21.1). The names are wal.bin, wal_NNNNNN.bin, wal_NNNNNN.crc, wal_NNNNNN.seal, wal_index_NNNNNN.bin, account_index_NNNNNN.bin, snapshot_NNNNNN.bin, snapshot_NNNNNN.crc, function_snapshot_NNNNNN.bin, function_snapshot_NNNNNN.crc, term.log, vote.log, and wal.stop. The flat layout means recovery can enumerate every artifact in one syscall and partition the results purely by name parse.

§17.2 Segment file naming

The active WAL segment is always wal.bin. Sealed segments are wal_NNNNNN.bin where NNNNNN is the segment id zero-padded to six decimal digits. The pad width is a hard limit; segment ids beyond 999_999 are unsupported by the on-disk naming. At the default transaction_count_per_segment = 10_000_000 (§15.5) that bound is ~10^13 transactions per ledger, comfortably beyond any expected scale. Index, snapshot, sidecar, and seal-marker files reuse the same six-digit segment id so a single text match identifies every artifact for a given segment.

§17.3 Three lifecycle states

A segment is in exactly one of three states: Active — it is wal.bin, the writer is appending records to it; Closed — it has been renamed to wal_NNNNNN.bin and fdatasynced, but the seal marker and CRC sidecar are not yet present; Sealed — the per-segment seal procedure (§11.3) has computed the file CRC, written the .crc sidecar, and written the empty .seal marker. Cold-tier index lookups (§22) require the Sealed state; recovery (§12.2) re-seals any Closed-but-not-Sealed segment it finds.

§17.4 Rotation is rename-then-open

When the WAL Writer crosses the rotation threshold (§7.6), it performs a synchronous fdatasync on wal.bin, then renames wal.bin to wal_NNNNNN.bin and opens a fresh wal.bin for the next segment. The rename is atomic on every supported filesystem. A reader that opened wal.bin before the rename retains its handle to the now-renamed inode; a reader that opens after the rename gets the new file. This is the foundation of the inode-based rotation detection used by replication (§23.3).

§17.5 The wal.stop clean-shutdown marker

A clean Ledger::shutdown writes an empty wal.stop file before exiting. On startup, recovery uses the marker to decide what to trust: both wal.stop and wal.bin present means the previous shutdown was clean and the active segment is fully trustworthy; only wal.bin present means the process exited without flushing and the active segment must be tail-trimmed (§23.4). The marker is removed at the start of every Ledger::start so a crashed restart cannot mistake a stale marker for clean state.

§17.6 Inode rotation detection

Both the leader-side replication tailer (§23.3) and any external reader of the active segment detect rotation by inode comparison: at open time they stash the inode of wal.bin; on every read after EOF they compare the current on-disk inode with the stashed one. A mismatch means the active segment was renamed and the reader must advance to the next segment. The pattern works because the rename in §17.4 produces a new inode for the new wal.bin; the old inode lives on as long as anyone holds a handle to it. [ADR-0006, ADR-0013]

§18 WAL record byte layout

§18.1 The 40-byte invariant

Every WAL record is exactly 40 bytes. This is the same invariant stated in §1.6 from the in-memory side; this section states the byte-level consequences. The first byte is the record’s entry_type discriminant; the remaining 39 bytes are the record’s payload. All multi-byte fields are little-endian. Records are dense in the file: there is no inter-record padding, no length prefix, no framing layer. A reader can scan a segment by stepping 40 bytes at a time and dispatching on byte 0.

§18.2 Zero-copy serialization

The in-memory layout matches the on-disk layout exactly. Records are defined as #[repr(C)] plain-old-data structs with explicit _pad fields filling unused bytes; the bytes that go to disk are obtained by casting the struct reference to a byte slice via bytemuck::bytes_of. There is no per-field encode step, no allocator pressure, no conversion between in-memory and on-disk representation. The same pattern is used on the read side: a 40-byte slice is cast back to the appropriate struct once the discriminant has been examined.

§18.3 CRC envelope and torn-write detection

A transaction’s CRC lives in its trailing TxMetadata.crc32c (§18.1) and covers the followers (TxEntry / TxLink) in stream order followed by the metadata record with that field zeroed for the digest. The CRC field sitting inside its own coverage forces the zero-while-computing convention. Because the metadata is the commit record, a transaction is complete only when its trailing metadata is present and its CRC matches; a torn write inside a transaction is caught when the metadata is missing or its CRC fails — there is no per-record CRC, and none is needed. FunctionRegistered carries no tx_id and is outside any transaction’s CRC envelope; the crc32c == 0 value is the unregister signal (§6.17). [ADR-0001, ADR-0014, ADR-0020]

§19 Sidecar files

§19.1 The CRC sidecar

Every Sealed segment has a paired wal_NNNNNN.crc file carrying the segment’s full-file CRC, the segment size at seal time, and a magic constant. The sidecar is written by the per-segment seal procedure (§11.3, §25.5) after the segment file itself has been closed. The size field catches truncation that does not change the CRC of the surviving prefix; the magic disambiguates a .crc against any unrelated file of the same width that might end up in the directory.

§19.2 The .seal marker

Alongside the .crc sidecar, Seal writes an empty wal_NNNNNN.seal file. Presence of the marker is what distinguishes a Sealed segment from a Closed-but-not-yet-sealed one: a crash between the WAL rename and the seal procedure leaves a Closed segment that recovery (§12.2) re-seals on the next startup. The marker file carries no content; only its existence matters. An empty marker file is cheaper than a flag inside the sidecar because its presence can be detected by a stat call alone.

§19.3 The wal.stop clean-shutdown marker

wal.stop is the symmetric marker for the active segment: present iff the last Ledger::shutdown call completed cleanly (§17.5). It is the one file in the directory whose absence is informative. Recovery uses its absence to decide whether the active segment needs tail-trimming (§23.4) versus full-trust load.

§19.4 Missing sidecars are tolerated

A Sealed segment that is missing its .crc sidecar is loaded with a warning, not a failure. The rationale: the sidecar is a redundant check; the segment’s own per-transaction CRCs (§18.3) are still authoritative. Missing sidecars are most commonly produced by a crash after the segment was renamed but before Seal ran, or by external tools that copied a segment without copying its sidecar. Neither case warrants a refusal to start. [ADR-0006]

§20 Snapshot files

§20.1 Naming and pairing

Every snapshot is a pair: snapshot_NNNNNN.bin plus snapshot_NNNNNN.crc. The function snapshot equivalent uses function_snapshot_NNNNNN.bin plus function_snapshot_NNNNNN.crc. The six-digit suffix is the segment id of the segment whose sealing triggered the snapshot (§11.5); the snapshot reflects the state through the end of that segment. A pair is the unit of trust: a snapshot whose .crc is missing is not loaded, even if the .bin parses cleanly.

§20.2 Double-CRC discipline

A balance snapshot file carries two independent CRCs: an inline data_crc32c in the header validates the records independent of the compression layer, and the sidecar CRC (§20.5) validates the written file as a whole. The first catches corruption introduced by a buggy decompressor; the second catches truncation or bit-rot after the rename.

§20.3 Body sort and non-zero filter

The body is sorted by account_id so the same set of balances always produces the same bytes — reproducible across restarts and amenable to offline diffing. Only non-zero balances are emitted; account ids that have never been touched are omitted.

§20.4 Snapshot atomic write

Snapshots are written atomically with the .tmp rename pattern: write the body to snapshot_NNNNNN.bin.tmp, fsync it, write the .crc to snapshot_NNNNNN.crc.tmp, fsync it, then rename both to their final names. If the process crashes between any two steps the partial work is identifiable by the .tmp suffix and is discarded on the next startup. The two renames are independent — a snapshot is not considered loaded until both finals exist (§20.1) — so the order of the two renames does not matter for correctness.

§20.5 The snapshot CRC sidecar

snapshot_NNNNNN.crc is the same 16-byte format as the WAL CRC sidecar (§19.1) but with a different magic (SNAPSHOT_MAGIC = 0x534E4150). The fields cover the entire snapshot_NNNNNN.bin file as written, so a truncation or bit-flip after the rename is detected on the next load.

§20.6 Function snapshot

function_snapshot_NNNNNN.bin mirrors the balance snapshot’s shape: a header (FUNCTION_SNAPSHOT_MAGIC = 0x46554E43 — "FUNC"), an LZ4-compressed body, a paired .crc sidecar. The body is a sorted-by-name sequence of (name, version, crc32c) triples — the registry’s complete state at the time of the snapshot, including names whose latest record is an unregister (crc32c == 0). Recording unregisters in the snapshot is what lets recovery jump straight to the latest snapshot boundary instead of replaying from segment 1 (§12.3).

§20.7 Empty function snapshots are still written

Even when the registry is empty, the function snapshot is emitted at every snapshot boundary. The reason is symmetry with the balance snapshot: recovery always loads the paired snapshot for a given segment id, and the absence of a function snapshot would otherwise require a special “function snapshot may legitimately be missing” code path. An empty file is cheaper than a special case. [ADR-0014]

§21 Function binary storage

§21.1 The functions/ subdirectory

WASM binaries live in {data_dir}/functions/. This is the only subdirectory of the data directory; it is created on first registration. Each binary is a file named {name}_v{N}.wasm where name is the ASCII snake_case identifier from the FunctionRegistered record and N is the per-name monotonic version starting at 1. The directory is the secondary signal of the registry’s history; the WAL is authoritative.

§21.2 Atomic write on registration

RegisterFunction writes the binary to {name}_v{N}.wasm.tmp, fsyncs, then renames to the final {name}_v{N}.wasm. The rename is atomic on every supported filesystem; readers therefore never observe a partial file. Only after the rename is committed does the registration emit its FunctionRegistered WAL record (§6.16); the disk-then-WAL order means a crash at any point leaves either no record of the registration, or both the file and the WAL record in their final states.

§21.3 Unregister produces a 0-byte file

Unregistration writes a 0-byte {name}_v{N}.wasm (where N is the next version after the last registered one) and emits a FunctionRegistered record with crc32c == 0 (§6.17). The 0-byte file is the on-disk audit trail; the WAL record is the authoritative signal. Past versions of the binary are preserved; the on-disk history of a function is therefore the directory listing of {name}_v*.wasm files in version order. [ADR-0014]

§22 On-disk indexes

§22.1 Per-segment, written at seal

Each Sealed segment has a paired transaction index (wal_index_NNNNNN.bin) and account index (account_index_NNNNNN.bin), both built once at seal time (§11.3) and never modified afterwards. The hot tier (§9) covers the active window; the on-disk indexes cover everything older. A query that misses in the hot tier (§10.4) consults the on-disk indexes for the segment that contains the requested tx_id or account history. The boundary between hot and cold is not bytes-on-disk — it is eviction in the hot tier.

§22.2 Query path

A cold-tier GetTransaction binary-searches the per-segment transaction index on tx_id and issues a positional read into the segment file to fetch the metadata plus its dependent entries and links. A cold-tier GetAccountHistory partition-points the per-segment account index on account_id, then iterates the tx_id range forward until the lower bound or limit is hit. Iteration order is ascending in tx_id; the caller reverses for newest-first.

§22.4 Why flat sorted arrays

The format has no per-record framing, no inline pointers, no tree metadata. The cost of binary search is O(log N) with predictable cache behaviour; the cost of construction is one sort_unstable over the segment’s records. The trade-off compared to a B-tree or hash index is that updates require a full rewrite — but indexes are written exactly once per segment and never updated, so the trade-off favours the simpler format. [ADR-0008]

§23 WAL-tail reader and runtime truncation

§23.1 WalTailer is a stateful cursor

The leader-side replication path (§31) reads raw WAL bytes via a stateful cursor that holds an open file handle and a byte position across calls. Reads use positional I/O (pread), so concurrent reads do not interfere with the writer’s append position. The cursor amortises the cost of opening the file: once seeded, every subsequent call pays only for the new bytes since the previous call’s position.

§23.2 Whole-record reads only

The tailer always returns a multiple of 40 bytes (§18.1). When the underlying file ends mid-record — a torn write or a write still in progress — the tailer rounds the byte count down and stops at the last full record boundary. The partial bytes are not lost; the next call resumes from the same position and re-reads them along with whatever new bytes have arrived. The replication path therefore never sees a partial record on the wire.

§23.3 Inode-based rotation detection

When the tailer opens wal.bin, it stashes the inode of the file. On EOF, it compares the stashed inode against the inode currently bound to the path: a mismatch means rotation happened (§17.4) and the tailer advances to the next segment file (wal_NNNNNN.bin if the rotation has fully completed, the new wal.bin once the next segment starts taking writes). The detection is cheap — one stat call per EOF — and avoids any coordination with the writer.

§23.4 Tail-trim on cold-boot recovery

On startup, recovery (§12) inspects the active segment for a torn tail. The procedure: forward-scan the segment, buffering each transaction’s followers until a TxMetadata closes and validates the group (buffered follower count equals sub_item_count, CRC matches). The valid region must end exactly on a metadata boundary; if the scan ends with buffered followers and no closing metadata, or on a torn sub-40-byte record, the file is truncated back to the last metadata boundary. This is the one tolerated case of corruption (§12.7); a torn middle is fatal. The trim is performed via set_len followed by an fsync_data, so the post-recovery file matches what is on disk. [ADR-0020]

§23.5 Runtime truncation classifies segments by watermark

Divergence-driven runtime truncation (§33) classifies each segment by its range of tx_ids relative to the recovery watermark. A segment whose last tx_id is at or below the watermark is kept untouched. A segment whose first tx_id is strictly above the watermark is removed entirely — wal_NNNNNN.bin, the .crc and .seal sidecars, the transaction and account indexes, the balance snapshot and its sidecar, the function snapshot and its sidecar — every artifact for that segment id. A segment that straddles the watermark is byte-truncated immediately after the trailing TxMetadata of the highest transaction whose tx_id ≤ watermark — so the followers of the first over-watermark transaction are dropped along with its metadata, leaving the surviving prefix ending exactly on a metadata boundary. The surviving prefix is treated as Closed-but-not-yet-Sealed and resealed on the next start.

§23.6 Sealed segments are immutable

A Sealed segment is never modified in place. The straddle case (§23.5) can only land in the active segment or in a segment that was Closed but not yet Sealed at the moment recovery began. The watermark is gated at the storage layer to prevent landing inside a Sealed segment; the higher layer (§33) is responsible for choosing a watermark that respects this. A request to truncate a Sealed segment is a programming error and panics rather than corrupt the on-disk audit trail. [ADR-0006, ADR-0016 §9, §10]

§24 Term and vote logs

§24.1 Two files, one purpose each

Cluster Raft state is persisted as two append-only files in the data directory: term.log and vote.log. They are storage-layer artifacts because their durability rules and on-disk format are storage’s responsibility, even though their semantic meaning belongs to the cluster (§28). Keeping them as separate files means the hot path that queries the current term (§36.4) does not pay the cost of pollution from a vote-grant write.

§24.2 Records share the WAL’s 40-byte width

Term and vote records are exactly 40 bytes — the same width as a WAL record (§18.1) — so the same offline tooling can inspect them. Each file has its own magic constant ("TERM" / "VOTE") so a misrouted record is rejected on read.

§24.3 Append, then fdatasync, then publish

The writer takes a per-file mutex, appends the record, calls sync_data, and only then publishes the corresponding atomic value (current_term, voted_for). Readers observe the atomic with Acquire ordering. The discipline enforces the same “atomic value observed ⇒ record on disk” rule that the WAL stage enforces for its own commit index (§7.5). A crash between append and publish is harmless: the next restart’s scan re-derives the atomic from the last record on disk.

§24.4 Crash-tolerant scan

Both files are read by walking from the head and parsing one 40-byte record at a time. The reader stops cleanly on EOF or on the first record whose magic mismatches or whose CRC fails — it does not throw. A torn write to either file is therefore tolerated: the surviving prefix is the authoritative state. The append path’s sync_data ordering means a record observed by the scanner was on disk before the writer published its atomic value; no committed term or vote is ever lost to a crash.

§24.5 No truncation, no rotation

The two logs grow without bound. In practice the term log adds one record per election and the vote log adds one record per granted vote; both are negligible in steady state. There is no rotation, no compaction, no truncation. A future bound (e.g. truncating the term log to records covering only the active window) is not part of the current design.

§25 ctl tools

§25.1 The roda_ctl binary

roda_ctl is the offline tooling for inspecting and reconstructing WAL segments. It is not a control plane: it does not connect to a running ledger, does not synchronise with the running writer, and does not modify any in-memory state. It operates purely on files in the data directory and assumes no other process holds a write handle to the artifacts it touches. Four subcommands exist: unpack, pack, verify, and seal.

§25.2 unpack

roda_ctl unpack SEGMENT [--out FILE] [--ignore_crc] reads a WAL segment binary and emits one JSON object per record to standard output (or --out FILE). The JSON shape is one-to-one with the record’s fields; CRC errors are surfaced as a "crc_error": true field on the offending record rather than aborting the dump, unless --ignore_crc is used to skip the check entirely. unpack is the diagnostic tool of record: any question about what is in a segment is answered by piping its output to jq.

§25.3 pack

roda_ctl pack INPUT [--out FILE] [--no_validate] is the inverse of unpack: it parses NDJSON from INPUT (or stdin) and writes a binary segment to --out (or stdout). It re-encodes each record using the zero-copy serialization (§18.2) and recomputes each transaction’s CRC32C over its followers (in order) then the zeroed-crc trailing TxMetadata (§18.3). By default it validates that tx_id values are monotonically increasing. --no_validate disables the check for cases where a deliberately partial or malformed segment is needed (typically for testing recovery).

§25.4 verify

roda_ctl verify DIR [--segment ID | --range FROM..TO] runs an integrity audit over a data directory or a subset of its segments. Per segment it checks: the file-level CRC against the .crc sidecar; the per-transaction CRC; the cumulative computed_balance chain (no unexplained jumps); the declared sub_item_count matches the actual follower count. Across segments it checks that sealed segments form a contiguous tx_id chain with no gaps and no overlaps. The output is a structured report; a non-zero exit code signals at least one failure.

§25.5 seal

roda_ctl seal SEGMENT [--force] finalises a Closed segment to Sealed state by computing the file-level CRC32C, writing the wal_NNNNNN.crc sidecar, and writing the empty wal_NNNNNN.seal marker. It refuses to operate on Active or already-Sealed segments unless --force is given. The operation is idempotent under --force: the sidecar and marker are simply rewritten. seal exists primarily for recovery scenarios where the running ledger crashed after a rotation but before its Seal thread (§11) had a chance to finish. [ADR-0007]

Stage 3 — The Cluster

The previous stages describe a single-node ledger and its on-disk artifacts. This stage describes the network layer that wraps the ledger to provide replication and high availability. The cluster module owns both gRPC services — the client-facing Ledger service and the peer-facing Node service — and the Raft state machine that drives leader election and log replication. The Ledger itself remains unchanged except for a small, narrow contract surface (§26.2).

§26 The cluster layer

§26.1 Single node is a cluster with zero peers

There is one binary and one configuration shape for both single-node and multi-node deployments. A “single node” is a cluster whose peer list contains only itself; a multi-node cluster has additional peers. There is no separate single-node binary, no feature flag for gRPC, and no distinct configuration path. The simplification is deliberate: every observable behaviour is exercised by the same code path that runs in production, and operational tooling does not need to branch by mode. [ADR-0015 decisions 1, 2]

§26.2 The minimal Ledger contract

The cluster module never reaches into the Ledger’s internals. It uses exactly four entry points beyond the standard query and submit surface:

• append_wal_entries(Vec<WalEntry>) — the follower-side write path that bypasses the Transactor and feeds pre-validated bytes directly to the WAL stage.
• wal_tailer() — produces a stateful raw-WAL byte cursor (§23.1) used by the leader to ship bytes to followers.
• An on_commit hook fired when the local commit index advances, used by the leader to feed its own slot of the quorum tracker (§30.4).
• start_with_recovery_until(watermark) — an alternative to start() invoked only by the runtime divergence path (§33).

Nothing else changes inside the Ledger. A standalone Ledger ignores all four. [ADR-0015, ADR-0016 §10]

§26.3 Two gRPC surfaces

The cluster runs two distinct gRPC services on two distinct ports. The Ledger service (proto/ledger.proto) is the client-facing API: submit, query, status, wait. The Node service (proto/node.proto) is the peer-facing API: append entries, request vote, ping, install snapshot. The two services are entirely separate; a client never speaks the Node service and a peer never speaks the Ledger service. Both servers are spawned by the supervisor and survive every role transition (§38.2) — only their posture (writable vs read-only) changes when the node’s role transitions.

§26.4 Naming follows role, not transport

Within the cluster module, types are named by their role (Leader, Follower, LedgerHandler, NodeHandler, Server, NodeServerRuntime) rather than by their transport. gRPC is an implementation detail; the same role surface could in principle be served by another transport without renaming the role types. [ADR-0015 decision 6]

§27 Roles and the runtime state machine

§27.1 Four roles encoded in one atomic

A node is in exactly one of four roles at any time: Initializing (post-boot or post-teardown, awaiting a signal), Candidate (running an election round), Leader (writable), or Follower (read-only, receiving AppendEntries). The role is encoded as a u8 inside a single atomic; reads use Acquire, writes use Release. The atomic is shared by every gRPC handler so the writability check on every submit RPC is one cache-line read.

§27.2 The supervisor is the sole writer

The role supervisor is the only entity that mutates the role atomic. Handlers, replication tasks, and ledger threads all read the atomic freely but never write it. The single-writer rule means a transitioning role cannot leave handlers observing a half-applied state: the supervisor publishes the new role only after the old role’s tasks have been torn down and the new role’s tasks have been brought up.

§27.3 Drop-and-rebuild on every transition

Role-specific state — peer replication tasks, the Quorum tracker, the gRPC handler’s posture — lives inside a LeaderHandles or FollowerHandles struct. A transition drops the outgoing struct (which joins or aborts every owned task) before constructing the new one. There is no “modify role in place” path; a node that goes Leader → Follower → Leader has fully re-instantiated its peer tasks, fresh quorum slots, and a freshly-bound writable handler each time. The discipline is what eliminates stale-state bugs during failover.

§27.4 What survives a transition

Three pieces of state are deliberately carried across a role boundary: the Arc<Ledger> (so the node’s data does not have to be recovered on every transition), the Arc<Term> (so the durable term log keeps a single owner), and the Arc<Vote> (same reason for vote durability). Every other piece of state is owned by a Handles struct and dies with it. The one exception to even the Ledger surviving a transition is divergence (§33), in which case the Arc is dropped and rebuilt via start_with_recovery_until.

§27.5 Initializing is the post-boot and post-teardown state

Initializing is the role a node is in immediately after the supervisor starts (before the first election timer fires) and immediately after any Handles is dropped (before the next role’s bring-up). The Node gRPC server is already running so the node can participate in elections and receive AppendEntries, but neither the writable client handler nor any peer-replication task is up. The client-facing handler in this state behaves like a follower’s read-only handler: queries succeed; submits return FAILED_PRECONDITION. [ADR-0016 §2, §3]

§28 Term and vote durability

§28.1 term.log and vote.log separate the hot from the cold

The two durable Raft state files (§24) are kept distinct precisely because their access patterns are different. term.log is on the GetStatus hot path: every status query reads the current term. It grows by one record per term boundary, which is rare in steady state. vote.log is purely cold: it is written by the RequestVote handler when a vote is granted, and it is read only at boot to seed the in-memory (current_term, voted_for) pair. Mixing them in one log would force the hot path to scan past vote records.

§28.2 Append-then-publish ordering

A term bump goes: take the term mutex, append the TermRecord, sync_data, publish the new term to the in-memory atomic with Release, drop the mutex. A vote grant goes: take the vote mutex, append the VoteRecord, sync_data, publish voted_for with Release, drop the mutex. A reader that observes either atomic via Acquire is therefore guaranteed the on-disk record exists. This is the same discipline used for the WAL’s own commit index (§7.5) and is what makes “committed term” meaningful across a crash.

§28.3 Monotonic term

Both Term::observe(term, start_tx_id) and Vote::observe_term(term) reject regressions: a request with a term smaller than the current term is a no-op. Term values are monotonic across the entire cluster’s lifetime; observe is the path by which a follower or candidate adopts a higher term seen on an incoming RPC. A peer that receives a higher term clears its voted_for (since voted_for is per-term) before publishing the new term.

§28.4 The term hot ring

In-memory, Term keeps a bounded ring (capacity 10_000) of recent records to serve GetStatus-style queries without disk I/O. A query for a tx_id whose covering term is in the ring returns directly; a miss falls through to a mutex-locked disk scan. The ring is sized so that under normal failover frequencies every term boundary in the recent past is in memory; clusters with extreme term churn fall back to the cold path more often but still see correct answers. [ADR-0016 §4, §11]

§29 Election protocol

§29.1 Randomised election timer

Every node not in Leader role runs an election timer with a randomised deadline in [election_timer_min_ms, election_timer_max_ms]. The deadline is re-randomised on every reset. Reset triggers: any valid AppendEntries from a current leader (empty heartbeat included), and the granting of a RequestVote. Without a reset, the timer fires and the node transitions to Candidate. The randomisation defeats split votes: two nodes that timed out at exactly the same instant would collide otherwise.

§29.2 Candidate round

A candidate round consists of: durably bumping the term via Term::new_term(last_commit_id), durably self-voting via Vote::vote(new_term, self_id), fanning out RequestVote to every other peer in parallel, and tallying responses. The outcome is one of three values: Won — at least majority granted, transition to Leader; HigherTermSeen — any peer reported a term higher than ours, transition back to Initializing after observing the new term; Lost — the deadline expired with neither outcome, transition back to Initializing and re-arm the timer with a new randomised deadline.

§29.3 Majority is (node_count / 2) + 1

Majority counts the candidate itself: a 5-node cluster needs 3 votes (of which 1 is the self-vote, so 2 peer grants suffice). A single-node cluster trivially wins by self-vote with no RPCs sent. The arithmetic matches Quorum’s majority calculation (§30.1) but is computed independently — the election’s vote count is not the running quorum.

§29.4 RequestVote grant rule

Implements Raft §5.4.1 verbatim — grant iff the request’s term is at least the current term, voted_for is unset or already this candidate within the term, and the candidate’s (last_term, last_tx_id) is lexicographically at least ours. The log-up-to-date clause is what enforces leader completeness: a candidate whose log lacks a quorum-committed transaction cannot win. A granted vote fsyncs vote.log before the RPC reply is sent (§28.2).

§29.5 Pre-vote round

A candidate first runs a pre-vote round before bumping its term: it asks peers whether they would grant a vote under the third clause of §29.4 (log up-to-date check) without persisting voted_for or bumping its own term. Only if pre-vote succeeds does the candidate proceed to the real round. Pre-vote prevents a partitioned node from forcing the rest of the cluster to step down on every reconnect — without it, a node coming back online would unconditionally bump its term high enough to depose the current leader.

§29.6 Step-down on higher term

Any RPC in any direction (request or response, RequestVote or AppendEntries or even a Ping) that carries a term higher than the node’s current term triggers an immediate step-down: Term::observe durably records the new term, voted_for is cleared, and the node transitions to Initializing. A node in Leader role drops its LeaderHandles (which drains all peer tasks); a node in Follower or Candidate drops the equivalent. The step-down is the universal correctness anchor — no Raft node ever continues operating in an old term once it has seen a higher one. [ADR-0016 §5, §6, §13]

§30 Quorum tracker

§30.1 One slot per node

Quorum is an array of AtomicU64s, one slot per node in the cluster. By convention slot 0 is the leader itself; slots 1.. are the peers in configuration order. The majority size — (node_count / 2) + 1 — is computed at construction and never changes. A separate atomic caches the current majority-committed index for lock-free reads.

§30.2 advance publishes via fetch_max

advance(slot, index) writes the slot’s atomic with Release, snapshots all slots with Relaxed, sorts the snapshot descending, takes the (majority - 1)-th element (the highest index acknowledged by a majority), and publishes it via fetch_max(Release) on the cached majority atomic. The fetch_max ensures the published majority never regresses, even when concurrent advance calls from different slots interleave. This is the lock-free property that lets every peer task call advance without coordination.

§30.3 Lock-free reads via get

Quorum::get() is one Acquire load of the cached majority atomic. There is no allocation, no fast-path lock, no per-call computation. Client wait paths (§34.1) call get() repeatedly while polling for a target index; the cost per poll is one cache-line load.

§30.4 The leader’s slot is fed by the on-commit hook

On Leader bring-up, the leader registers an on_commit callback with the Ledger. The callback fires every time the local commit index advances and calls advance(self_slot, ledger.last_commit_id()). Without this callback the leader’s own progress would be invisible to the quorum calculation and the cached majority would be stuck at the slowest peer. The hook is the reason the leader counts toward its own quorum.

§30.5 reset_peers zeros peer slots on bring-up

When a leader is freshly elected, the prior leader’s match-index state may still be reflected in the slots; carrying it forward could let a stale peer slot inflate the new leader’s perceived majority. reset_peers(leader_slot) is called as part of Leader bring-up and zeros every slot except the leader’s own. The next advance call from each peer task republishes from a clean baseline. [ADR-0015 decisions 3, 5; ADR-0016 §3]

§31 Leader replication tasks

§31.1 One task per peer

A Leader spawns one tokio task per peer (PeerReplication). The leader owns these tasks directly — there is no per-peer-supervisor indirection — and they all share the leader’s Arc<Quorum>, Arc<AtomicBool> running shutdown flag, and a transition channel for out-of-band signals. Spawning is part of Leader::run_role_tasks; joining is part of LeaderHandles::drop.

§31.2 Each task’s private state

Each peer task owns: a WalTailer cursor seeded from the leader’s ledger (§23.1); a tonic Node-service client connected to the peer; the peer’s slot index in Quorum; and two watermarks — from_tx_id (Raft’s nextIndex, the next tx_id to ship) and peer_last_tx (Raft’s matchIndex snapshot, the highest tx_id the peer acknowledged). Nothing else is shared between peer tasks.

§31.3 The replication loop

Each iteration: call tailer.tail(from_tx_id, &mut buf); if it returns zero bytes, send an empty AppendEntries heartbeat carrying only leader_commit_tx_id and sleep for replication_poll_ms; if it returns bytes, call the ship-until-accepted path which retries on transport failures and observes higher-term replies. On a successful non-empty reply the task advances from_tx_id past the shipped batch’s last tx_id, updates peer_last_tx, and calls quorum.advance(slot, peer_last_tx).

§31.4 Lagged single-phase replication

A successful AppendEntries returns the follower’s current fsynced last_commit_id — not a fresh fsync of the shipped batch. The follower queues the bytes and replies immediately; its WAL stage fsyncs on its own schedule (§7.7) and the next RPC’s reply reflects the result. Replies are therefore one batch stale relative to the shipped data, which is harmless for quorum tracking because the gap closes on the next reply. The model avoids blocking the network RPC on disk latency. [ADR-0015]

§31.5 Idle heartbeats close the staleness gap

Without idle heartbeats, the leader’s last knowledge of a peer’s commit progress would be the reply to the previous shipment — itself one batch stale. After the writer goes idle, the peer’s true commit watermark would never be observed. Sending an empty AppendEntries every replication_poll_ms fixes this: the reply carries the peer’s now-fsynced last_commit_id, which advances the peer’s quorum slot. Transport errors on heartbeats are intentionally swallowed — the next interval retries; a heartbeat is purely observational, not a durability event.

§31.6 Step-down on higher-term reply

A peer reply whose term is higher than the leader’s current term triggers an immediate step-down: the peer task posts a step-down transition to the supervisor and returns. The supervisor drains all peer tasks (drops LeaderHandles), observes the higher term, clears voted_for, and re-enters Initializing. The write side is also drained: any in-flight client submit returns the error generated by the dropping handler. [ADR-0015, ADR-0016 §3]

§32 Per-peer replication and consistency

§32.1 Raft naming map

The Roda cluster module uses Raft’s terminology where possible: from_tx_id is nextIndex; peer_last_tx and the Quorum peer slot together correspond to matchIndex. The tx_id stream is the Raft log; the term boundary records in term.log (§28) supply the term for any historical tx_id. This mapping is documented so a reader who knows Raft can navigate the implementation without re-deriving the correspondence.

§32.2 prev_tx_id and prev_term consistency check

Every AppendEntries request carries prev_tx_id (the tx_id immediately before the batch) and prev_term (the term of that tx_id). The follower validates: it has prev_tx_id durably on disk, and the term covering prev_tx_id matches prev_term. If the follower’s log is shorter, the reply carries REJECT_PREV_MISMATCH with the follower’s own last_commit_id. If the follower’s log has the tx_id but with a different term, the reply also carries REJECT_PREV_MISMATCH and divergence handling kicks in (§33).

§32.3 The first RPC

For the first AppendEntries against an empty follower (or a fully-replicated follower starting a fresh batch from tx_id = 0), both prev_tx_id and prev_term are zero. The follower treats (0, 0) as “no precondition” and accepts. Subsequent RPCs use the actual preceding tx_id and its covering term.

§32.4 Reject reasons are explicit

AppendEntriesResponse carries an enum reject reason rather than a bare bool. The four currently emitted reasons are: TERM_STALE (request’s term is below the follower’s current term), PREV_MISMATCH (consistency check failed — see §33), CRC_FAILED (the bytes did not parse), WAL_APPEND_FAILED (the follower’s WAL stage rejected the queued bytes). The leader uses the reject reason to decide whether to step down (TERM_STALE), trigger a divergence path on the follower side (PREV_MISMATCH), or log and continue.

§32.5 Batch sizing is exact

WAL records are exactly 40 bytes (§18.1), so the leader’s per-RPC byte cap is always a clean multiple of 40 — there is no padding, no partial record at the end of a batch. The configured append_entries_max_bytes is rounded down to the nearest multiple of 40 at startup.

§32.6 gRPC message-size override

Tonic’s default decoding/encoding limit of 4 MiB would otherwise reject batches sized at exactly append_entries_max_bytes. The Node server and client both raise max_decoding_message_size and max_encoding_message_size to append_entries_max_bytes × 2 + 4 KiB to cover both the WAL byte payload and protobuf framing overhead with margin. [ADR-0015, ADR-0016 §8]

§33 Log divergence and runtime ledger reseed

§33.1 The live WAL is append-only while running

Truncation of the WAL is permitted, but only via a Ledger restart. The running Ledger never modifies records that have already been written. This rule preserves the in-memory pipeline’s invariants — the Snapshotter (§8) and the indexes (§9) assume a monotonically growing WAL and would not survive in-place truncation. The runtime divergence path (§33.3) achieves truncation by restarting the Ledger, which lets the Recover path do the truncation safely against a quiescent disk.

§33.2 Divergence detection on the follower

When AppendEntries arrives with a prev_tx_id that the follower has on disk but with a prev_term that disagrees with the follower’s own term covering that tx_id, the follower has diverged from the new leader’s history. The follower stashes leader_commit_tx_id from the request as the recovery watermark and replies REJECT_PREV_MISMATCH. The supervisor’s divergence watcher reads the stashed watermark and triggers the reseed.

§33.3 Reseed sequence

The supervisor: drops the current FollowerHandles (so all tasks holding the Arc<Ledger> are gone); calls a LedgerSlot::replace with a freshly-built Ledger::start_with_recovery_until(watermark), which gives the slot a new Arc<Ledger> and returns the old one for asynchronous drop on a background task; brings up a fresh FollowerHandles over the new Ledger. The next AppendEntries from the leader either resumes cleanly (prev_tx_id ≤ watermark) or triggers a normal catch-up backfill from the leader.

§33.4 start_with_recovery_until semantics

start_with_recovery_until(watermark) is the alternative entry point to Ledger::start. It walks the WAL and replays snapshots up to and including watermark, physically truncates any WAL content above the watermark (§23.5), and skips any snapshot whose covered-up-to-tx exceeds the watermark — recovery falls back to an earlier snapshot or to genesis. Cold boot (first start of a process) always uses plain start(); start_with_recovery_until is strictly a runtime response to divergence and is never invoked at boot.

§33.5 Balances rewind for free

Because balances are a derived state of the WAL (§1.2), rewinding the WAL rewinds balances as a side-effect of Recover re-walking the truncated log. There is no undo log, no in-place rollback of applied entries, no staged balance layer. The simplification is what makes divergence recovery tractable: the Ledger’s recovery path already knows how to reconstruct balances from a WAL prefix.

§33.6 LedgerSlot is the indirection that lets handlers survive

A reseed swaps the Ledger out from under the gRPC handlers without restarting the gRPC servers. The mechanism is LedgerSlot: an atomic swap of Arc<Ledger>. Every handler reads the current Arc on every call (slot.ledger()) and never retains it across operations. The swap is lock-free; the old Arc’s strong count drops to zero on a background task that joins the old Ledger’s pipeline threads synchronously, which never blocks a gRPC request because gRPC handlers have already released their reference. The supervisor is the sole writer of the slot. [ADR-0016 §9, §10]

§34 Cluster commit and the cluster_commit wait level

§34.1 A fifth wait level

Stage 1 introduces four wait levels (§14.1). The cluster adds a fifth: cluster_commit blocks the submitter until quorum.get() ≥ tx_id. The handler driving the wait is the same shared wait strategy (§13.2) used for the in-process levels; the only difference is which atomic the predicate reads.

§34.2 The leader feeds its own slot

The leader’s slot in Quorum is advanced by the on-commit hook (§30.4); each peer task advances its peer’s slot on every successful AppendEntries. Both contributions are required for the majority to move forward — without the leader’s own contribution the cached majority would be stuck at the slowest peer’s progress.

§34.3 Local commit is independent of quorum

The leader’s own commit_index (§2.5) advances from its local WAL stage as soon as the local fdatasync returns, independent of any peer’s progress. Only the client’s wait choice determines the guarantee: a client that picks wal sees the local commit; a client that picks cluster_commit waits for quorum. The two indexes coexist in the same pipeline; nothing about cluster mode slows down the local commit path. [ADR-0015 decision 5]

§34.4 Followers do not durably track cluster commit

leader_commit_tx_id arrives on every AppendEntries (including heartbeats) but the follower does not persist it. Its only consumer on the follower side is divergence handling (§33.2), where it is captured as the recovery watermark. A follower’s view of “cluster commit” is not authoritative — only the leader’s Quorum::get() is. Reads from a follower that need cluster-commit semantics are out of scope for this implementation.

§35 gRPC: Node service

§35.1 Service-level invariant

The Node service (proto/node.proto) is the peer-facing API. Every handler in the service except Ping shares a single per-node tokio::sync::Mutex so that updates to (current_term, voted_for, log) appear atomic to peers. Tonic dispatches RPCs in parallel by default; without the mutex, two concurrent RequestVote handlers could race on vote.log and grant the same term to two candidates. [ADR-0016 §7]

§35.2 AppendEntries

An empty wal_bytes is a valid heartbeat: the follower runs the consistency check and returns its commit id without queueing any records. The reply’s last_tx_id carries the follower’s current fsynced commit id, which lags the shipped batch by one round (§31.4).

§35.3 RequestVote

Implements the grant rule of §29.4 verbatim and persists the vote via Vote::vote before replying. A grant that requires a term observation also durably bumps the term via Vote::observe_term.

§35.4 Ping

Reads only atomics — no WAL, no vote log — and bypasses the per-node mutex. Its purpose is operational: health checks, RTT measurement, and leader discovery without the cost of the full Raft handler stack.

§35.5 InstallSnapshot

The wire format mirrors Raft’s InstallSnapshot and exists in the proto to lock in the shape. The handler is scaffolded but not fully implemented; followers that fall too far behind a leader’s retention window will need this RPC to catch up via a snapshot rather than via WAL backfill. Currently returns UNIMPLEMENTED.

§36 gRPC: Ledger service

§36.1 Submit family

SubmitOperation, SubmitAndWait, SubmitBatch, SubmitBatchAndWait. All four are gated on role.is_leader(); a non-leader returns FAILED_PRECONDITION with the leader’s id (the node’s best-known leader hint) carried in response metadata. Clients hitting a non-leader can therefore redirect without a separate lookup RPC. The wait variants accept any of the five wait levels (§14, §34) and block until the threshold is crossed.

§36.2 Query family

GetTransaction, GetTransactionStatus, GetTransactionStatuses, GetAccountHistory, GetBalance, GetBalances. GetBalance and GetBalances are direct atomic reads against the read-side balance cache and never touch the snapshot query queue (§10.6); the rest enqueue queries on the Snapshotter’s query queue (§8.4). Reads are served on every role including Initializing and Follower; only the freshness signal (last_tx_id) tells the caller how stale the answer is.

§36.3 GetPipelineIndex

Returns the five progress indexes — compute, commit, snapshot, term, cluster_commit — in one call. Used by the load_cluster benchmark and external monitoring to track replication lag and pipeline depth. Cheap: every field is one atomic load.

§36.4 WaitForTransaction

Request fields: tx_id, wait_level, optional term fence. Response field: outcome enum with REACHED, NOT_FOUND, TERM_MISMATCH. The optional term fence tells the server “I expect to be talking to term N”; if the server’s current term differs, the response carries TERM_MISMATCH along with the actual term and term_start_tx_id so the client can decide whether to redirect or re-query.

§36.5 Function registry RPCs

RegisterFunction, UnregisterFunction, ListFunctions. Writes are gated on role.is_leader(); on a leader they go through the Transactor like any other operation (§6.16, §6.17). ListFunctions reads the live registry and can be served on any role.

§36.6 Read-only mode on followers and Initializing

A follower’s client handler is constructed in read-only mode; every submit_* and register_function returns FAILED_PRECONDITION unconditionally, regardless of leader-hint availability. The Initializing role uses the same read-only handler. This is what prevents a partitioned former leader from accepting writes that would later have to be discarded. [ADR-0004, ADR-0015]

§37 Client library

§37.1 LedgerClient is a thin tonic wrapper

The client library is one type, LedgerClient, holding a tonic Channel (cloneable; tonic pools internally). Construction is LedgerClient::connect(addr).await. The library does not maintain its own connection pool, retry policy, or leader-discovery cache — applications wrap calls themselves and use response metadata (term, leader hint) when steering.

§37.2 One async method per RPC

Every gRPC RPC has a corresponding async fn on LedgerClient that takes the natural Rust types as parameters and returns a Rust-side response type (SubmitResult, Balance, Transaction, PipelineIndex, WaitOutcome, FunctionInfo, …). The wrapper hides protobuf message construction; callers never touch generated types. The methods take &self so a single LedgerClient can be cloned and shared across many concurrent callers — the underlying tonic Channel handles multiplexing.

§37.3 No built-in failover

A LedgerClient is bound to one address. Cluster-aware applications construct one client per node, make the call against the suspected leader, and on a FAILED_PRECONDITION with leader hint redirect to the indicated peer. Implementing leader-hint following inside the client library is deliberately deferred: the hint is on every error metadata, but the application owns the policy (try once and fail vs. retry on the hinted node vs. consult an external service).

§38 Cluster startup, supervisor, shutdown

§38.1 ClusterNode::new constructs the Ledger and the Term log

ClusterNode::new(config) constructs the in-process Ledger via Ledger::start() — the cold-boot path, never the divergence path — opens Term, durably bumps the term once on every restart so that any in-flight election is forced to start from a higher term than was observed pre-crash, and wraps the Ledger in LedgerSlot for later indirection (§33.6).

§38.2 Supervisor boot

The role supervisor builds the long-lived shared resources: Quorum (sized to peers + 1), the cooperative running: Arc<AtomicBool>, and the transition channel. It then spawns the client-facing gRPC server, the peer-facing gRPC server, the divergence watcher, and the role driver loop. The two gRPC servers are bound for the entire lifetime of the process; they never restart on a role transition.

§38.3 Initial role

A single-node cluster (peers.len() == 1 and the only entry is self) is brought up with initial_role = Leader so the node is immediately writable without waiting for an election timer. A multi-node cluster starts in Initializing and waits for the first election timer to expire before becoming a candidate.

§38.4 The role-driver loop

The driver awaits the appropriate signal per role: an election-timer expiry while in Initializing or Follower; the candidate round’s outcome while in Candidate; a transition-channel message while in Leader. Each transition drops the previous role’s Handles (which joins or aborts every owned task) before constructing the next role’s Handles. The driver itself owns no role-specific state; it is purely the coordinator that decides which role to bring up next.

§38.5 Shutdown ordering

Shutdown flips running to false, sends a Shutdown transition message (which wakes the role driver if it is parked), and aborts every spawned task in order — the role driver first, the divergence watcher, the peer-facing server, the client-facing server. Peer replication tasks observe running at the top of their loop and exit cleanly; the gRPC servers run their tonic shutdown protocol before the abort takes effect.

§39 Standalone, single-node, and testing modes

§39.1 Standalone (no [cluster] config block)

When the configuration has no [cluster] section, ClusterNode does not invoke the supervisor at all. It constructs a single client-facing Server task with a hardcoded Role::Leader posture, no RoleFlag, no Node gRPC server, no Term, no Vote, no Quorum, no peer tasks. The deployment is identical in observable behaviour to a single-node cluster but pays none of the supervisor’s coordination cost. This is the configuration roda ships with by default.

§39.2 Single-node cluster (peers.len() == 1)

A cluster whose configuration lists exactly one peer (itself) runs the full supervisor infrastructure: a RoleFlag, a Term, a Vote, a single-slot Quorum. The node boots as Leader (§38.3) and the replication loop has zero peers to fan out to; every on_commit hook firing immediately advances the cluster watermark since there is no one to wait for. This mode exercises the cluster code path against a trivial cluster shape and is the one used by most cluster integration tests.

§39.3 Multi-node cluster

A cluster with two or more peers runs the full Raft state machine: a node boots as Initializing, awaits the election timer, transitions through Candidate to either Leader or back to Initializing, runs Leader with peer replication tasks until it sees a higher term or loses its majority, etc. This is the production deployment shape.

§39.4 ClusterTestingControl

ClusterTestingControl is the in-process test harness for composing N-node clusters in a single process: distinct ports on localhost, temporary data directories per node, helpers to inject network partitions, drop messages, kill nodes mid-flight, and wait for specific cluster-wide states. It is not a production code path; it exists to give the test suite, the load_cluster benchmark, and manual debugging a uniform way to construct and tear down clusters without spawning processes.

Load Test Report

Generated by the Load Test workflow on a CCX33 server (Hetzner, 8 dedicated vCPU), 1,000,000 accounts, deposit operations, release build with WAL persistence enabled.

• Throughput — sustained tx/s with submit-side latency (load).
• Latency under load — end-to-end latency to each pipeline stage (load_latency --wait-level).
• WASM deposit throughput — the same deposit driven through a registered WASM function (load_wasm).

Throughput

  backlog seq>cmp/cmp>cmt/cmt>snp = txs waiting before transactor / WAL / snapshot

  +-----+--------+------------+------------+----------+----------+------------+--------------------------+
  |   # |   time |        TPS |        TPC |      P50 |      P99 |  in-flight |  seq>cmp/cmp>cmt/cmt>snp |
  +-----+--------+------------+------------+----------+----------+------------+--------------------------+
  |   1 |     1s |    5463224 |    5465533 |     50ns |    180ns |      34468 |          1.4k/33.1k/4.6k |
  |   2 |     2s |    5814949 |   11283071 |     50ns |    170ns |      26930 |           1.2k/25.7k/580 |
  |   3 |     3s |    5772757 |   17057020 |     50ns |     80ns |      22981 |         2.0k/21.0k/13.6k |
  |   4 |     4s |    5972568 |   23034601 |     50ns |    130ns |      35400 |             1.6k/33.8k/0 |
  |   5 |     5s |    5950633 |   28986992 |     50ns |    110ns |      23009 |         1.4k/21.6k/21.0k |
  |   6 |     6s |    5948182 |   34943844 |     50ns |     90ns |      26157 |          1.7k/24.5k/7.5k |
  |   7 |     7s |    5855536 |   40809676 |     50ns |     90ns |      20325 |         1.4k/19.0k/12.4k |
  |   8 |     8s |    5662418 |   46474347 |     50ns |    150ns |      25654 |          1.5k/24.2k/9.6k |
  |   9 |     9s |    5751787 |   52232937 |     50ns |    120ns |      27064 |          1.5k/25.6k/4.0k |
  |  10 |    10s |    5678696 |   57918017 |     50ns |     80ns |      21984 |         1.3k/20.6k/19.0k |
  |  11 |    11s |    5865205 |   63785450 |     50ns |    120ns |      14551 |         2.0k/12.5k/20.0k |
  |  12 |    12s |    5616781 |   69411862 |     50ns |     90ns |      18139 |          1.0k/17.1k/5.9k |
  |  13 |    13s |    5543264 |   74961838 |     60ns |    110ns |      28163 |             1.7k/26.4k/0 |
  |  14 |    14s |    5636401 |   80604530 |     60ns |     90ns |      25471 |          1.5k/23.9k/2.8k |
  |  15 |    15s |    5994236 |   86599972 |     50ns |     90ns |      30029 |         1.9k/28.1k/16.3k |
  |  16 |    16s |    5625982 |   92234100 |     50ns |     90ns |      25901 |          2.0k/23.9k/4.7k |
  |  17 |    17s |    5760654 |   97998172 |     50ns |     90ns |      31829 |             1.7k/30.1k/0 |
  |  18 |    18s |    5319210 |  103327933 |     50ns |     80ns |      22068 |             1.0k/21.0k/0 |
  |  19 |    19s |    5224668 |  108562153 |     50ns |     80ns |      27848 |          1.2k/26.6k/2.6k |
  |  20 |    20s |    4942713 |  113513667 |     50ns |    100ns |      26334 |             1.2k/25.1k/0 |
  |  21 |    21s |    5283754 |  118800235 |     50ns |     80ns |      29766 |             1.2k/28.5k/0 |
  |  22 |    22s |    5400792 |  124204777 |     50ns |     80ns |      25224 |          1.7k/23.6k/5.1k |
  |  23 |    23s |    5501954 |  129713837 |     50ns |     80ns |      36164 |          1.3k/34.9k/4.4k |
  |  24 |    24s |    5663565 |  135380518 |     60ns |     90ns |      29483 |         1.6k/27.8k/10.3k |
  |  25 |    25s |    5887831 |  141276169 |     50ns |     90ns |      23832 |         1.6k/22.2k/21.2k |
  |  26 |    26s |    5834397 |  147116469 |     50ns |    100ns |      13532 |         2.0k/11.6k/11.5k |
  |  27 |    27s |    5513843 |  152633907 |     50ns |     80ns |      36094 |            1.1k/35.0k/84 |
  |  28 |    28s |    6061387 |  158700409 |     60ns |     90ns |      29592 |          1.1k/28.5k/7.1k |
  |  29 |    29s |    6047687 |  164757515 |     50ns |     90ns |      22486 |         1.3k/21.2k/19.3k |
  |  30 |    30s |    5742092 |  170506496 |     50ns |     90ns |      23505 |          1.5k/22.0k/6.2k |
  |  31 |    31s |    5758640 |  176273012 |     50ns |    110ns |      26989 |         1.3k/25.7k/14.5k |
  |  32 |    32s |    5814866 |  182093153 |     50ns |     80ns |      36848 |          1.6k/35.2k/2.8k |
  |  33 |    33s |    5837653 |  187931634 |     50ns |    100ns |      28367 |         2.0k/26.4k/12.4k |
  |  34 |    34s |    5520128 |  193460329 |     50ns |    120ns |      19672 |         1.6k/18.0k/14.4k |
  |  35 |    35s |    5848399 |  199313599 |     50ns |     80ns |      36402 |         1.0k/35.4k/11.8k |
  |  36 |    36s |    5677018 |  205001043 |     50ns |     80ns |      18958 |             1.1k/17.8k/0 |
  |  37 |    37s |    5546804 |  210552615 |     50ns |     80ns |      17386 |          1.1k/16.3k/7.7k |
  |  38 |    38s |    5598484 |  216153991 |     50ns |    140ns |      36010 |          1.4k/34.7k/4.6k |
  |  39 |    39s |    5896890 |  222055281 |     50ns |     90ns |      34720 |          1.1k/33.7k/9.7k |
  |  40 |    40s |    5822545 |  227882729 |     50ns |    120ns |      47272 |             1.2k/46.1k/0 |
  |  41 |    41s |    5826042 |  233709677 |     50ns |     80ns |      30324 |          2.0k/28.3k/9.5k |
  |  42 |    42s |    6008616 |  239718376 |     50ns |     80ns |      21625 |         1.4k/20.2k/14.8k |
  |  43 |    43s |    5700633 |  245427490 |     60ns |     90ns |      22511 |          1.6k/20.9k/9.0k |
  |  44 |    44s |    5535999 |  250966840 |     50ns |    100ns |      23161 |          1.7k/21.4k/4.5k |
  |  45 |    45s |    5389133 |  256363365 |     60ns |     90ns |      46636 |          1.7k/44.9k/3.9k |
  |  46 |    46s |    5766583 |  262130911 |     60ns |     90ns |      29090 |             1.9k/27.1k/0 |
  |  47 |    47s |    5406387 |  267545967 |     60ns |     90ns |      34034 |         1.9k/32.1k/10.9k |
  |  48 |    48s |    5951848 |  273508073 |     50ns |    110ns |      21928 |          1.4k/20.5k/6.8k |
  |  49 |    49s |    5627332 |  279144169 |     60ns |     90ns |      55832 |             1.6k/54.3k/0 |
  |  50 |    50s |    5726611 |  284878573 |     50ns |    100ns |      41428 |             1.5k/39.9k/0 |
  |  51 |    51s |    5752839 |  290635800 |     50ns |    100ns |      24201 |          2.0k/22.2k/6.4k |
  |  52 |    52s |    5799717 |  296438901 |     50ns |     90ns |      21100 |         2.0k/19.1k/17.7k |
  |  53 |    53s |    5697888 |  302142026 |     50ns |     90ns |      47975 |         1.7k/46.2k/39.9k |
  |  54 |    54s |    5897563 |  308041583 |     50ns |     90ns |      28418 |         1.9k/26.5k/28.7k |
  |  55 |    55s |    5929769 |  313975938 |     50ns |     90ns |      24063 |         1.9k/22.1k/18.7k |
  |  56 |    56s |    5897507 |  319877724 |     50ns |    120ns |      42277 |           1.2k/41.1k/169 |
  |  57 |    57s |    5934438 |  325820044 |     60ns |    100ns |      29957 |         1.2k/28.8k/10.7k |
  |  58 |    58s |    5446729 |  331271405 |     60ns |     80ns |      28596 |          1.5k/27.1k/6.6k |
  |  59 |    59s |    5581038 |  336858745 |     60ns |     90ns |      31256 |          1.4k/29.8k/1.7k |
  +-----+--------+------------+------------+----------+----------+------------+--------------------------+

  ╔══════════════════════════════════════════════╗
  ║              LOAD TEST SUMMARY               ║
  ╠══════════════════════════════════════════════╣
  ║  Duration      :      60.00s                 ║
  ║  Submitted     :  341880001                  ║
  ║  Avg TPS       :    5697963                  ║
  ║  Mode          :      async                  ║
  ╠══════════════════════════════════════════════╣
  ║  P50  Latency  :       50ns                  ║
  ║  P99  Latency  :       90ns                  ║
  ║  P999 Latency  :      220ns                  ║
  ║  Min  Latency  :       30ns                  ║
  ║  Max  Latency  :     11.2µs                  ║
  ║  Samples       :      34189                  ║
  ╚══════════════════════════════════════════════╝

Latency Under Load

Each probe submits one deposit and times it until the selected pipeline index reaches it, repeated 1000x per scenario across five load levels. --wait-level picks the stage:

wait-level: compute

  latency under load | accounts=1000000 samples=1000 probe=deposit→compute (executed)

  +---------------+--------+--------------+----------+----------+----------+----------+
  | Scenario      | Target | Achieved TPS |   P50    |   P99    |   P999   |   Max    |
  +---------------+--------+--------------+----------+----------+----------+----------+
  | No Load       |      - |        1.03M |    820ns |    1.6µs |   23.8µs |   32.0µs |
  | Sustain 10k   |  10.0k |        1.54M |    570ns |    790ns |    5.2µs |    6.9µs |
  | Sustain 100k  | 100.0k |        1.65M |    550ns |    930ns |    9.1µs |   13.0µs |
  | Sustain 1M    |  1.00M |        2.37M |    620ns |    1.0µs |    2.5µs |   16.5µs |
  | Full (Spike)  |    max |        5.67M |  292.4µs |  788.0µs |  967.2µs |    1.2ms |
  +---------------+--------+--------------+----------+----------+----------+----------+

[2026-06-08 07:08:36.139] [info] Cleaning up temporary storage...

wait-level: write

  latency under load | accounts=1000000 samples=1000 probe=deposit→write (page cache, pre-fsync)

  +---------------+--------+--------------+----------+----------+----------+----------+
  | Scenario      | Target | Achieved TPS |   P50    |   P99    |   P999   |   Max    |
  +---------------+--------+--------------+----------+----------+----------+----------+
  | No Load       |      - |       151.1k |    3.7µs |   31.9µs |   84.1µs |  351.5µs |
  | Sustain 10k   |  10.0k |       240.2k |    2.8µs |   16.8µs |   80.2µs |  145.8µs |
  | Sustain 100k  | 100.0k |       432.0k |    2.8µs |    9.0µs |   26.4µs |   31.2µs |
  | Sustain 1M    |  1.00M |        1.29M |    2.4µs |   12.3µs |   79.9µs |  131.1µs |
  | Full (Spike)  |    max |        4.43M |  403.2µs |  776.7µs |    1.3ms |    3.2ms |
  +---------------+--------+--------------+----------+----------+----------+----------+

[2026-06-08 07:08:39.757] [info] Cleaning up temporary storage...

wait-level: commit

  latency under load | accounts=1000000 samples=1000 probe=deposit→commit (durable, fsync)

  +---------------+--------+--------------+----------+----------+----------+----------+
  | Scenario      | Target | Achieved TPS |   P50    |   P99    |   P999   |   Max    |
  +---------------+--------+--------------+----------+----------+----------+----------+
  | No Load       |      - |         1.5k |  597.0µs |    1.6ms |    3.2ms |    4.4ms |
  | Sustain 10k   |  10.0k |        10.8k |    1.3ms |    2.1ms |    3.0ms |    5.1ms |
  | Sustain 100k  | 100.0k |       100.7k |    1.4ms |    2.2ms |    2.6ms |    2.7ms |
  | Sustain 1M    |  1.00M |        1.00M |    2.5ms |    4.0ms |    5.2ms |    5.4ms |
  | Full (Spike)  |    max |        5.64M |    5.8ms |    9.4ms |   13.4ms |   13.9ms |
  +---------------+--------+--------------+----------+----------+----------+----------+

wait-level: snapshot

  latency under load | accounts=1000000 samples=1000 probe=deposit→snapshot (readable)

  +---------------+--------+--------------+----------+----------+----------+----------+
  | Scenario      | Target | Achieved TPS |   P50    |   P99    |   P999   |   Max    |
  +---------------+--------+--------------+----------+----------+----------+----------+
  | No Load       |      - |         1.5k |  599.0µs |    1.2ms |    1.3ms |    1.4ms |
  | Sustain 10k   |  10.0k |        10.8k |    1.2ms |    1.9ms |    3.2ms |    5.2ms |
  | Sustain 100k  | 100.0k |       100.8k |    1.3ms |    1.6ms |    2.9ms |    3.5ms |
  | Sustain 1M    |  1.00M |        1.00M |    2.5ms |    4.2ms |    6.9ms |    7.1ms |
  | Full (Spike)  |    max |        4.44M |    4.5ms |    7.6ms |   11.3ms |   15.8ms |
  +---------------+--------+--------------+----------+----------+----------+----------+

[2026-06-08 07:09:09.274] [info] Cleaning up temporary storage...

WASM Deposit Throughput

The deposit executed as a registered WASM Operation::Function instead of the built-in op; the gap vs Throughput above is the WASM invocation cost.

  wasm deposit load | accounts=1000000 fn=wasm_deposit mode=async

  backlog seq>cmp/cmp>cmt/cmt>snp = txs waiting before transactor / WAL / snapshot

  +-----+--------+------------+------------+----------+----------+------------+--------------------------+
  |   # |   time |        TPS |        TPC |      P50 |      P99 |  in-flight |  seq>cmp/cmp>cmt/cmt>snp |
  +-----+--------+------------+------------+----------+----------+------------+--------------------------+
  |   1 |     1s |    3058061 |    3065345 |     60ns |  228.7µs |       4656 |           1.1k/3.6k/1.5k |
  |   2 |     2s |    3127276 |    6202275 |     60ns |  262.9µs |       7726 |            1.5k/6.2k/858 |
  |   3 |     3s |    3070127 |    9280377 |     60ns |  268.8µs |       9624 |           1.3k/8.3k/2.9k |
  |   4 |     4s |    3190317 |   12473951 |     60ns |  258.7µs |       6050 |           1.5k/4.5k/4.0k |
  |   5 |     5s |    3170902 |   15649991 |     60ns |  257.5µs |      10010 |              2.0k/8.0k/0 |
  |   6 |     6s |    3094147 |   18749333 |     60ns |  264.2µs |      10668 |              1.3k/9.3k/0 |
  |   7 |     7s |    3243125 |   21999809 |     60ns |  240.9µs |      10192 |              1.2k/9.0k/0 |
  |   8 |     8s |    3167886 |   25170129 |     60ns |  248.3µs |       9872 |           1.9k/8.0k/1.2k |
  |   9 |     9s |    3193883 |   28370812 |     60ns |  262.0µs |       9189 |           1.9k/7.3k/6.0k |
  |  10 |    10s |    3110298 |   31491073 |     60ns |  261.2µs |       8928 |           1.8k/7.2k/2.7k |
  |  11 |    11s |    3066702 |   34561596 |     60ns |  265.2µs |       8405 |              1.8k/6.6k/0 |
  |  12 |    12s |    3015670 |   37578787 |     60ns |  266.2µs |      21214 |             1.8k/19.4k/0 |
  |  13 |    13s |    3189255 |   40772304 |     60ns |  256.3µs |       7697 |           1.2k/6.4k/6.3k |
  |  14 |    14s |    3082180 |   43857255 |     60ns |  250.9µs |      12746 |             1.8k/10.9k/0 |
  |  15 |    15s |    3005812 |   46866270 |     50ns |  280.3µs |      13731 |             1.3k/12.5k/0 |
  |  16 |    16s |    3011165 |   49882080 |     70ns |  259.2µs |       7921 |           1.7k/6.2k/2.7k |
  |  17 |    17s |    2950472 |   52839962 |     60ns |  279.3µs |      10039 |              1.4k/8.7k/0 |
  |  18 |    18s |    3042705 |   55887956 |     60ns |  257.0µs |      12045 |             1.9k/10.2k/0 |
  |  19 |    19s |    3156148 |   59049365 |     60ns |  258.6µs |      10636 |              1.8k/8.8k/0 |
  |  20 |    20s |    3075858 |   62128626 |     60ns |  268.5µs |      11375 |              1.6k/9.7k/0 |
  |  21 |    21s |    3174224 |   65308673 |     60ns |  255.9µs |      11328 |          1.1k/10.2k/3.1k |
  |  22 |    22s |    3116242 |   68433293 |     60ns |  256.0µs |       6708 |           2.0k/4.7k/4.7k |
  |  23 |    23s |    3046152 |   71487411 |     60ns |  280.3µs |      12590 |             1.2k/11.3k/0 |
  |  24 |    24s |    2885426 |   74377159 |     60ns |  314.1µs |      12842 |             1.5k/11.3k/0 |
  |  25 |    25s |    2899715 |   77279720 |     60ns |  282.9µs |      10281 |              1.6k/8.7k/0 |
  |  26 |    26s |    3051189 |   80331774 |     60ns |  269.3µs |       8227 |              1.1k/7.2k/0 |
  |  27 |    27s |    3144253 |   83481194 |     60ns |  253.8µs |       8807 |           1.2k/7.6k/2.8k |
  |  28 |    28s |    3265458 |   86750010 |     60ns |  241.4µs |       9991 |           1.6k/8.4k/3.3k |
  |  29 |    29s |    3150766 |   89908301 |     60ns |  246.0µs |      11700 |             1.5k/10.2k/0 |
  |  30 |    30s |    3038921 |   92949790 |     60ns |  253.2µs |      10211 |              1.3k/8.9k/0 |
  |  31 |    31s |    3011219 |   95968421 |     60ns |  270.3µs |      11580 |             1.5k/10.1k/0 |
  |  32 |    32s |    3024085 |   98998529 |     60ns |  284.4µs |      11472 |             1.5k/10.0k/0 |
  |  33 |    33s |    3019366 |  102022125 |     60ns |  209.0µs |       7876 |           1.7k/6.2k/5.1k |
  |  34 |    34s |    2997864 |  105021962 |     60ns |  260.2µs |       8039 |              1.4k/6.6k/0 |
  |  35 |    35s |    3049249 |  108076331 |     60ns |  264.4µs |      13670 |             1.7k/12.0k/0 |
  |  36 |    36s |    3204308 |  111282386 |     60ns |  253.7µs |       7615 |           1.7k/5.9k/2.6k |
  |  37 |    37s |    3204331 |  114495315 |     60ns |  252.2µs |      14686 |             1.2k/13.5k/0 |
  |  38 |    38s |    3080555 |  117581358 |     60ns |  254.8µs |       8643 |              2.0k/6.6k/0 |
  |  39 |    39s |    2963014 |  120549658 |     60ns |  266.8µs |      10343 |              1.4k/8.9k/0 |
  |  40 |    40s |    3124954 |  123682305 |     60ns |  256.5µs |       7696 |           1.0k/6.7k/6.0k |
  |  41 |    41s |    3076104 |  126759734 |     60ns |  264.4µs |      10267 |              1.9k/8.4k/0 |
  |  42 |    42s |    3170380 |  129938671 |     60ns |  252.7µs |      11330 |             1.3k/10.0k/0 |
  |  43 |    43s |    3169783 |  133109761 |     60ns |  269.6µs |      10240 |           1.0k/9.2k/2.0k |
  |  44 |    44s |    3146964 |  136260669 |     60ns |  266.5µs |       9332 |           1.2k/8.1k/7.7k |
  |  45 |    45s |    3201220 |  139469031 |     60ns |  257.8µs |      10970 |           2.0k/9.0k/6.6k |
  |  46 |    46s |    3040801 |  142510399 |     60ns |  258.0µs |       9602 |           1.7k/7.9k/3.5k |
  |  47 |    47s |    3105154 |  145621139 |     60ns |  256.4µs |       8862 |           1.8k/7.0k/3.6k |
  |  48 |    48s |    3050428 |  148672513 |     60ns |  252.9µs |       7488 |           1.3k/6.1k/2.8k |
  |  49 |    49s |    3115016 |  151789915 |     60ns |  266.0µs |      10086 |              1.2k/8.9k/0 |
  |  50 |    50s |    3060645 |  154852671 |     60ns |  270.8µs |       7330 |           1.5k/5.8k/5.4k |
  |  51 |    51s |    3117129 |  157974052 |     60ns |  262.7µs |      15949 |             1.1k/14.8k/0 |
  |  52 |    52s |    3028376 |  161002713 |     60ns |  256.0µs |       7288 |              1.4k/5.9k/0 |
  |  53 |    53s |    2965837 |  163972436 |     60ns |  284.2µs |       7565 |           1.8k/5.8k/1.4k |
  |  54 |    54s |    3005336 |  166980651 |     70ns |  273.7µs |       9350 |              1.2k/8.2k/0 |
  |  55 |    55s |    3044718 |  170030443 |     60ns |  258.3µs |       9558 |              1.7k/7.8k/0 |
  |  56 |    56s |    3050977 |  173081922 |     60ns |  267.0µs |       8079 |              1.2k/6.8k/0 |
  |  57 |    57s |    3055107 |  176137507 |     60ns |  270.3µs |      12494 |             1.5k/11.0k/0 |
  |  58 |    58s |    3069220 |  179208800 |     60ns |  264.7µs |      11201 |            1.6k/9.6k/330 |
  |  59 |    59s |    3158371 |  182367233 |     60ns |  260.2µs |      12768 |             1.5k/11.3k/0 |
  +-----+--------+------------+------------+----------+----------+------------+--------------------------+

  ╔══════════════════════════════════════════════╗
  ║          WASM DEPOSIT LOAD SUMMARY           ║
  ╠══════════════════════════════════════════════╣
  ║  Duration      :      60.00s                 ║
  ║  Submitted     :  185360001                  ║
  ║  Avg TPS       :    3089220                  ║
  ║  Mode          :      async                  ║
  ╠══════════════════════════════════════════════╣
  ║  P50  Latency  :       60ns                  ║
  ║  P99  Latency  :    263.2µs                  ║
  ║  P999 Latency  :    310.0µs                  ║
  ║  Min  Latency  :       30ns                  ║
  ║  Max  Latency  :    357.6µs                  ║
  ║  Samples       :      18537                  ║
  ╚══════════════════════════════════════════════╝

Architecture Decision Records

This section contains Architecture Decision Records (ADRs) documenting significant design choices made during the development of Roda-Ledger.

ADR-001: Entries-based Execution Model

Status: Accepted
Date: 2026-03-25
Author: Taleh Ibrahimli

Context

Roda-ledger currently uses a generic Ledger<TransactionType, BalanceType> model where user-defined transaction logic executes in two places:

• Transactor — executes business logic, produces balance changes
• Snapshotter — re-executes the same business logic to materialize balances

This double-execution model creates several compounding problems:

1. Double execution problem
Business logic runs twice per transaction — once in the Transactor, once in the Snapshotter. Even though all operations are deterministic by design (including future WASM operations which enforce determinism architecturally via sandbox constraints), executing business logic twice is wasteful, tightly couples the Snapshotter to user code, and complicates the recovery path unnecessarily.

2. WAL stores logical operations, not physical changes
The WAL currently stores raw Transaction<Data> structs — the user’s operation, not its result. On crash recovery, the system must re-execute business logic to reconstruct balances. This couples the WAL format to the user’s code and makes the WAL format non-generic — it cannot be read or replayed by any component that does not understand the user’s transaction type.

3. Snapshot stage re-executes business logic
The Snapshotter receives raw transactions and must execute them to derive balance changes. This means the Snapshotter must be aware of user-defined logic, preventing a clean separation between the execution layer and the persistence layer.

4. WASM custom operations require single execution
Future WASM sandbox support requires that custom logic executes exactly once in a controlled environment and produces a set of entries. Re-execution in the Snapshotter is architecturally incompatible with this model.

Decision

Replace the generic transaction execution model with an entries-based execution model.

Business logic executes exactly once — in the Transactor. The result of execution is a set of immutable Credit and Debit entries. Everything downstream — WAL Storer, Snapshotter — operates only on entries, never on business logic.

The Transactor produces WAL records directly. The WAL record format is the single shared structure used by all pipeline stages. There is no intermediate representation — no copying from one structure to another between stages.

Core types

#![allow(unused)]
fn main() {
#[repr(u8)]
#[derive(Copy, Clone, Debug)]
pub enum WalEntryKind {
    TxMetadata = 0,
    TxEntry    = 1,
}

#[repr(u8)]
#[derive(Copy, Clone, Debug)]
pub enum EntryKind {
    Credit = 0,
    Debit  = 1,
}
}

FailReason is a newtype wrapper over u8 to support user-defined custom codes in the 128–255 range while remaining bytemuck compatible:

#![allow(unused)]
fn main() {
pub struct FailReason(u8);
impl FailReason {
    pub const NONE:                  Self = Self(0);  // success
    pub const INSUFFICIENT_FUNDS:    Self = Self(1);
    pub const ACCOUNT_NOT_FOUND:     Self = Self(2);
    pub const ZERO_SUM_VIOLATION:    Self = Self(3);
    pub const ENTRY_LIMIT_EXCEEDED:  Self = Self(4);
    pub const INVALID_OPERATION:     Self = Self(5);
    // 6–127   reserved for future standard reasons
    // 128–255 user-defined custom reasons

    pub fn is_success(&self) -> bool { self.0 == 0 }
    pub fn is_failure(&self) -> bool { self.0 != 0 }
}
}

WAL record structures

Both record types are exactly 32 bytes. The WAL scanner reads 1 discriminant byte, then always reads exactly 32 bytes of payload — no branching on size, no variable-length parsing.

#![allow(unused)]
fn main() {
#[repr(C)]
#[derive(Copy, Clone, Debug, Pod, Zeroable)]
pub struct TxMetadata {
    pub tx_id:       u64,        // 8
    pub timestamp:   u64,        // 8
    pub user_ref:    u64,        // 8 — opaque external reference, caller owns context
    pub entry_count: u8,         // 1 — 0 if transaction failed
    pub fail_reason: FailReason, // 1 — FailReason::NONE if succeeded
    pub _pad:        [u8; 6],    // 6 — total: 32 bytes
}

#[repr(C)]
#[derive(Copy, Clone, Debug, Pod, Zeroable)]
pub struct TxEntry {
    pub tx_id:      u64,       // 8 — links back to TxMetadata
    pub account_id: u64,       // 8
    pub amount:     u64,       // 8 — integer minor units, no floats, no decimals
    pub kind:       EntryKind, // 1 — Credit or Debit
    pub _pad:       [u8; 7],   // 7 — total: 32 bytes
}
}

Transaction status

Transaction status is not stored. It is derived at query time from pipeline stage indexes:

tx_id <= transactor_index  →  Computed
tx_id <= wal_index         →  Committed
tx_id <= snapshot_index    →  Snapshoted

Transactions are immutable once written. Status is a view over pipeline progress, not a property of the record itself.

Failed transactions

A failed transaction is written to WAL as a TxMetadata record with fail_reason != 0 and entry_count == 0. No TxEntry records follow. This preserves the audit trail — every attempt is recorded — while consuming minimal space (32 bytes per failed attempt).

fail_reason == 0, entry_count > 0  →  succeeded, entries follow
fail_reason != 0, entry_count == 0 →  failed, no entries

Execution flow

User Operation
    ↓ execute once (Transactor)
    ↓ produces WAL records directly
TxMetadata + TxEntry[]
    ↓
WAL Storer     — persists records, advances wal_index
Snapshotter    — applies entries to balance cache, advances snapshot_index

The WAL record format is the single shared structure across all stages. The Transactor produces it, the WAL Storer persists it, the Snapshotter consumes it. No intermediate copying between representations.

External accounts

For operations where value enters or leaves the system (deposit, withdrawal), reserved system accounts absorb the imbalance to maintain the zero-sum invariant:

Deposit 100 to account A:
  Credit(SYSTEM, 100)
  Debit(A,              100)
  sum = 0 ✓

Withdrawal 50 from account A:
  Credit(A,           50)
  Debit(SYSTEM,  50)
  sum = 0 ✓

SYSTEM is reserved system accounts. Their balances are not tracked — they exist only as entry anchors. Every transaction in the system balances to zero.

Zero-sum invariant

The ledger enforces automatically:

sum(credits) == sum(debits)

Any operation violating this is rejected with FailReason::ZERO_SUM_VIOLATION before reaching the WAL. This invariant ensures no value is created or destroyed inside the ledger — every unit that enters via SYSTEM must exit via SYSTEM or reside in a user account.

Why manual discriminant, not Rust enum

A Rust enum wrapping both TxMetadata and TxEntry variants would be sized to the largest variant plus discriminant alignment padding — likely 40 or 48 bytes. Manual discriminant keeps both record types at exactly 32 bytes payload with no compiler padding tax, and makes the WAL format language-agnostic — any language can read it from the spec alone.

Metadata and strings

The WAL stores no strings. All metadata is encoded as fixed-width integer types:

• fail_reason — FailReason newtype, 1 byte
• user_ref — opaque u64, the external system’s reference key

Human-readable context (description, notes, reason text) is the external system’s responsibility. They store it in their own database and join on user_ref. The ledger does not own this concern.

Consequences

Positive

• Business logic executes exactly once — clean separation between execution and persistence
• WAL format is generic and language-agnostic — readable by any component without knowing user code
• Snapshotter applies entries directly — no dependency on user-defined transaction types
• WASM sandbox becomes viable — custom logic runs once, produces entries, done
• Crash recovery replays entries — no dependency on code version at recovery time
• Immutable audit trail — every attempt recorded, entries append-only, never edited
• Zero-sum invariant enforced architecturally — no value created or destroyed inside the ledger
• Single shared structure across pipeline stages — no copying between representations

Negative

• Breaking API change — existing Ledger<TransactionType, BalanceType> generics are removed
• Wallet example requires rewriting against the new entry-based API
• User migration required for anyone using custom TransactionDataType implementations
• Failed transactions occupy WAL space (32 bytes per TxMetadata) even with no entries — acceptable given audit trail value

References

• PostgreSQL WAL internals — SQL executes once on primary, WAL contains physical page changes, replicas replay WAL
• TigerBeetle transfer model — explicit debit/credit primitives, no custom logic
• ISO 20022 — EndToEndId as opaque external reference key (maps to user_ref)
• Double-entry bookkeeping — every transaction balances: sum(debits) = sum(credits)

ADR-002: Vec-based Balance Storage

Status: Accepted
Date: 2026-03-27
Author: Taleh Ibrahimli

Context

During the entries-based refactor (ADR-001), balance storage for the pipeline stages needed a new design. Several concurrent map structures were evaluated:

SkipMap (crossbeam) was the original implementation. Benchmarks showed:

Accounts    SkipMap TPS
───────────────────────
1K          ~1.4M
1M          ~960K
10M         ~755K
50M         ~460K

evmap was evaluated as a replacement — theoretically O(1) vs SkipMap’s O(log n). Benchmarks showed evmap wins at small scale but loses at large scale:

Accounts    evmap TPS    SkipMap TPS    Winner
──────────────────────────────────────────────
1K          4.21M        ~1.4M          evmap
1M          1.20M        ~960K          evmap
10M         748K         ~755K          SkipMap
50M         627K         ~460K          SkipMap

evmap is theoretically O(1) but degrades at large account counts due to refresh() synchronization cost, reader contention, and maintaining two full internal copies of the map. SkipMap is O(log n) but wins at scale due to predictable memory access and no synchronization overhead.

Both are the wrong data structure. The fundamental insight is that account IDs are sequential by design — this enables direct array indexing with zero overhead.

Decision

Replace all balance storage structures with flat Vec-based arrays using account ID as a direct index.

Core insight

Account IDs are sequential and guaranteed contiguous — you cannot have account_id=78 without also having account_id=77. This constraint enables direct array indexing:

balance = vec[account_id]   // single CPU instruction, O(1), no hashing, no traversal

Structures

Transactor — plain Vec:

#![allow(unused)]
fn main() {
balances: Vec<u64>
// read:  balances[account_id as usize]
// write: balances[account_id as usize] = new_balance
}

Single contiguous allocation. CPU prefetcher works perfectly. No hashing, no pointer chasing, no tree traversal. Single writer — no synchronization needed.

Snapshotter — Vec of atomics:

#![allow(unused)]
fn main() {
balances: Vec<AtomicU64>
// read:  balances[account_id as usize].load(Ordering::Acquire)
// write: balances[account_id as usize].store(value, Ordering::Release)
}

Same contiguous layout. External balance reads are a single array index plus atomic load. No map lookup, no synchronization beyond the atomic operation itself.

Capacity management

The ledger accepts max_accounts at configuration time:

#![allow(unused)]
fn main() {
pub struct LedgerConfig {
    pub max_accounts: usize,
}
}

Both Vecs are pre-allocated at startup — zero reallocation during operation, zero rehashing stalls. If account_id >= max_accounts, the operation is rejected with FailReason::AccountLimitExceeded before reaching the Transactor.

Resize at runtime:

#![allow(unused)]
fn main() {
fn resize(&mut self, new_size: usize) {
    self.balances.resize(new_size, 0u64);  // new slots zero-initialized
    // existing balances untouched
}
}

Resize increases memory allocation only — existing account data is never affected.

Initialization note: Vec<AtomicU64> cannot use vec![] macro since AtomicU64 is not Copy. Use:

#![allow(unused)]
fn main() {
(0..max_accounts).map(|_| AtomicU64::new(0)).collect::<Vec<_>>()
}

Snapshot isolation

The Snapshotter owns its Vec<AtomicU64> exclusively as single writer. Consistent point-in-time snapshots are achieved via a checkpoint mechanism:

#![allow(unused)]
fn main() {
struct Balance {
    snapshot:      AtomicU64,   // always current, external reads
    checkpoint:    AtomicU64,   // frozen at checkpoint time, disk writes
    checkpoint_id: AtomicU8,    // which checkpoint this balance belongs to
}

// Global state
checkpoint_requested: AtomicBool  // SnapshotStorer requests
checkpoint_mode:      AtomicBool  // SnapshotRunner controls
current_checkpoint:   AtomicU8    // SnapshotRunner increments
checkpoint_tx_id:     AtomicU64   // set at freeze moment
checkpoint_wal_id:    AtomicU64   // set at freeze moment
}

Checkpoint sequence:

1. SnapshotStorer sets checkpoint_requested = true

2. SnapshotRunner catches request at transaction boundary
   (pending_entries == 0, clean state)
   → stores checkpoint_tx_id, checkpoint_wal_id
   → increments current_checkpoint (N → N+1)
   → sets checkpoint_mode = ON
   → clears checkpoint_requested = false

3. While checkpoint_mode = ON:
   SnapshotRunner updates balance.snapshot normally
   SnapshotRunner does NOT update balance.checkpoint
   New entries written with checkpoint_id = N+1

4. SnapshotStorer iterates Vec — reads balance.checkpoint
   All values consistent at checkpoint_tx_id (still at N)
   Writes to snapshot.bin

5. SnapshotStorer sets checkpoint_mode = OFF

6. SnapshotStorer iterates Vec again
   For each balance where checkpoint_id == N+1:
     CAS(balance.checkpoint, old_frozen_value, balance.snapshot.load())
     CAS ensures no accidental overwrite if snapshot updated mid-iteration

7. Done — all balances caught up, ready for next checkpoint

With Vec, steps 4 and 6 are sequential scans of a contiguous array — maximally cache-friendly, extremely fast even at 100M accounts.

Memory comparison

Structure              10M accounts    50M accounts    100M accounts
───────────────────────────────────────────────────────────────────
HashMap<u64,u64>       ~480MB          ~2.4GB          ~4.8GB
evmap (2 copies)       ~960MB          ~4.8GB          ~9.6GB
SkipMap                ~600MB          ~3.0GB          ~6.0GB
Vec<u64>               80MB            400MB           800MB
Vec<AtomicU64>         80MB            400MB           800MB
Both Vecs combined     160MB           800MB           1.6GB

6x less memory than evmap. Zero fragmentation. Zero allocator overhead.

Consequences

Positive

• O(1) balance read and write — single array index, no hashing, no traversal
• CPU prefetcher works perfectly — contiguous memory, sequential access pattern
• 6x memory reduction vs evmap at all scales
• Zero rehashing stalls — single pre-allocation at startup
• Performance flat across all account counts — no crossover degradation point
• Snapshot iteration is a sequential array scan — fast and consistent at any scale
• Simpler code — vec[id] replaces all map lookup logic

Negative

• Sequential account ID constraint — IDs must be assigned monotonically, gaps not allowed
• max_accounts must be configured upfront — operations beyond limit are rejected
• Memory reserved upfront for full capacity — at 100M accounts: 1.6GB at startup regardless of actual account count
• Resize requires reallocation of backing array

Neutral

• Account ID assignment is the ledger’s responsibility — external systems cannot use arbitrary IDs
• Sparse workloads are not supported — by design, not a limitation

Alternatives considered

evmap Evaluated and rejected as primary store — degrades at 10M+ accounts despite theoretical O(1) advantage. Benchmarks confirmed 15-27% TPS regression vs SkipMap at large scale due to refresh synchronization overhead and two-copy memory layout.

HashMap<u64, u64> Rejected — hash computation overhead, non-contiguous memory, 6x memory overhead vs Vec.

SkipMap (crossbeam) Rejected — O(log n) traversal, pointer chasing, designed for MPMC which single-writer pipeline does not need.

DashMap Rejected — sharded RwLock contention, MPMC overhead not needed for single-writer pipeline.

References

• ADR-001 — entries-based execution model, pipeline stage ownership
• CPU cache line optimization — Intel optimization reference manual
• Mechanical sympathy — contiguous memory layout for maximum hardware utilization

ADR-003: Ledger API — Removal of Generics and Operation Model

Status: Accepted
Date: 2026-03-28
Author: Taleh Ibrahimli

Context

The original Ledger<Data, BalanceData> design used Rust generics to allow user-defined transaction types. After the entries-based refactor (ADR-001), this design has several problems:

1. Generics no longer serve a purpose

After ADR-001, the Transactor executes business logic once and produces Credit/Debit entries. The WAL stores entries, not raw transactions. The Snapshotter applies entries. None of the pipeline stages need to know the user’s transaction type after the Transactor. The generic parameter leaked into every stage unnecessarily.

2. Wallet was a confusing abstraction

Wallet was introduced as a concrete example wrapping Ledger with three operations: deposit, withdraw, transfer. After the API simplification, Wallet becomes redundant — the Ledger itself can expose these operations directly. Having two separate types for the same thing confused users and complicated the codebase.

3. Custom TransactionDataType trait was the wrong extensibility model

The trait required users to implement Rust code compiled into the binary. This is incompatible with the WASM sandbox direction (ADR-008) where custom logic is uploaded at runtime, not compiled in.

4. No way to submit user-defined multi-step atomic operations

Simple operations (deposit, withdrawal, transfer) cover common cases but financial systems frequently need multi-step atomic operations — fee deductions, multi-leg settlements, conditional transfers. These require user-defined steps with declarative constraints evaluated atomically.

Decision

Remove all generics from Ledger. Replace TransactionDataType trait with a concrete Operation enum. Remove Wallet entirely. Add Complex variant for user-defined multi-step atomic operations.

Operation enum

#![allow(unused)]
fn main() {
pub enum Operation {
    Transfer {
        from:     u64,
        to:       u64,
        amount:   u64,
        user_ref: u64,
    },
    Deposit {
        account:  u64,
        amount:   u64,
        user_ref: u64,
    },
    Withdrawal {
        account:  u64,
        amount:   u64,
        user_ref: u64,
    },
    Complex(Box<ComplexOperation>),
    // Custom(Vec<u8>) — reserved for WASM, ADR-008, not yet implemented
}
}

Simple operations are stack-allocated, zero heap allocation. Complex uses Box<ComplexOperation> to keep the enum small — pointer-sized regardless of complex operation size.

ComplexOperation

#![allow(unused)]
fn main() {
pub struct ComplexOperation {
    pub steps:    SmallVec<[Step; 8]>,      // inline up to 8 steps, heap beyond
    pub flags:    ComplexOperationFlags,    // u64 bitfield, controls checks
    pub user_ref: u64,
}

pub enum Step {
    Credit { account_id: u64, amount: u64 },
    Debit  { account_id: u64, amount: u64 },
}

bitflags::bitflags! {
    pub struct ComplexOperationFlags: u64 {
        // Reject entire operation if any account balance goes negative after execution
        const CHECK_NEGATIVE_BALANCE = 0b00000001;
        // 1–63 reserved for future checks
    }
}
}

Why flags instead of Vec:

A Vec<Check> would require heap allocation per complex operation. Checks are a small, enumerable set — a u64 bitfield covers all realistic checks with zero allocation. Single bitwise AND to evaluate any check — one instruction, branch predictor friendly.

Why SmallVec<[Step; 8]> instead of Vec<Step>:

Vec<Step> always allocates on the heap. SmallVec<[Step; 8]> stores up to 8 steps inline on the stack — zero heap allocation for the common case. 8 covers fee deduction (3-4 steps), multi-leg settlement (4-6 steps), and most real-world complex operations. A 9+ step atomic operation is rare and can afford a heap allocation.

Allocation analysis:

Operation::Transfer/Deposit/Withdrawal  → zero allocation, stack only
Operation::Complex with ≤8 steps       → 1 allocation (Box)
Operation::Complex with >8 steps       → 2 allocations (Box + SmallVec spill)

Execution model

Complex operations execute atomically in the Transactor:

1. Apply all steps to balance cache (tentative)
2. Evaluate flags:
   CHECK_NEGATIVE_BALANCE → if any touched account balance < 0, rollback
3. Verify zero-sum invariant — always enforced regardless of flags
   sum(credits) == sum(debits), else FailReason::ZERO_SUM_VIOLATION
4. If all checks pass → commit entries to WAL
5. If any check fails → rollback all steps, emit TxMetadata with fail_reason

Atomicity is guaranteed — either all steps commit or none do. Partial execution is impossible.

Operation vs Transaction distinction

Operation is what the client submits — the intent. Transaction is what the server creates internally — the immutable fact stored in the WAL.

Client submits  → Operation   (intent)
Server creates  → Transaction (fact, tx_id + TxMetadata + Vec<TxEntry>)
Client receives → tx_id       (reference to the fact)

This maps to ISO 20022 — PaymentInstruction (intent) vs settled Transaction (fact).

Built-in operation entry mapping

Deposit { account, amount }:
  Debit(account, amount)
  Credit(SYSTEM_SOURCE, amount)
  Precondition: none — deposits always succeed

Withdrawal { account, amount }:
  Credit(account, amount)
  Debit(SYSTEM_SINK, amount)
  Precondition: balance(account) >= amount → FailReason::INSUFFICIENT_FUNDS

Transfer { from, to, amount }:
  Credit(from, amount)
  Debit(to, amount)
  Precondition: balance(from) >= amount → FailReason::INSUFFICIENT_FUNDS
  Precondition: from != to → no-op if equal

Ledger public API

#![allow(unused)]
fn main() {
pub struct Ledger { /* no generics */ }

impl Ledger {
    pub fn new(config: LedgerConfig) -> Self
    pub fn start(&mut self)
    pub fn submit(&self, operation: Operation) -> u64
    pub fn get_balance(&self, account_id: u64) -> Balance
    pub fn get_transaction_status(&self, tx_id: u64) -> TransactionStatus
    pub fn wait_for_transaction(&mut self, tx_id: u64)
    pub fn last_compute_id(&self) -> u64
    pub fn last_commit_id(&self) -> u64
    pub fn last_snapshot_id(&self) -> u64
}
}

Example usage

#![allow(unused)]
fn main() {
// simple operations
ledger.submit(Operation::Deposit    { account: 1, amount: 1000, user_ref: 0 });
ledger.submit(Operation::Withdrawal { account: 1, amount: 500,  user_ref: 0 });
ledger.submit(Operation::Transfer   { from: 1, to: 2, amount: 200, user_ref: 0 });

// complex — fee deduction: user pays amount + fee, merchant gets amount, fee_account gets fee
ledger.submit(Operation::Complex(Box::new(ComplexOperation {
    steps: smallvec![
        Step::Credit { account_id: user,        amount: amount + fee },
        Step::Debit  { account_id: merchant,    amount: amount },
        Step::Debit  { account_id: fee_account, amount: fee },
    ],
    flags: ComplexOperationFlags::CHECK_NEGATIVE_BALANCE,
    user_ref: order_id,
})));
}

Consequences

Positive

• Public API is simple — no generics, no trait implementations required
• Operation enum is self-documenting — exhaustive, immediately understandable
• Wallet removed — one less type, no confusion
• Complex enables multi-step atomic operations without WASM
• Flags as u64 bitfield — zero allocation, single instruction evaluation
• SmallVec<[Step; 8]> — zero allocation for common case (≤8 steps)
• WASM extensibility path preserved — Custom variant reserved, no breaking change needed
• ISO 20022 alignment — Operation (intent) vs Transaction (fact)

Negative

• Breaking API change — existing Ledger<Data, BalanceData> code must be rewritten
• TransactionDataType trait removed — users with custom logic must wait for WASM (ADR-008)
• Complex with >8 steps causes SmallVec heap spill — acceptable, rare in practice
• Box<ComplexOperation> — one heap allocation per complex operation

Neutral

• Internal pipeline unchanged — Transactor produces Vec<TxEntry>, WAL stores entries
• Balance type remains i64 — supports negative balances for liability accounts

Alternatives considered

Keep generics, add Operation enum alongside Rejected — two parallel extensibility models. Generic model incompatible with WASM direction.

Keep Wallet as convenience wrapper Rejected — adds no value after simplification. Ledger with Operation is already as simple.

Vec instead of flags Rejected — heap allocation per complex operation. Checks are a small enumerable set — u64 bitfield covers all realistic checks with zero allocation.

SmallVec<[Step; 16]> instead of 8 Rejected — would make ComplexOperation much larger. 8 covers all common financial operations. A 9+ step transaction is rare and can afford heap spill.

No Box — put ComplexOperation inline in enum Rejected — would make every Operation the size of ComplexOperation. Simple operations (95% of traffic) would pay the memory cost of complex operations.

References

• ADR-001 — entries-based execution model
• ADR-002 — Vec-based balance storage
• ADR-008 — WASM sandbox (planned)
• ISO 20022 — PaymentInstruction vs Transaction terminology
• TigerBeetle — single Transfer primitive, no custom logic
• smallvec crate — stack-allocated Vec with heap fallback
• bitflags crate — type-safe bitfield flags

ADR-004: gRPC External Interface

Status: Accepted
Date: 2026-03-29
Author: Taleh Ibrahimli

Context

roda-ledger currently exposes a custom binary TCP protocol (src/protocol/, src/server/, src/client/) implemented over raw Tokio TCP streams. The protocol uses fixed-size bytemuck-compatible structs (ProtocolHeader, RegisterTransactionResponse, etc.) with a serde_json payload for Operation.

This works for the benchmark harness (wallet_server_bench) and internal testing, but blocks external adoption:

1. Protocol is opaque
There is no machine-readable API contract. Clients must read Rust source to understand the wire format. OperationKind::REGISTER_TRANSACTION, BATCH, GET_STATUS, GET_BALANCE are defined only in src/protocol/definitions.rs. There is no schema file.

2. Cross-language usage requires reimplementing the protocol
The binary framing (8-byte ProtocolHeader + variable payload) and serde_json-encoded Operation body must be re-implemented per client language. No tooling exists to generate this.

3. Blocks Docker adoption
A Docker image without a language-agnostic interface requires users to write Rust or reverse-engineer the binary protocol. The planned docker run tislib/roda-ledger is not usable without this.

4. No standard tooling
The current protocol has no equivalent of grpcurl, grpcui, or Postman support. Testing requires writing a client from scratch.

Decision

Replace the existing TCP binary protocol with gRPC as the sole external interface for the standalone server mode. The TCP binary protocol (src/protocol/, src/server/, src/client/) is removed entirely.

roda-ledger continues to support two usage modes:

Library mode — embed roda-ledger as a Rust crate. Interact directly with Ledger via the Operation API. Zero network overhead, zero serialization cost. This is the mode used by benchmarks, stress tests, and Rust applications embedding the ledger.

Server mode — run roda-ledger as a standalone Docker container. Interact via gRPC over HTTP/2. This is the mode for cross-language clients, Docker deployments, and external users who do not write Rust.

Both modes are first-class. The library API (src/ledger.rs, src/transaction.rs) is unchanged. gRPC is a thin translation layer on top of it.

All RPCs are unary

Every RPC in this API is unary — one request, one response. No server-side streaming, no client-side streaming, no bidirectional streaming.

Why not streaming:

gRPC streaming operates per-message: each message sent or received triggers a Tokio wakeup. At the throughput roda-ledger targets, streaming would produce millions of Tokio wakeups per second on the gRPC layer alone — before any pipeline work is done. The Tokio async runtime is not designed to handle millions of independent message events as a transport mechanism. This overhead would make the gRPC layer the bottleneck, not the pipeline.

Batching solves the same problem correctly: the client groups N operations into one SubmitBatchRequest, pays one round-trip, one proto deserialization, one Tokio task wakeup, and receives N results in one SubmitBatchResponse. Throughput scales with batch size, which the client controls. No streaming state machine, no flow control windows to tune, no wakeup-per-message.

Interface Design

The gRPC interface is designed for high-throughput financial operations while maintaining a simple mental model for clients.

Key Principles

1. Unary Only: All RPCs are unary. No streaming is used to avoid Tokio wakeup overhead and complex flow control at the gRPC layer.
2. Client-Side Batching: High throughput is achieved by grouping multiple operations or IDs into a single request. This minimizes network round-trips and deserialization costs.
3. Monotonic Transaction IDs: Every successful submission receives a transaction_id which defines the global order of operations.
4. Atomicity and Ordering:
   • SubmitOperation: Each operation is atomic and strictly serialized.
   • SubmitBatch: Operations are executed in the order provided in the request. However, the batch itself is not atomic as a whole (failures are independent), and there is no guarantee of global contiguous order — operations from other clients may be interleaved between them.
5. Explicit Pipeline Stages: Clients can poll for the status of a transaction through strictly ordered stages: PENDING -> COMPUTED -> COMMITTED -> ON_SNAPSHOT.
6. Zero-Sum Invariant: All operations, including composite ones, must maintain a zero-sum balance across the system.
7. Isolation Guarantees:

Core Operations

The interface supports:

• Deposit/Withdrawal: Moving funds in/out of the system.
• Transfer: Atomic movement between accounts.
• Composite: User-defined atomic multi-step operations with optional validation flags (e.g., overdraft protection).
• Named: Invoke a registered operation by name with positional repeated int64 params. Ties to WASM roadmap — no breaking change when ADR-008 lands.
• Queries: Account balances (returning last_snapshot_tx_id to indicate currency/freshness) and transaction status polling.

The full protobuf definition is maintained in proto/ledger.proto.

Deployment Modes

Library mode (unchanged)

#![allow(unused)]
fn main() {
// Cargo.toml
roda-ledger = { git = "https://github.com/tislib/roda-ledger" }

// Usage — zero network, zero serialization overhead
let mut ledger = Ledger::new(LedgerConfig::default());
ledger.start();
let tx_id = ledger.submit(Operation::Deposit { account: 1, amount: 1000, user_ref: 0 });
}

The grpc feature is opt-in. Library users pull zero gRPC dependencies and do not require protoc in their build environment.

Server mode (Docker)

docker run -p 50051:50051 -v ./data:/data tislib/roda-ledger

# Health check
grpcurl -plaintext localhost:50051 roda.ledger.v1.Ledger/GetPipelineIndex

# Submit a deposit
grpcurl -plaintext -d '{"deposit": {"account": "1", "amount": "1000", "user_ref": "0"}}' \
  localhost:50051 roda.ledger.v1.Ledger/SubmitOperation

# Submit a batch
grpcurl -plaintext -d '{"operations": [
  {"deposit":  {"account": "1", "amount": "500",  "user_ref": "1"}},
  {"transfer": {"from":    "1", "to":     "2",    "amount": "200", "user_ref": "2"}}
]}' localhost:50051 roda.ledger.v1.Ledger/SubmitBatch

Configuration via environment variables:

RODA_GRPC_ADDR           default: 0.0.0.0:50051
RODA_DATA_DIR            default: /data
RODA_MAX_ACCOUNTS        default: 1000000
RODA_SNAPSHOT_INTERVAL   default: 600 (seconds)
RODA_IN_MEMORY           default: false

Removal of TCP Binary Protocol

src/protocol/, src/server/, src/client/ are removed entirely. wallet_server_bench is removed — it measured TCP throughput with a network layer, which is not a meaningful pipeline ceiling. The direct library benchmark wallet_bench measures pipeline throughput correctly with zero network overhead.

The TCP binary protocol served two purposes — external interface and benchmark harness. gRPC replaces the external interface. The library API is the correct benchmark harness.

Server Structure

src/
  grpc/
    mod.rs       — feature gate, pub mod
    server.rs    — GrpcServer struct, tonic::transport::Server setup, env config
    handler.rs   — LedgerService impl (generated tonic trait)
    mapping.rs   — proto ↔ Operation, proto ↔ TransactionStatus conversion
  bin/
    server.rs    — main() for Docker binary: reads env vars, starts GrpcServer
proto/
  ledger.proto
build.rs         — tonic_build::compile_protos("proto/ledger.proto")

GrpcServer wraps Arc<Ledger>. The handler maps proto messages to Operation via mapping.rs and calls ledger.submit(). No new pipeline stage. No changes to Ledger, Transactor, Wal, or Snapshot.

Crate Additions

[dependencies]
tonic = { version = "0.12", features = ["transport"], optional = true }
prost = { version = "0.13", optional = true }

[features]
default = []
grpc    = ["dep:tonic", "dep:prost"]

[build-dependencies]
tonic-build = { version = "0.12", optional = true }

Consequences

Positive

• Proto file is the API contract — self-documenting, version-controlled, language-agnostic
• Auto-generated clients in Go, Python, Java, TypeScript with no manual protocol work
• grpcurl works immediately after docker run
• All RPCs unary — one Tokio task per call, no streaming wakeup overhead
• Client controls batch size via SubmitBatch and GetBalances — throughput scales naturally
• GetTransactionStatuses enables polling a bulk load without N individual round trips
• GetBalance and GetBalances return last_snapshot_tx_id, allowing clients to track exactly what transactions have been applied to the balance.
• GetPipelineIndex enables lag monitoring and health checks without per-transaction queries
• Library mode unchanged — zero overhead added for Rust users
• TCP protocol removal simplifies codebase — three source directories deleted
• grpc feature opt-in — zero dependency cost for library users

Negative

• Protobuf serialization overhead vs direct library calls — expected, this is the external interface
• protoc required in Docker build image
• HTTP/2 framing overhead vs raw TCP — acceptable for an external interface
• No push notifications — callers poll GetTransactionStatuses for finality

Neutral

• wallet_server_bench removed — wallet_bench already covers pipeline throughput correctly
• FailReason encoded as uint32 in proto (u8 in Rust) — proto3 has no uint8, no data loss

Alternatives Considered

gRPC streaming for SubmitBatch
Evaluated. Rejected — streaming operates per-message, meaning one Tokio wakeup per operation. At millions of transactions per second this makes the gRPC layer the bottleneck. Unary with repeated gives the client control over batch size and reduces wakeups to one per batch.

Keep TCP binary protocol alongside gRPC
Rejected — the binary protocol existed as external interface and benchmark harness. gRPC replaces the external interface. The library API replaces the benchmark harness. Keeping both adds maintenance overhead with no benefit.

REST/HTTP JSON API
Rejected — no schema enforcement, no generated clients, more boilerplate for equivalent batch semantics. No path to streaming-equivalent patterns if needed in future.

References

• ADR-001 — entries-based execution model
• ADR-003 — Operation enum, Composite, CompositeOperationFlags (u64 bitfield)
• src/ledger.rs — Ledger::last_compute_id(), last_commit_id(), last_snapshot_id()
• src/transaction.rs — Operation, TransactionStatus, CompositeOperationFlags
• src/entities.rs — FailReason codes
• tonic — https://github.com/hyperium/tonic

ADR-005: PressureManager — Adaptive Pipeline Execution Mode

Status: Postponed (due to existing problems)
Date: 2026-03-29
Author: Taleh Ibrahimli

Context

Every pipeline stage — Transactor, WAL Storer, Snapshotter — runs in a dedicated thread and polls its inbound ArrayQueue in a tight loop. When the queue is empty the thread calls yield_now() and immediately polls again. This is a busy-wait loop.

The result: all pipeline threads consume 100% CPU at all times, regardless of whether there is any work to do. On a machine running other workloads, or during idle periods between transaction bursts, this wastes CPU resources with no benefit.

The problem is structural — there is no signal between stages about whether work exists. Each stage makes its own decision independently, and the only decision available is “yield and try again immediately.”

Decision

Introduce a shared PressureManager that observes work and empty signals from all pipeline stages and owns the decision of how each thread executes: spin, yield, or park.

Responsibility split

Each stage reports to PressureManager on every loop iteration. The stage has no awareness of its execution mode — it only reports what it found (work or empty). PressureManager owns the state machine and decides what the calling thread does next.

This keeps stage code simple and ensures execution mode decisions are made in one place.

Two pressure signals, opposite directions

The pipeline has two observable pressure signals:

BackPressure:  Sequencer → Transactor → WAL → Snapshotter
               Sequencer spinning to push → downstream is slow, pipeline is full

Pressure drop: Snapshotter → WAL → Transactor → Sequencer
               Transactor inbound empty → upstream has no work, pipeline is dry

Both signals are captured implicitly by on_work() and on_empty() calls from each stage. No explicit directional signal is needed — the global work/empty streak captures both.

Global pressure level

Pressure is a single global state shared across all pipeline stages. All stages read and respond to the same level. This is correct because:

Every transaction passes through every stage exactly once. When there is work, all stages have work. When there is no work, no stage has work. Pressure is a global property of the pipeline, not a per-stage property. Per-stage pressure levels would desynchronise transitions and cause some stages to park while others spin with nothing to process.

Three execution states

Spin   — queue has work, process at full speed, no OS involvement
Yield  — queue borderline, give up timeslice but stay hot
Park   — queue dry, thread sleeps until unparked

State transitions:

              work crosses threshold
  ┌──────────────────────────────────────────────────────┐
  │                                                      ▼
Idle                Low  ◄──────────────────────────── High
  ▲                  │      empty crosses HIGH_TO_LOW    │
  │                  │                                   │
  └──────────────────┘                                   │
   empty crosses LOW_TO_IDLE              (stays High while work present)

New work always transitions directly to High regardless of current level — no ramp-up. The Sequencer calls on_work() before pushing the first transaction into the queue, so threads are unparked and spinning before the Transactor sees the first item. Zero wakeup lag on the hot path.

Sequencer is the entry point

The Sequencer calls on_work() on every submit() before pushing to the queue. This ensures the pipeline is awake before work arrives, not after. The Sequencer is the natural owner of this signal — it is the only stage that receives external input.

WaitStrategy — user-facing configuration

Threshold values are internal to PressureManager. Users do not configure raw numbers. Instead they choose a WaitStrategy that expresses intent:

LowLatency   Threads spin continuously. Minimum latency, maximum CPU.
             Suitable for dedicated hardware where roda-ledger owns all cores.

Balanced     Adaptive spin → yield → park. Balances latency and CPU.
             Default. Suitable for most deployments.

LowCpu       Parks quickly when idle. Minimum CPU, slightly higher wakeup latency
             on burst start. Suitable for shared hardware or background ledgers.

The user answers “what matters more — latency or CPU?” The thresholds behind each mode are owned by PressureManager and are not exposed.

Approximate threshold behaviour per mode:

Mode          Spin duration    Yield duration    Park
────────────────────────────────────────────────────────────────
LowLatency    indefinite       never             never
Balanced      short            moderate          after sustained idle
LowCpu        minimal          short             quickly on idle

LowLatency preserves current behaviour exactly — no regression for users who want maximum throughput on dedicated hardware.

WaitStrategy is set in LedgerConfig. Default is Balanced.

Thread registration

Each stage registers its thread handle with PressureManager at startup. PressureManager holds these handles to issue unpark() calls when transitioning out of Idle. All stage threads must be registered before the first transaction is submitted.

Asymmetric transition speed

Transitions to higher pressure (more work) are fast. Transitions to lower pressure (less work) are slow. This prevents thrashing on bursty workloads where a short idle gap between bursts would otherwise cause unnecessary park/unpark cycles.

The specific thresholds per mode are determined during implementation and validated against stress test scenarios before being finalised.

Behaviour at low throughput (e.g. 10 req/s)

At 10 req/s the inter-arrival gap is ~100ms. The pipeline will naturally cycle:

Request arrives → Sequencer on_work() → unpark all → High
Process transaction → queue empty → empty streak climbs → Low → Idle → park
~100ms later → next request → repeat

Park/unpark wakeup latency is 1–10µs on Linux. At 100ms inter-arrival gaps this is negligible. LowLatency mode avoids park entirely for deployments where even 10µs wakeup latency is unacceptable. LowCpu mode parks aggressively, spending the 99.99% idle time at ~0% CPU. Balanced falls between the two.

Consequences

Positive

• Idle CPU drops from ~100% to ~0% in Balanced and LowCpu modes
• Wakeup latency under sustained load is zero — threads never park while High
• Single shared struct, single atomic level — minimal coordination overhead
• Each stage is simplified — no spin/yield/park logic per stage
• User-facing configuration is three named modes, no internal knowledge required
• LowLatency preserves current behaviour — no regression for existing deployments
• Sequencer-driven wakeup — new work wakes all threads before entering the queue

Negative

• Thread registration at startup adds a small initialisation dependency — all stage threads must be spawned and registered before the first submit() call
• Shared atomics written by all stages on every loop iteration may cause false sharing at very high throughput — to be verified with benchmarks, mitigated with cache line padding if needed
• thread::park() wakeup latency is OS-dependent (~1–10µs on Linux) — only relevant at Idle → High transitions, which only occur after genuine idle periods
• Threshold values behind each WaitStrategy require tuning and stress test validation before being finalised

Neutral

• Under sustained load, all modes except LowLatency behave identically to LowLatency — threads stay in High and spin continuously
• Existing benchmarks (wallet_bench, grpc_bench) are unaffected — they run under sustained load where PressureManager stays at High

Alternatives Considered

Per-stage adaptive spinning
Each stage manages its own spin/yield/park threshold independently. Rejected — pressure is a global pipeline property. Per-stage thresholds desynchronise transitions and produce stages spinning with nothing to process while other stages are parked.

Blocking crossbeam channels
Replace ArrayQueue with blocking crossbeam::channel. Each stage blocks on recv(). Rejected — changes the queue type across all stages, loses the bounded backpressure guarantee of ArrayQueue, and moves blocking decisions into the channel rather than a single owned place.

thread::sleep with fixed interval
Sleep a fixed duration (e.g. 1ms) when the queue is empty. Rejected — adds fixed minimum wakeup latency on every idle period. Unacceptable for a latency-sensitive financial ledger.

Condvar
Mutex + Condvar for blocking. Rejected — requires acquiring a mutex on every wakeup. park/unpark is the correct primitive when a known set of threads is the target and no shared state needs to be guarded during the wakeup.

Exposing raw thresholds in config
Allow users to configure high_to_low, low_to_idle directly. Rejected — these values are meaningless without understanding the pipeline internals. WaitStrategy expresses the same intent in terms users can reason about.

References

• ADR-001 — entries-based execution model, pipeline stage ownership
• ADR-002 — Vec-based balance storage, single-writer Transactor
• src/transactor.rs — current busy-wait loop
• src/wal.rs — current busy-wait loop
• src/snapshot.rs — current busy-wait loop
• src/sequencer.rs — entry point, owns submit path
• Linux NAPI — same pattern: spin under load, park when idle, wake on interrupt
• std::thread::park / std::thread::unpark — chosen primitive for blocking

ADR-006: WAL, Snapshot, and Seal Durability

Status: Accepted Date: 2026-03-31 Updated: 2026-04-03 Author: Taleh Ibrahimli

Context

The current WAL implementation is a single append-only file that grows indefinitely. The snapshot mechanism writes a full balance dump on a timer. Both have correctness and operational problems that must be addressed before a public Docker release.

Current WAL problems:

• Single file grows indefinitely — recovery must scan the entire file on restart
• No file-level integrity check — silent corruption goes undetected
• No per-transaction CRC — a flipped bit in a TxAmount produces wrong balances silently
• No format versioning — format changes have no detection mechanism
• No segment boundary markers — replay has no structural anchors

Current snapshot problems:

• Timer-based trigger — arbitrary relationship between snapshot and WAL position
• Full balance dump every time — at 50M accounts this is 400MB uncompressed
• No integrity check — corrupt snapshot loads silently, produces wrong balances
• Single file — no history, no ability to recover from a bad snapshot write
• Atomic write (tmp → rename) already implemented — good, kept as-is

Financial ledger requirement:

A financial ledger must refuse to operate on corrupt data. Silent corruption is worse than a crash. Every read from disk must be verified before being applied to in-memory state.

Decision

Pipeline: four stages plus background sealer

The processing pipeline has four queue-connected stages in order:

Sequencer → Transactor → WAL → Snapshot

Each stage runs in its own thread, connected by ArrayQueue<WalEntry> queues. The Seal process runs as a separate background thread that polls the filesystem for unsealed segments — it is not connected to any queue and does not receive entries from the pipeline.

WAL: segmented files with two-level integrity

Replace the single WAL file with fixed-size segments. Each segment has a header record, per-transaction CRC, and a sealed record. A sidecar .crc file provides file-level integrity for sealed segments.

Snapshots: retained, stored as part of Seal stage

Snapshots use a new format with a 36-byte header, LZ4 compression, and two-level CRC (data CRC in header, file CRC in sidecar). Snapshot storing is the responsibility of the Seal thread. Snapshots are written at a configurable frequency — every Nth sealed segment (default: every 4th segment).

Balances: computed once in Transactor, stored in TxEntry

Balances are not computed dynamically during sealing or snapshotting. Instead, the Transactor computes the running balance for each account after every entry and stores it directly in the TxEntry record:

#![allow(unused)]
fn main() {
pub struct TxEntry {
    // ...
    pub computed_balance: i64,  // replaces _pad1 — set by Transactor, read by Seal/Snapshot
}
}

The computed_balance field holds the account balance after this entry is applied. Snapshot and Seal stages read TxEntry.computed_balance directly — no balance recomputation is needed anywhere downstream.

Seal thread: background poller, discovers work by listing

The Seal thread runs independently of the pipeline. It periodically scans the data directory for segments that are CLOSED but not yet SEALED (i.e., wal_NNNNNN.bin exists but no wal_NNNNNN.seal marker). It loads the segment, computes CRC, writes the .crc sidecar and .seal marker, then conditionally writes a snapshot.

Storage module: separated concerns

All storage I/O is consolidated into a src/storage/ module with separated concerns:

• layout.rs — file path conventions and segment ID parsing
• segment.rs — segment lifecycle (ACTIVE → CLOSED → SEALED)
• snapshot.rs — snapshot serialization/deserialization (impl on Segment)
• storage.rs — Storage facade, segment factory, directory enumeration
• wal_reader.rs — WAL binary loading and CRC verification
• wal_serializer.rs — zero-copy record codec via bytemuck

Communication: WAL forwards entries downstream; Seal polls independently

The WAL stage writes entries to the active segment and forwards them downstream to the Snapshot stage via ArrayQueue. When a segment exceeds the size threshold, the WAL appends a SegmentSealed record, closes the segment (rename), and opens a new active segment. The Seal thread discovers closed segments by polling the filesystem.

WAL Design

Segmented files

The WAL is divided into fixed-size segment files. All files live flat in the data directory — no subdirectories:

data/
  wal_000001.bin          ← sealed segment
  wal_000001.crc          ← file-level CRC sidecar
  wal_000001.seal         ← seal marker (empty file)
  wal_000002.bin
  wal_000002.crc
  wal_000002.seal
  wal_000003.bin          ← closed but not yet sealed
  wal_000003.crc
  wal.bin                 ← current active segment
  snapshot_000004.bin     ← snapshot (if taken at segment 4)
  snapshot_000004.crc     ← snapshot integrity sidecar

The active segment is wal.bin — no numeric suffix until closed. When the WAL closes a segment, it is renamed to wal_NNNNNN.bin (CLOSED state). The Seal thread then writes .crc and .seal sidecars to transition it to SEALED state.

Segment lifecycle: three states

ACTIVE → CLOSED → SEALED

State is determined by filesystem inspection:

• wal_NNNNNN.seal exists → SEALED
• wal_NNNNNN.bin exists without .seal → CLOSED
• wal.bin → ACTIVE

Segment size

Default: 64MB. Configurable via StorageConfig.wal_segment_size_mb.

At 40 bytes per record and 5M records/s, a new segment is created approximately every 340ms at peak load. At 100K records/s (sustained), approximately every 17 seconds.

Segment ID assignment

The active segment ID is last_segment_id + 1, where last_segment_id is the highest numeric suffix found among wal_NNNNNN.bin files in the data directory. If no segments exist, the first active segment gets ID 1.

Four WalEntry variants

All WAL records are exactly 40 bytes with entry_type: u8 as the first byte, enabling self-describing record scanning. Zero-copy serialization via bytemuck (Pod + Zeroable traits).

#![allow(unused)]
fn main() {
pub enum WalEntry {
    Metadata(TxMetadata),           // entry_type = 0
    Entry(TxEntry),                 // entry_type = 1
    SegmentHeader(SegmentHeader),   // entry_type = 2
    SegmentSealed(SegmentSealed),   // entry_type = 3
}
}

Compile-time assertions enforce the 40-byte invariant:

#![allow(unused)]
fn main() {
const _: () = assert!(std::mem::size_of::<TxMetadata>() == 40);
const _: () = assert!(std::mem::size_of::<TxEntry>() == 40);
const _: () = assert!(std::mem::size_of::<SegmentHeader>() == 40);
const _: () = assert!(std::mem::size_of::<SegmentSealed>() == 40);
}

TxMetadata (40 bytes)

#![allow(unused)]
fn main() {
#[repr(C)]
pub struct TxMetadata {
    pub entry_type:   u8,           // 1 @ 0  — WalEntryKind::TxMetadata (0)
    pub entry_count:  u8,           // 1 @ 1  — 0 if transaction failed
    pub fail_reason:  FailReason,   // 1 @ 2  — FailReason::NONE if succeeded
    pub flags:        u8,           // 1 @ 3  — reserved flags
    pub crc32c:       u32,          // 4 @ 4  — CRC32C; zero this field when computing
    pub tx_id:        u64,          // 8 @ 8
    pub timestamp:    u64,          // 8 @ 16
    pub user_ref:     u64,          // 8 @ 24
    pub tag:          [u8; 8],      // 8 @ 32 — 8-byte metadata tag
}                                   // total: 40 bytes
}

SegmentHeader (40 bytes)

Written as the first record of every new segment:

#![allow(unused)]
fn main() {
#[repr(C)]
pub struct SegmentHeader {
    pub entry_type:   u8,           // 1 @ 0  — WalEntryKind::SegmentHeader (2)
    pub version:      u8,           // 1 @ 1  — WAL_VERSION (1)
    pub _pad0:        [u8; 2],      // 2 @ 2
    pub magic:        u32,          // 4 @ 4  — WAL_MAGIC (0x524F4441 = "RODA")
    pub segment_id:   u32,          // 4 @ 8  — monotonic, must match filename
    pub _pad1:        [u8; 4],      // 4 @ 12
    pub first_tx_id:  u64,          // 8 @ 16 — first transaction in this segment
    pub _pad2:        [u8; 16],     // 16 @ 24
}                                   // total: 40 bytes
}

Validates that the right file is being read. segment_id must match the numeric suffix in the filename.

SegmentSealed (40 bytes)

Written as the last record before segment closure:

#![allow(unused)]
fn main() {
#[repr(C)]
pub struct SegmentSealed {
    pub entry_type:    u8,          // 1 @ 0  — WalEntryKind::SegmentSealed (3)
    pub _pad0:         [u8; 3],     // 3 @ 1
    pub segment_id:    u32,         // 4 @ 4  — matches SegmentHeader.segment_id
    pub last_tx_id:    u64,         // 8 @ 8  — last committed transaction
    pub record_count:  u64,         // 8 @ 16 — total records (including header + sealed)
    pub _pad1:         [u8; 16],    // 16 @ 24
}                                   // total: 40 bytes
}

The record_count includes the sealed record itself (incremented by 1 in the constructor).

TxEntry (40 bytes)

#![allow(unused)]
fn main() {
#[repr(C)]
pub struct TxEntry {
    pub entry_type:        u8,          // 1 @ 0  — WalEntryKind::TxEntry (1)
    pub kind:              EntryKind,   // 1 @ 1  — Credit (0) or Debit (1)
    pub _pad0:             [u8; 6],     // 6 @ 2
    pub tx_id:             u64,         // 8 @ 8
    pub account_id:        u64,         // 8 @ 16
    pub amount:            u64,         // 8 @ 24 — integer minor units
    pub computed_balance:  i64,         // 8 @ 32 — set by Transactor
}                                       // total: 40 bytes
}

The Transactor sets computed_balance to the account’s balance after applying this entry. All downstream stages read this field directly.

Per-transaction CRC32C

CRC is computed in the Transactor — the only stage that has both TxMetadata and all TxEntry records simultaneously. CRC covers the complete logical transaction: TxMetadata fields (with crc32c field zeroed) + all TxEntry records.

The crc32c field in TxMetadata is set to zero during computation, then filled with the computed value.

Why CRC32C over CRC32: CRC32C (Castagnoli) is hardware-accelerated on all modern CPUs (SSE4.2 on x86, ARM CRC extension). Same cost as CRC32, better error detection. Standard for storage systems — PostgreSQL, TigerBeetle, RocksDB, iSCSI all use CRC32C.

crc32c = "0.6"   # uses hardware instruction when available, pure Rust fallback

File-level CRC sidecar

Each sealed segment has a companion .crc file:

wal_000004.crc:
  [crc32c: 4 bytes LE]   ← CRC32C of entire wal_000004.bin
  [size:   8 bytes LE]   ← expected file size in bytes
  [magic:  4 bytes LE]   ← WAL_MAGIC (0x524F4441)
                          = 16 bytes

Written by the Seal thread after loading the segment data and computing CRC. The .seal marker file (empty) is created after the .crc sidecar to signal that sealing is complete.

The size field catches truncation independently of CRC — if file size does not match, the file is corrupt even if CRC somehow passes.

Write buffer

The active segment uses a 16MB write buffer (Vec<u8>) to batch serialized entries before writing. Each append_entries call serializes all entries into the buffer, writes the buffer to the file, and calls fsync. The buffer is cleared between calls.

#![allow(unused)]
fn main() {
wal_buffer: vec![0; 16 * 1024 * 1024], // 16MB
}

If write_all or sync_data fails, the segment retries after a 1-second sleep until the I/O succeeds.

Zero-copy serialization

WAL records are serialized using bytemuck::bytes_of() (zero-copy, returns borrowed slice) and deserialized using bytemuck::pod_read_unaligned(). The first byte of each 40-byte chunk is the entry_type discriminant, used to select the correct struct for deserialization.

#![allow(unused)]
fn main() {
pub fn serialize_wal_records(entry: &WalEntry) -> &[u8]   // zero-copy
pub fn parse_wal_record(data: &[u8]) -> Result<WalEntry>   // pod_read_unaligned
}

Snapshot Design

Snapshot stage (in-memory)

The Snapshot stage sits at the end of the pipeline, consuming entries from the WAL stage via ArrayQueue<WalEntry>. It maintains Arc<Vec<AtomicI64>> — the live in-memory balance cache. For every TxEntry it receives, it stores entry.computed_balance into balances[account_id] using Ordering::Release.

No arithmetic is performed in the Snapshot stage. The balance value comes directly from TxEntry.computed_balance. Balance reads use Ordering::Acquire.

Snapshot storing (part of Seal thread)

Writing snapshots to disk is the responsibility of the Seal thread, not the Snapshot stage. The Seal thread conditionally writes a snapshot based on StorageConfig.snapshot_frequency:

• snapshot_frequency = 0 — disabled, no snapshots written
• snapshot_frequency = N — snapshot written when segment_id % N == 0
• Default: snapshot_frequency = 4 (snapshot every 4th sealed segment)

The Seal thread maintains its own Vec<i64> balance cache, updated by replaying WAL records from each sealed segment. Snapshots are written from the Seal thread’s balance state, not from the Snapshot stage’s AtomicI64 vector.

Snapshot file format

New binary format with magic number, LZ4 compression, and two-level CRC:

36-byte header:

Offset  Size  Field            Value
0       4     magic            0x534E4150 ("SNAP")
4       1     version          1
5       4     segment_id       segment being snapshotted
9       8     checkpoint_id    = segment_id as u64
17      8     account_count    number of account records
25      1     compressed       1 (always compressed for new snapshots)
26      6     pad              reserved zeros
32      4     data_crc32c      CRC32C of raw (uncompressed) record data

Body (after header):

LZ4-compressed (using lz4_flex::compress_prepend_size). After decompression, each account record is 16 bytes:

[account_id: u64 LE][balance: i64 LE]   per account, 16 bytes

Sidecar (.crc):

snapshot_000004.crc:
  [crc32c: 4 bytes LE]   ← CRC32C of entire snapshot_000004.bin
  [size:   8 bytes LE]   ← expected file size
  [magic:  4 bytes LE]   ← SNAPSHOT_MAGIC (0x534E4150)
                          = 16 bytes

Two-level CRC:

• Data CRC (in header): CRC32C of raw uncompressed record data — catches decompression errors
• File CRC (in sidecar): CRC32C of entire snapshot file including header — catches on-disk corruption

Atomic writes via temp files

Snapshots use temp files for crash safety:

1. Write header + LZ4 body to snapshot_NNNNNN.bin.tmp
2. Compute file CRC, write sidecar to snapshot_NNNNNN.crc.tmp
3. Atomic rename snapshot_NNNNNN.bin.tmp → snapshot_NNNNNN.bin
4. Atomic rename snapshot_NNNNNN.crc.tmp → snapshot_NNNNNN.crc

Crash during write leaves only .tmp files — recovery ignores them.

Sparse storage

Only non-zero balances are saved. Accounts with zero balance are filtered out. Records are sorted by account ID for deterministic output.

Seal Thread Design

Responsibilities

The Seal thread is a dedicated background thread (not a pipeline stage) with two responsibilities:

1. WAL segment sealing — load segment, compute CRC, write .crc sidecar, write .seal marker
2. Conditional snapshot storing — collect balances from WAL replay, write snapshot_NNNNNN.bin + .crc at configured frequency

Polling loop

The Seal thread runs a loop:

loop:
  1. Call seal_pending_segments()
       → list_all_segments()
       → skip segments with status == SEALED
       → process_seal() for each non-sealed segment
  2. Check running flag → exit if false
  3. Sleep for seal_check_interval (configurable Duration)

Process seal sequence

For each non-sealed segment:

process_seal(segment):
  1. segment.load()         → read wal_NNNNNN.bin into memory
  2. segment.seal()         → compute CRC, write .crc sidecar, write .seal marker
  3. Update last_sealed_id  → AtomicU32::store(segment_id)
  4. segment.visit_wal_records() → replay entries, update balances Vec<i64>
  5. If segment_id % snapshot_frequency == 0:
       → filter non-zero balances
       → sort by account_id
       → segment.save_snapshot(records)

Segment.close() — called by WAL thread

WAL thread (segment full):
  1. Append WalEntry::SegmentSealed to active segment
  2. fsync active segment
  3. Rename wal.bin → wal_NNNNNN.bin          ← status becomes CLOSED
  4. Drop old segment handle
  5. Open new active segment (ID = last_segment_id + 1)
  6. Continue writing (SegmentHeader on first write)

Segment.seal() — called by Seal thread

Seal thread (on discovering CLOSED segment):
  1. Compute CRC32C of entire wal_NNNNNN.bin data
  2. Write wal_NNNNNN.crc (16 bytes: crc32c + size + magic)
  3. fsync wal_NNNNNN.crc
  4. Create empty wal_NNNNNN.seal marker file
  5. fsync wal_NNNNNN.seal                    ← status becomes SEALED

The WAL thread is never blocked by sealing I/O. Sealing happens asynchronously on the Seal thread.

Balance tracking

The Seal thread maintains its own Vec<i64> (sized to account_count). During process_seal, it replays all TxEntry records from the segment and updates balances[account_id] = entry.computed_balance. This provides the balance state needed for snapshot writing without depending on the Snapshot stage’s atomics.

Storage Module

Module structure

All storage I/O is organized under src/storage/:

src/storage/
  mod.rs              ← re-exports public API
  layout.rs           ← file path conventions, segment ID parsing
  segment.rs          ← Segment struct, lifecycle (ACTIVE/CLOSED/SEALED)
  snapshot.rs         ← snapshot save/load (impl on Segment)
  storage.rs          ← Storage facade, StorageConfig, segment factory
  wal_reader.rs       ← WAL binary loading, CRC verification
  wal_serializer.rs   ← zero-copy record codec (bytemuck)

StorageConfig

#![allow(unused)]
fn main() {
pub struct StorageConfig {
    pub data_dir: String,           // where to store files (default: "data/")
    pub temporary: bool,            // if true, delete data_dir on drop
    pub wal_segment_size_mb: u64,   // segment size threshold (default: 64)
    pub snapshot_frequency: u32,    // every Nth sealed segment (default: 4, 0=disabled)
}
}

Storage facade

Storage provides the entry point for segment operations:

• active_segment() — open or create the current wal.bin for writes
• segment(id) — open an existing segment by ID
• list_all_segments() — enumerate all wal_NNNNNN.bin segments, sorted by ID
• last_segment_id() — find the maximum segment ID (0 if none)

If temporary = true, the data directory is recursively deleted on Drop.

Segment struct

#![allow(unused)]
fn main() {
pub struct Segment {
    data_dir: String,
    segment_id: u32,
    loaded: bool,

    wal_file: Option<File>,     // file handle (None when unloaded)
    wal_data: Vec<u8>,          // raw binary data
    status: SegmentStaus,       // ACTIVE, CLOSED, or SEALED

    // ACTIVE segment only
    wal_buffer: Vec<u8>,        // 16 MB write buffer
    wal_position: usize,        // current file offset
    record_count: u64,          // count of appended records
}
}

Lazy loading: CLOSED/SEALED segments can be opened without reading data into memory. load() must be called explicitly before accessing WAL records.

State assertions: Operations panic if called in the wrong state (e.g., append_entries only works on ACTIVE, seal only works on CLOSED).

Storage Layout

data/
  wal_000001.bin          ← sealed WAL segment
  wal_000001.crc          ← WAL segment integrity sidecar (16 bytes)
  wal_000001.seal         ← seal marker (empty file)
  wal_000002.bin
  wal_000002.crc
  wal_000002.seal
  ...
  wal.bin                 ← current active segment, 0–64MB
  snapshot_000004.bin     ← snapshot at segment 4 (if frequency=4)
  snapshot_000004.crc     ← snapshot integrity sidecar (16 bytes)
  snapshot_000008.bin     ← snapshot at segment 8
  snapshot_000008.crc

Snapshot files use the segment_id of the sealed segment at which they were written. On recovery, the latest valid snapshot_NNNNNN.bin is loaded first to seed balances, then WAL segments from that point forward are replayed.

Recovery

Recovery is coordinated by the Recover struct (src/recover.rs), which holds mutable references to all pipeline stages:

#![allow(unused)]
fn main() {
pub struct Recover<'r> {
    transactor: &'r mut Transactor,
    wal: &'r Wal,
    snapshot: &'r Snapshot,
    seal: &'r mut Seal,
    sequencer: &'r Sequencer,
    storage: &'r Storage,
    segments: Vec<Segment>,
}
}

Recovery algorithm

1. List all segments (sorted by ID)
2. Locate latest snapshot:
     → scan segments for has_snapshot()
     → use the highest segment_id with a snapshot
3. Restore snapshot balances (if found):
     → load_snapshot() from snapshot segment
     → for each (account_id, balance):
          restore to Transactor, Snapshot stage, and Seal runner
     → set last_tx_id from snapshot
4. Replay sealed segments after snapshot:
     for each segment with id > snapshot_segment_id:
       → if not SEALED, pre-seal it (seal.recover_pre_seal())
       → load and visit WAL records
       → update last_tx_id from TxMetadata
       → restore balances from TxEntry.computed_balance
         to Transactor and Snapshot stage
5. Replay active segment (wal.bin):
     → visit WAL records
     → update last_tx_id, restore balances
6. Restore sequencer state:
     → sequencer.set_next_id(last_tx_id + 1)
     → transactor.store_last_processed_id(last_tx_id)
     → snapshot.store_last_processed_id(last_tx_id)

Recovery guarantees

• Unsealed segments encountered during recovery are pre-sealed before replay
• Balance state is restored to three systems: Transactor, Snapshot stage, Seal runner
• Sequencer resumes with the next ID after the last recovered transaction
• Active segment (wal.bin) is replayed last, handling partial writes at EOF

Source Module Organization

src/
  entities.rs           ← TxMetadata, TxEntry, SegmentHeader, SegmentSealed, WalEntry
  entries.rs            ← constructor helpers for SegmentHeader, SegmentSealed
  wal.rs                ← WAL pipeline stage (WalRunner thread)
  snapshot.rs           ← Snapshot pipeline stage (in-memory balance cache)
  seal.rs               ← Seal background thread (sealing + snapshot storing)
  recover.rs            ← Recover struct (crash recovery coordinator)
  sequencer.rs          ← Sequencer (atomic tx_id generator)
  transactor.rs         ← Transactor (balance validation, CRC computation)
  ledger.rs             ← Ledger orchestrator (pipeline assembly, startup)
  storage/
    mod.rs              ← re-exports
    layout.rs           ← file path conventions
    segment.rs          ← Segment lifecycle
    snapshot.rs         ← snapshot format (impl on Segment)
    storage.rs          ← Storage facade, StorageConfig
    wal_reader.rs       ← WAL loading and verification
    wal_serializer.rs   ← zero-copy record codec

Deleted modules: src/replay.rs, src/snapshot_store.rs — functionality absorbed into src/recover.rs and src/storage/snapshot.rs.

Consequences

Positive

• Per-transaction CRC32C catches bit rot before it corrupts balances
• File-level CRC catches full-file corruption and truncation
• SegmentHeader catches wrong-file errors immediately on replay
• Three-state segment lifecycle (ACTIVE → CLOSED → SEALED) provides clear phase boundaries and crash-safe transitions
• .seal marker file makes sealed status deterministic via filesystem inspection
• Seal thread is fully independent — WAL writer never blocks on fsync or CRC
• computed_balance in TxEntry eliminates balance recomputation downstream; Snapshot and Seal stages are read-only consumers of pre-computed balances
• Single authoritative balance computation point (Transactor) simplifies correctness
• Seal thread discovers unsealed work by directory listing — robust across crash/restart
• Configurable snapshot frequency balances storage cost vs recovery time
• LZ4-compressed snapshots reduce disk footprint; two-level CRC catches both decompression errors and on-disk corruption
• Atomic temp-file writes for snapshots prevent partial-write corruption
• bytemuck zero-copy serialization avoids allocation on the write path
• 16MB write buffer batches I/O for throughput
• Storage module separation provides clean layering between I/O and pipeline logic
• Recover struct coordinates multi-system state restoration in a single pass

Negative

• TxEntry._pad1 repurposed as computed_balance: i64 — breaking wire format change for any existing WAL files
• TxMetadata layout changed (added flags, tag, moved crc32c) — breaking change
• SegmentHeader and SegmentSealed add two new WalEntry variants — code change across WAL reader, WAL writer, recovery path
• New crate dependencies: crc32c, lz4_flex, bytemuck
• Seal thread has two responsibilities (sealing + snapshot storing) — more complex than a single-purpose process, but tightly coupled by design
• Seal thread maintains its own balance vector (duplicated from Snapshot stage), needed because Seal runs independently without queue access

Neutral

• Snapshot format is completely new (not backward-compatible with old format)
• snapshot_interval config field replaced by snapshot_frequency (segment-based)
• Existing tmp → rename atomic write pattern kept for snapshot files
• Zero-balance filtering for snapshots preserved from original design

Alternatives Considered

Compute balances in Seal/Snapshot stages (previous approach) Rejected — balance computation in multiple stages creates multiple sources of truth and risks divergence. Computing once in Transactor and propagating computed_balance through TxEntry is simpler, testable at one point, and avoids redundant arithmetic.

Remove snapshots entirely (previous ADR revision) Superseded — snapshots bound recovery time. Without snapshots, recovery time grows linearly with retained WAL history. Keeping snapshots in the Seal stage gives the best of both worlds: deterministic write path and bounded recovery time.

Separate SnapshotStorer process Superseded — moving snapshot storing into the Seal thread reduces thread count and makes the snapshot-per-seal relationship explicit. No separate coordination needed.

Seal as pipeline stage (receiving SegmentSealed via queue) Superseded — making Seal a queue consumer would couple it to the pipeline and require the WAL to wait for Seal backpressure. A polling thread is simpler, crash-safe (discovers work by listing), and allows the WAL to proceed without blocking.

Per-record CRC instead of per-transaction CRC Rejected — CRC per TxEntry individually does not verify that entries belong to the correct transaction. Per-transaction CRC covers the complete logical unit.

Synchronous sealing on WAL writer thread Rejected — fsync latency (1-10ms) on the WAL writer thread would directly reduce transaction throughput. Async Seal thread keeps the write path clean.

Two-state segment lifecycle (ACTIVE → SEALED) Superseded — without an intermediate CLOSED state, a crash between rename and CRC write would leave a segment in an ambiguous state. The three-state model with .seal marker makes each transition atomic and inspectable.

References

• ADR-001 — entries-based execution model, TxMetadata structure, WAL record format
• ADR-002 — Vec-based balance storage, Vec balance cache
• ADR-003 — Ledger API, LedgerConfig
• ADR-005 — PressureManager, park/unpark, Seal thread idle behaviour
• src/entities.rs — TxMetadata, TxEntry, SegmentHeader, SegmentSealed, WalEntry
• src/entries.rs — SegmentHeader/SegmentSealed constructors
• src/wal.rs — WAL pipeline stage
• src/snapshot.rs — Snapshot pipeline stage (in-memory)
• src/seal.rs — Seal background thread
• src/recover.rs — Recovery coordinator
• src/storage/segment.rs — Segment lifecycle
• src/storage/snapshot.rs — Snapshot serialization format
• src/storage/storage.rs — Storage facade, StorageConfig
• src/storage/wal_serializer.rs — Zero-copy record codec
• PostgreSQL WAL — CRC32C per record, segmented files
• TigerBeetle — CRC32C, explicit corruption rejection
• RocksDB — segmented SST files, async compaction
• crc32c crate — hardware-accelerated CRC32C
• lz4_flex crate — pure Rust LZ4 compression
• bytemuck crate — zero-copy Pod/Zeroable serialization

ADR-007: CLI Operational Tools

Status: Proposed
Date: 2026-04-03
Author: Taleh Ibrahimli
Branch: feature/adr-007-cli-tools

Context

roda-ledger stores all state in binary WAL segments and snapshot files (ADR-006). These files are the source of truth — but they are opaque. An operator who needs to inspect a specific transaction, verify data integrity, or recover from a manual edit has no way to do so without writing custom tooling.

Three operational needs have been identified:

Inspection — read a WAL segment and view its contents as human-readable JSON. Needed for debugging, auditing, and understanding what is in a specific segment.

Verification — check that WAL segments and snapshot files are intact. Needed after a disk event, before an archival operation, or to confirm recovery is safe.

Re-sealing — after a manual edit (e.g. removing a corrupt record, correcting a CRC), recompute the .crc sidecar and restore the .seal marker. Without this, the modified segment is permanently in CLOSED state and recovery will treat it as unverified.

These tools operate at the segment level — one segment at a time. They do not interact with a running ledger process.

Decision

Add a roda-ctl binary (src/bin/roda_ctl.rs) with three subcommands:

roda-ctl unpack <segment.bin> [--out <file.json>]
roda-ctl pack   <file.json>   [--out <segment.bin>]
roda-ctl verify [<path>] [--segment <id>] [--range <from>..<to>]
roda-ctl seal   <segment.bin>

No interaction with a running ledger. All commands operate on files directly. The running ledger is not affected — these are offline tools.

Command: unpack

Reads a WAL segment binary file and writes its contents as JSON.

roda-ctl unpack wal_000004.bin
roda-ctl unpack wal_000004.bin --out wal_000004.json
roda-ctl unpack wal_000004.bin --out -            # stdout

Output format

One JSON object per line (NDJSON / JSON Lines). Each line is one WAL record:

{"type":"SegmentHeader","segment_id":4,"version":1,"magic":"RODA","first_tx_id":441001}
{"type":"TxMetadata","tx_id":441001,"entry_count":2,"fail_reason":0,"crc32c":"0x4a2f1c3d","user_ref":0,"timestamp":1743600000}
{"type":"TxEntry","tx_id":441001,"account_id":42,"amount":100,"kind":"Debit","computed_balance":1200}
{"type":"TxEntry","tx_id":441001,"account_id":0,"amount":100,"kind":"Credit","computed_balance":-4200}
{"type":"TxMetadata","tx_id":441002,...}
...
{"type":"SegmentSealed","segment_id":4,"last_tx_id":974231,"record_count":533230}

One line per record. Streaming — does not load entire segment into memory before writing output. Large segments (64MB, ~1.6M records) can be unpacked and piped to grep, jq, or other tools without loading everything into memory first.

CRC verification during unpack

unpack verifies per-transaction CRC32C as it reads. If a record fails CRC:

ERROR: CRC mismatch at record 12441 (tx_id 453442)
  expected: 0x4a2f1c3d
  actual:   0x7f3a9b11
  offset:   498040

Unpack stops at the first CRC failure by default. Use --ignore-crc to continue past failures and emit the corrupt record with a "crc_error": true field.

Command: pack

Reads a JSON Lines file produced by unpack and writes a WAL segment binary.

roda-ctl pack wal_000004.json
roda-ctl pack wal_000004.json --out wal_000004_fixed.bin
roda-ctl pack - --out wal_000004_fixed.bin   # read from stdin

Use case: manual editing

The intended workflow for manual recovery:

# 1. unpack
roda-ctl unpack wal_000004.bin --out wal_000004.json

# 2. edit wal_000004.json in any text editor
#    e.g. remove a corrupt transaction, correct an amount

# 3. pack back to binary
roda-ctl pack wal_000004.json --out wal_000004_fixed.bin

# 4. replace original (keep backup)
cp wal_000004.bin wal_000004.bin.bak
cp wal_000004_fixed.bin wal_000004.bin

# 5. reseal (see seal command)
roda-ctl seal wal_000004.bin

CRC recomputation

pack always recomputes per-transaction CRC32C from the packed data. The crc32c field in the JSON input is ignored — the output always has fresh, correct CRCs. This ensures that editing a transaction and repacking produces a valid binary.

Validation during pack

pack validates the JSON input before writing:

• SegmentHeader must be the first record
• SegmentSealed must be the last record
• segment_id in SegmentHeader and SegmentSealed must match
• record_count in SegmentSealed must match actual record count
• tx_id sequence must be monotonically increasing within the segment

Validation failures abort the pack with an error. Use --no-validate to skip validation and pack regardless (dangerous — for forensics only).

Command: verify

Verifies the integrity of WAL segments and snapshot files. Can verify a single segment, a range of segments, or the entire data directory.

roda-ctl verify data/               # whole: E2E — all segments + any snapshots found
roda-ctl verify data/ --segment 4   # segment 4 + snapshot_000004 if it exists
roda-ctl verify data/ --range 1..10 # segments 1-10 + any snapshots found alongside

No direct file paths. Always operates on a data directory with an optional segment filter. Snapshots are never verified independently — they are verified as part of their paired segment check. If snapshot_NNNN.bin exists alongside wal_NNNN.bin, it is verified when that segment is checked. If no snapshot exists for a segment, nothing is expected — no warning, no error.

Snapshot verification rule

For each segment being verified:
  → verify WAL segment (file CRC + record CRC)
  → if snapshot_NNNN.bin exists alongside → verify it (file CRC + data CRC)
  → if no snapshot_NNNN.bin → nothing to check, move on

For each snapshot_NNNN.bin found in data/:
  → wal_NNNN.bin must exist and be SEALED → if not, WARNING (orphan snapshot)
  → verify balances in snapshot against balances in wal_NNNN.bin (and for accounts found only in previouse segments)

No assumptions about which segments should have snapshots. verify does not know or care what snapshot_frequency was configured as. It verifies what exists.

Verification levels

File-level (always performed for WAL segments):

• .crc sidecar exists for SEALED segments
• File size matches .crc recorded size
• CRC32C of entire file matches .crc recorded checksum

Snapshot-level (performed when snapshot exists alongside segment):

• File CRC32C matches .crc sidecar
• File size matches .crc recorded size
• Data CRC32C in snapshot header matches decompressed record data
• Check balances in snapshot against balances in WAL segment (and for accounts found only in previous segments)

Record-level (performed on WAL segments):

• SegmentHeader magic and version valid
• segment_id matches filename
• Per-transaction CRC32C valid for every TxMetadata
• record_count in SegmentSealed matches actual count
• tx_id monotonically increasing throughout segment

Cross-segment (performed on directory or range only):

• No gaps in segment numbering
• first_tx_id of segment N+1 = last_tx_id of segment N + 1
• Active segment (wal.bin) tx_id continues from last sealed segment

Output

Verifying data/ ...

  wal_000001.bin   SEALED    533,230 records   tx 1..533230        OK
  wal_000002.bin   SEALED    533,231 records   tx 533231..1066461  OK
  wal_000003.bin   SEALED    412,000 records   tx 1066462..1478461 OK
  wal_000004.bin   SEALED    533,230 records   tx 1478462..2011691 OK
    snapshot_000004.bin      1,200 accounts                        OK
  wal_000005.bin   CLOSED    (awaiting seal)                       OK
  wal.bin          ACTIVE    12,441 records    tx 2011692..        OK

Summary: 6 segments, 1 snapshot, 2,024,133 transactions — all OK

Snapshot verification appears indented under the segment it belongs to — makes the pairing explicit. If no snapshot exists for a segment, nothing is shown.

On failure:

  wal_000003.bin   SEALED    412,000 records   tx 1066462..1478461 FAIL
    Record CRC mismatch at record 204,441 (tx_id 1,270,902)
    expected: 0x4a2f1c3d  actual: 0x7f3a9b11  offset: 7,763,358

  wal_000004.bin   SEALED    533,230 records   tx 1478462..2011691 OK
    snapshot_000004.bin      1,200 accounts                        FAIL
    Data CRC mismatch: expected 0x1a2b3c4d  actual 0x9f8e7d6c

Exit code 0 on success, 1 on any failure. Suitable for scripts and CI.

Command: seal

Recomputes the .crc sidecar and restores the .seal marker for a WAL segment in CLOSED state. Used after manual editing to restore a segment to SEALED state.

roda-ctl seal wal_000004.bin
roda-ctl seal data/ --segment 4     # equivalent

Seal sequence

1. Verify segment is in CLOSED state (wal_NNNNNN.bin exists, no .seal marker)
2. Read entire wal_NNNNNN.bin into memory
3. Compute CRC32C of file contents
4. Write wal_NNNNNN.crc (16 bytes: crc32c + size + magic)
5. fsync wal_NNNNNN.crc
6. Create empty wal_NNNNNN.seal marker file
7. fsync wal_NNNNNN.seal

This is identical to Segment.seal() in ADR-006. The CLI command exposes the same operation for manual use.

Guards

seal refuses to run if:

• The file does not exist
• The segment is already SEALED (.seal marker exists) — use --force to reseal
• The segment is ACTIVE (wal.bin) — cannot seal the active segment

--force removes existing .crc and .seal files before resealing. Used when a manual edit requires resealing a previously sealed segment.

# After editing an already-sealed segment:
roda-ctl seal wal_000004.bin --force

Safety Notes

pack and seal are dangerous. They modify the source of truth. Always back up the original files before using them.

Do not run CLI tools while the ledger is running. These are offline tools. Running seal or pack on files that the ledger is actively reading or writing will produce undefined behaviour.

CRC recomputation in pack is not magic. Packing a file with logically wrong data (e.g. wrong amounts) will produce a valid binary with correct CRCs. The tools verify format integrity, not business logic correctness.

Implementation Notes

Binary location

src/bin/roda_ctl.rs

Compiled as a separate binary alongside stress_testing. Uses the storage module from ADR-006 (src/storage/) directly — same serialization, same CRC computation, same segment lifecycle logic.

Dependencies

No new dependencies beyond what ADR-006 already requires (crc32c, lz4_flex, bytemuck). JSON output uses serde_json already present in Cargo.toml.

Feature flag

[[bin]]
name = "roda-ctl"
path = "src/bin/roda_ctl.rs"

No feature flag — always built. The binary is small and adds no compile-time cost to the library.

Consequences

Positive

• Operators can inspect WAL contents without writing custom code
• Integrity verification is scriptable and CI-compatible (exit codes)
• Manual recovery from corruption is possible via unpack → edit → pack → seal
• pack always recomputes CRC — edited segments are always valid after packing
• verify covers file-level, record-level, and cross-segment consistency
• Streaming unpack handles large segments without memory pressure
• Tools reuse existing storage module — no new serialization logic

Negative

• pack and seal are dangerous — documented but not preventable
• No protection against running while ledger is live — operator responsibility
• pack recomputes CRC but cannot validate business logic correctness
• --force on seal allows resealing without any warning beyond documentation

Neutral

• roda-ctl is a separate binary — not part of the server, not part of the library
• All commands are offline — no gRPC, no running ledger required
• NDJSON output format is streamable and compatible with standard Unix tools

Alternatives Considered

Single roda-ctl inspect command with flags Rejected — unpack, pack, verify, and seal are distinct operations with distinct safety profiles. Separate subcommands make intent clear and allow separate --help documentation per command.

Binary-level hex dump instead of JSON Rejected — JSON is human-readable and editable without knowledge of the binary format. The purpose of unpack is inspection and editing, not low-level debugging.

Verify as part of recovery (not a separate command) Recovery already verifies during replay. The verify command is for offline pre-checks — before archiving, after a disk event, in CI. Different use case.

Integrate tools into the server binary as subcommands Rejected — mixing operational tools with the server binary conflates concerns. roda-ctl is a standalone tool that can be run on any machine with access to the data directory, not just on the server.

References

• ADR-006 — WAL segment lifecycle (ACTIVE/CLOSED/SEALED), .seal marker, CRC sidecar format, Segment.seal(), Storage facade
• src/storage/segment.rs — Segment struct, lifecycle states
• src/storage/wal_serializer.rs — zero-copy record codec (reused by pack/unpack)
• src/storage/wal_reader.rs — WAL loading and CRC verification (reused by verify)
• src/storage/snapshot.rs — snapshot format (reused by verify)
• src/bin/stress_testing.rs — existing binary pattern

ADR-008: Transaction Index and Query Serving

Status: Accepted Date: 2026-04-04
Author: Taleh Ibrahimli

Context

roda-ledger currently has no way to retrieve a specific transaction by ID or retrieve the transaction history for a specific account. Both are fundamental operations for a production financial ledger:

• GetTransaction(tx_id) — retrieve the full details of a specific transaction. Required for payment status checks, reconciliation, and audit.

• GetAccountHistory(account_id, from_tx_id, limit) — retrieve transactions involving a specific account, newest first, with pagination. Required for account statements, balance verification, and dispute resolution.

GetBalance is already served correctly via Vec<AtomicI64> indexed directly by account_id — a direct atomic read from any gRPC thread. No changes needed.

Query frequency ratio

SubmitTransaction  :  GetTransaction  :  GetAccountHistory  :  GetBalance
      100          :       10         :          1          :     50

GetAccountHistory is the rarest operation. The index design reflects this — hot path is optimized for GetTransaction, account history is best-effort in-memory with disk fallback.

Decision

Two-tier query architecture

Hot tier  (active segment):   Snapshot stage owns TransactionIndexer
                              queries routed via SnapshotMessage queue
                              no shared mutable state between threads

Cold tier (sealed segments):  immutable index files on disk
                              gRPC reads directly, concurrently safe

Single queue for WAL entries and queries

Both WalEntry records and QueryRequest messages flow through a single ArrayQueue<SnapshotMessage>:

#![allow(unused)]
fn main() {
pub enum SnapshotMessage {
  Entry(WalEntry),
  Query(QueryRequest),
}
}

A single FIFO queue gives read-your-own-writes consistency for free. A query submitted after a transaction is enqueued after its WalEntry. The Snapshot stage processes them in order — the query always sees the correct committed state. Two separate queues would lose this ordering guarantee.

Snapshot stage owns TransactionIndexer exclusively

TransactionIndexer is owned by the Snapshot stage thread. No other thread accesses it. This eliminates all thread safety concerns — no locks, no atomics, no unsafe. gRPC threads never touch the indexer directly; they send queries via SnapshotMessage::Query and receive responses via a per-request callback channel.

In-Memory Index: TransactionIndexer

Three pre-allocated, fixed-size structures. Zero heap allocation after construction. No cold start on segment rotation — circular buffers self-evict, never reset.

#![allow(unused)]
fn main() {
pub struct TransactionIndexer {
  circle1:            Vec<TxSlot>,          // tx_id → location in circle2
  circle2:            Vec<IndexedTxEntry>,  // entry storage + account prev_link chain
  account_heads:      Vec<(u64, u32)>,      // account_id → latest circle2 index
  circle1_mask:       usize,                // circle1_size - 1, for bitmask
  circle2_mask:       usize,                // circle2_size - 1
  account_heads_mask: usize,                // account_heads_size - 1
  write_head2:        usize,                // monotonically increasing circle2 position
}
}

Circle 1 — transaction index (TxSlot, 16 bytes)

Maps tx_id to its starting position in circle2. Direct-mapped cache — tx_id & mask gives the slot index. Self-evicting: a newer transaction with the same masked tx_id overwrites the slot. slot.tx_id != queried_tx_id = evicted.

#![allow(unused)]
fn main() {
pub struct TxSlot {
  pub tx_id:       u64,      // 0 = empty slot; mismatch = evicted
  pub offset:      u32,      // start index in circle2 for this transaction's entries
  pub entry_count: u8,       // number of IndexedTxEntry records for this transaction
  pub _pad:        [u8; 3],
}                              // 16 bytes
}

Circle 2 — entry storage with account link chain (IndexedTxEntry, 48 bytes)

Stores TxEntry records sequentially. Each entry carries a prev_link field — the same account previous transaction link in circle2. This forms a per-account backward linked list through the buffer without any separate data structure.

#![allow(unused)]
fn main() {
pub struct IndexedTxEntry {
  pub entry_type:       u8,        // WalEntryKind::TxEntry
  pub kind:             EntryKind, // Credit or Debit
  pub _pad:             [u8; 6],
  pub tx_id:            u64,
  pub account_id:       u64,
  pub amount:           u64,
  pub computed_balance: i64,       // balance after this entry (set by Transactor)
  pub prev_link:        u32,       // same account previous transaction link in circle2
  // 1-based: 0 = no previous entry in buffer for this account
  //          N = entry is at circle2 slot N-1
  pub _pad_end:         [u8; 4],
}                                    // 48 bytes
}

prev_link is 1-based to distinguish “no link” (0) from “slot 0” (1). When a slot is overwritten, any prev_link pointing to it encounters an account_id mismatch — the chain breaks naturally. No cleanup needed.

Account heads — direct-mapped

#![allow(unused)]
fn main() {
account_heads: Vec<(u64, u32)>
// index:   account_id & mask
// value:   (account_id, circle2_slot_index)
// purpose: entry point into prev_link chain for each account
}

Same direct-mapped collision handling as circle1. If account_heads[slot].0 != account_id — head overwritten by another account — treat as cache miss, fall back to cold tier.

Write path (per WalEntry in Snapshot stage)

On TxMetadata(m):
  circle1[m.tx_id & mask] = TxSlot {
      tx_id:       m.tx_id,
      offset:      write_head2 & circle2_mask,
      entry_count: m.entry_count,
  }

On TxEntry(e):
  head_slot = e.account_id & account_heads_mask
  prev_link = if account_heads[head_slot].0 == e.account_id {
      account_heads[head_slot].1 + 1   // 1-based: existing head slot → link
  } else {
      0                                 // no previous entry for this account
  }

  circle2[write_head2 & circle2_mask] = IndexedTxEntry {
      tx_id, account_id, amount, kind, computed_balance,
      prev_link,
      ...
  }

  account_heads[head_slot] = (e.account_id, write_head2 & circle2_mask)
  write_head2 += 1

GetTransaction (hot)

slot = circle1[tx_id & mask]
if slot.tx_id != tx_id → miss → cold tier

for i in 0..slot.entry_count:
    entry = circle2[(slot.offset + i) & mask]
    if entry.tx_id != tx_id → evicted → miss → cold tier
    collect entry

return entries

GetAccountHistory (hot)

head_slot = account_id & account_heads_mask
if account_heads[head_slot].0 != account_id → miss → cold tier

current = account_heads[head_slot].1

loop:
    entry = circle2[current & circle2_mask]
    if entry.account_id != account_id → evicted, stop
    if from_tx_id != 0 && entry.tx_id < from_tx_id → stop
    collect entry
    if collected == limit → stop
    if entry.prev_link == 0 → stop
    current = entry.prev_link - 1   // 1-based → 0-based slot

return collected (newest first)

Sizing

Both circle sizes must be power-of-2 for bitmask indexing. Configured via LedgerConfig:

#![allow(unused)]
fn main() {
pub struct LedgerConfig {
  pub circle1_size:       usize,   // default: 0 (auto — fit active segment at startup)
  pub circle2_size:       usize,   // default: 0 (auto — circle1_size × avg_entries)
  pub account_heads_size: usize,   // default: 1_048_576 (1M slots)
}
}

Auto-sizing: count records in active wal.bin at startup, round up to next power of two. Guarantees active segment fits in cache on startup.

No cold start on segment rotation. Segment sealing does not reset the circles. Buffers continue filling with new entries. Old entries from the previous segment naturally expire as slots are overwritten. Latency is stable across rotation.

Memory at default config (64MB segment, ~1.6M entries):

circle1:       2M slots × 16 bytes = 32MB
circle2:       2M slots × 48 bytes = 96MB   (circle2 ≥ circle1 × avg_entries_per_tx)
account_heads: 1M slots × 12 bytes = 12MB
Total:         ~140MB fixed, pre-allocated, never grows

Query Serving

SnapshotMessage and QueryRequest

#![allow(unused)]
fn main() {
pub enum SnapshotMessage {
  Entry(WalEntry),
  Query(QueryRequest),
}

pub struct QueryRequest {
  pub kind:    QueryKind,
  pub respond: FnOnce(QueryResponse),
}

pub enum QueryKind {
  GetTransaction    { tx_id: u64 },
  GetAccountHistory { account_id: u64, from_tx_id: u64, limit: usize },
}

pub enum QueryResponse {
  Transaction(Option<TransactionRecord>),
  AccountHistory(Vec<TransactionRecord>),
}
}

Per-request response channel

Each QueryRequest carries its own mpsc::SyncSender<QueryResponse>. The Snapshot stage calls respond.send(result) directly — no shared response queue, no routing ambiguity when multiple gRPC threads query simultaneously. Each caller blocks on its own private rx.recv().

#![allow(unused)]
fn main() {
// gRPC handler:
let (tx, rx) = std::sync::mpsc::sync_channel(1);  // oneshot semantics
snapshot_queue.push(SnapshotMessage::Query(QueryRequest {
kind:    QueryKind::GetTransaction { tx_id },
respond: |entry| {tx.send(entry).unwrap();},
}));
let response = rx.recv().unwrap();  // blocks on private channel only
}

No tokio dependency. Pure std. One sender, one receiver, capacity 1 = oneshot.

Snapshot stage loop

loop:
  match inbound.pop():
    Some(SnapshotMessage::Entry(e)):
      apply entry to balances Vec<AtomicI64>     (existing)
      indexer.insert_tx or insert_entry           (new)
      pressure.on_work()

    Some(SnapshotMessage::Query(q)):
      result = match q.kind:
        GetTransaction(tx_id):
          indexer.get_transaction(tx_id)
          → Some → return TransactionRecord
          → None → cold tier lookup
        GetAccountHistory(account_id, from, limit):
          indexer.get_account_history(account_id, from, limit)
          → non-empty → return results
          → empty → cold tier lookup
      q.respond.send(result)
      pressure.on_work()

    None:
      pressure.on_retry_receive(retry, last_tx_id)
      retry++

Cold Tier — Sealed Segments

gRPC handler reads sealed files directly when Snapshot stage returns a miss. Sealed files are immutable — concurrent reads from multiple gRPC threads are safe with no coordination.

Sealed segment indexes loaded at startup

#![allow(unused)]
fn main() {
pub struct SealedSegmentIndex {
  pub segment_id:  u32,
  pub first_tx_id: u64,
  pub last_tx_id:  u64,
  pub wal_offsets: Vec<(u64, u64)>,   // (tx_id, byte_offset), sorted by tx_id
  pub account_txs: Vec<(u64, u64)>,   // (account_id, tx_id), sorted by (account_id, tx_id)
}

// atomic swap when new segment sealed:
sealed_indexes: ArcSwap<Vec<SealedSegmentIndex>>
}

Loaded once at startup, never mutated. ArcSwap updated when Seal thread seals a new segment and writes index files. gRPC threads holding old Arc finish safely.

GetTransaction (cold)

find segment: binary search sealed_indexes for tx_id in [first_tx_id..last_tx_id]
binary search wal_offsets for tx_id → byte_offset
seek to byte_offset in wal_NNNN.bin
read TxMetadata + TxEntry records
return TransactionRecord

GetAccountHistory (cold)

for each sealed segment, newest to oldest:
    binary search account_txs for account_id range
    collect tx_ids >= from_tx_id, up to limit
    for each tx_id → GetTransaction cold
    if limit reached → stop
return merged results

On-Disk Index Files (built by Seal thread)

Built as side effect of existing WAL replay loop in Seal thread. Two flat Vec<(u64, u64)> appended during replay, one sort at seal time.

wal_index_000004.bin:
  [record_count: 8 bytes LE]
  [tx_id: 8][offset: 8] × record_count   ← naturally sorted (WAL append-only)
wal_index_000004.crc

account_index_000004.bin:
  [record_count: 8 bytes LE]
  [account_id: 8][tx_id: 8] × record_count  ← sort_unstable at seal time
account_index_000004.crc

Memory during seal:

#![allow(unused)]
fn main() {
wal_entries:     Vec<(u64, u64)>   // (tx_id, offset)      — one per TxMetadata
account_entries: Vec<(u64, u64)>   // (account_id, tx_id)  — one per TxEntry
}

One flat allocation each. No per-account allocation. ~51MB total during seal, freed after write.

Storage Layout (updated from ADR-006)

data/
  wal_000004.bin
  wal_000004.crc
  wal_000004.seal
  wal_index_000004.bin
  wal_index_000004.crc
  account_index_000004.bin
  account_index_000004.crc
  snapshot_000004.bin
  snapshot_000004.crc
  wal.bin

GetBalance — unchanged

Direct Vec<AtomicI64> read with Ordering::Acquire from any gRPC thread. O(1), lock-free, no queue involvement. Routing through SnapshotMessage would add enormous overhead for a single atomic load instruction.

gRPC API (ADR-004 extension)

rpc GetTransaction(GetTransactionRequest) returns (GetTransactionResponse);
        rpc GetAccountHistory(GetAccountHistoryRequest) returns (GetAccountHistoryResponse);

message GetTransactionRequest  { uint64 tx_id = 1; }

message GetTransactionResponse {
  uint64               tx_id       = 1;
  uint64               user_ref    = 2;
  uint64               timestamp   = 3;
  uint32               fail_reason = 4;
  repeated TxEntryRecord entries   = 5;
}

message TxEntryRecord {
  uint64    account_id       = 1;
  uint64    amount           = 2;
  EntryKind kind             = 3;
  int64     computed_balance = 4;
}

enum EntryKind { CREDIT = 0; DEBIT = 1; }

message GetAccountHistoryRequest {
  uint64 account_id = 1;
  uint64 from_tx_id = 2;  // 0 = latest
  uint32 limit      = 3;  // default 20, max 1000
}

message GetAccountHistoryResponse {
  repeated TxEntryRecord entries    = 1;
  uint64                 next_tx_id = 2;  // 0 = no more entries
}

Consequences

Positive

• Single queue gives read-your-own-writes consistency for free — no special handling
• Snapshot stage owns TransactionIndexer exclusively — zero thread safety complexity
• No locks, no atomics, no unsafe anywhere in the index
• circle1 + circle2 + account_heads are pre-allocated, fixed size, never grow
• Self-evicting circular buffers — no cold start on segment rotation
• prev_link chain gives O(account_transactions_in_buffer) account history — no scan
• Per-request mpsc::sync_channel(1) — oneshot semantics, no tokio, pure std
• Cold tier reads immutable files — concurrent, no coordination
• ArcSwap for sealed indexes — atomic update, readers never blocked
• On-disk indexes built as side effect of existing Seal thread WAL replay

Negative

• Hot query adds queue round-trip latency vs direct memory access — acceptable for a cold query path (10:1 ratio vs submit)
• gRPC caller thread blocks on rx.recv() — wrap in spawn_blocking for async handlers
• account_heads collisions cause cache misses for affected accounts — disk fallback
• circle1 collisions evict older transactions — expected, graceful disk fallback
• account_heads at 1M slots: higher collision rate at 50M accounts — tunable
• Deep account history across many sealed segments requires multiple disk seeks — application layer should cache account history

Neutral

• SnapshotMessage enum wraps existing WalEntry — minimal change to Snapshot stage
• circle sizes power-of-2 enforced at startup via next_power_of_two()
• GetBalance unchanged — Vec direct read, no queue involvement
• Application layer responsible for caching account history for user-facing queries

Alternatives Considered

WalBuffer (append-only byte buffer) Evaluated extensively. Rejected — thread safety between Snapshot stage writer and gRPC reader threads required unsafe code, atomics, or locks. The query message passing approach eliminates the problem entirely: Snapshot stage owns the buffer, gRPC never touches it directly.

Two separate queues (WalEntry queue + Query queue) Rejected — loses ordering between WAL entries and queries. A query submitted after a transaction could be processed before the transaction’s WalEntry, returning a stale miss. Single queue preserves FIFO ordering and gives read-your-own-writes.

Linear scan of active segment for GetTransaction Rejected — O(segment_size) for a query that occurs 10:1 vs submit. circle1 direct- mapped cache gives O(1) for the common case.

Linear scan for GetAccountHistory Acceptable for active segment but replaced by prev_link chain — O(account_txs) traversal is strictly better than O(all_entries) scan.

Tokio oneshot channel for query response Rejected — introduces tokio dependency into the core ledger. mpsc::sync_channel(1) provides identical oneshot semantics with pure std.

Global account index in memory Rejected — grows unboundedly, expensive to rebuild on every seal. Per-segment account_index files loaded into SealedSegmentIndex at startup provide the same capability with bounded memory.

References

• ADR-001 — TxMetadata, TxEntry, WalEntry, computed_balance
• ADR-002 — Vec balance cache
• ADR-004 — gRPC interface, existing RPCs
• ADR-006 — WAL segment lifecycle, Seal thread, storage layout
• ADR-007 — CLI tools (verify extended to cover index files)
• src/snapshot.rs — Snapshot stage (extended with TransactionIndexer + query handling)
• src/transaction_indexer.rs — TransactionIndexer implementation (already written)
• src/storage/segment.rs — Seal thread WAL replay (extended for index file build)
• std::sync::mpsc::sync_channel — per-request oneshot response channel
• arc-swap crate — ArcSwap for sealed index atomic update (already in Cargo.toml)

ADR-009: Transaction Links, Deduplication, and Reversal

Status: Accepted (TxLink + Dedup implemented; Reversal deferred) Date: 2026-04-04 Author: Taleh Ibrahimli

Amends:

• ADR-001 — adds WalEntryKind::Link, TxLink record, link_count to TxMetadata
• ADR-003 — adds FailReason::DUPLICATE, deduplication to Transactor

Context

Three related concerns are addressed together because they share the same underlying mechanism — TxLink — and are best understood as a coherent feature:

1. Transactions have no relationships. A duplicate rejection carries FailReason::DUPLICATE but no reference to the original committed transaction. A reversal is just a regular transaction with no link to what it reverses. Audit trails are incomplete and callers must maintain their own cross-reference tables outside the ledger.

2. No duplicate protection. user_ref was designed as an opaque external reference but nothing enforces uniqueness. A client that retries after a network failure or crash recovery will create duplicate transactions — double charges, duplicate deposits. The crash recovery resubmission pattern in ADR-008 explicitly requires retry after crash, which without deduplication corrupts balances.

3. No reversal mechanism. Reversing a transaction requires submitting a mirror Complex operation. The relationship between the reversal and the original exists only in the application layer, not in the ledger itself.

Decision

TxLink — first-class WAL record

Introduce TxLink as a new WAL record type. TxLink is a sub-record of a transaction — it follows the TxEntry records for the same TxMetadata and is counted in a new link_count field. The tx_id of the link is the enclosing TxMetadata.tx_id — not stored in TxLink itself.

Deduplication — flip-flop HashMap in Transactor

The Transactor maintains two HashMap<u64, u64> maps (user_ref → tx_id) that flip on a configurable time window. A duplicate user_ref within the window is rejected with FailReason::DUPLICATE and a TxLink { kind: Duplicate } is written to the WAL pointing to the original committed transaction.

Reversal Operation (deferred — not yet implemented)

A reversal will be submitted as Operation::Reversal { to_tx_id, user_ref }. The ledger will locate the original transaction, create opposite entries for it (debit <=> credit, credit <=> debit), and add a TxLink { kind: Reversal, to_tx_id: original_tx_id }.

The TxLinkKind::Reversal variant is reserved in the WAL format (value = 1) but the Operation::Reversal variant and its Transactor logic are not yet implemented. This will be added in a future ADR or follow-up to this one.

WAL Record Changes

Updated TxMetadata (40 bytes, unchanged total size)

link_count is added at offset 2, shifting fail_reason and replacing flags. Total size remains 40 bytes.

#![allow(unused)]
fn main() {
#[repr(C)]
pub struct TxMetadata {
    pub entry_type:  u8,           // 1 @ 0  — WalEntryKind::TxMetadata
    pub entry_count: u8,           // 1 @ 1  — number of TxEntry records following
    pub link_count:  u8,           // 1 @ 2  — number of TxLink records after entries
    pub fail_reason: FailReason,   // 1 @ 3
    pub crc32c:      u32,          // 4 @ 4
    pub tx_id:       u64,          // 8 @ 8
    pub timestamp:   u64,          // 8 @ 16
    pub user_ref:    u64,          // 8 @ 24
    pub tag:         [u8; 8],      // 8 @ 32
}                                  // 40 bytes total
}

TxLink (40 bytes)

#![allow(unused)]
fn main() {
#[repr(C)]
pub struct TxLink {
    pub entry_type: u8,            // 1 @ 0  — WalEntryKind::Link (4)
    pub link_kind:  TxLinkKind,    // 1 @ 1
    pub _pad:       [u8; 6],       // 6 @ 2
    pub to_tx_id:   u64,           // 8 @ 8  — referenced transaction
    pub _pad2:      [u8; 24],      // 24 @ 16
}                                  // 40 bytes total

#[repr(u8)]
pub enum TxLinkKind {
    Duplicate = 0,   // this transaction is a duplicate of to_tx_id
    Reversal  = 1,   // this transaction reverses to_tx_id
}
}

Updated WalEntry

#![allow(unused)]
fn main() {
pub enum WalEntry {
    Metadata(TxMetadata),          // kind = 0
    Entry(TxEntry),                // kind = 1
    SegmentHeader(SegmentHeader),  // kind = 2
    SegmentSealed(SegmentSealed),  // kind = 3
    Link(TxLink),                  // kind = 4 ← new
}
}

WAL sequence

TxMetadata { tx_id, entry_count: N, link_count: M, ... }
TxEntry    × N
TxLink     × M

Reader advances through entry_count TxEntry records then link_count TxLink records. Exact counts known from TxMetadata — no scanning needed.

CRC coverage

CRC32C in TxMetadata.crc32c covers the complete logical transaction:

CRC input:
  TxMetadata bytes (crc32c field zeroed)
  + all TxEntry records (entry_count)
  + all TxLink records (link_count)

Computed by the Transactor before pushing any record to the WAL queue. Corrupt or missing links detected at WAL replay time.

Deduplication

Flip-flop HashMap in Transactor

#![allow(unused)]
fn main() {
struct DedupCache {
    active:       HashMap<u64, u64>,   // user_ref → tx_id, current window
    previous:     HashMap<u64, u64>,   // user_ref → tx_id, previous window
    window_ms:    u64,                 // deduplication window in milliseconds
    window_start: u64,                 // timestamp when active window started
}
}

Flip logic — checked on every transaction:

now = current_time_ms()
if now - window_start >= window_ms:
    swap(active, previous)   // previous becomes active's old data
    active.clear()           // reuse allocation, zero re-alloc
    window_start = now

Lookup:

if user_ref == 0 → skip check (no idempotency key provided)

check active[user_ref]   → found → DUPLICATE
check previous[user_ref] → found → DUPLICATE
not found → proceed, insert into active after commit
found  → proceed, insert into active after commit (rejected transactions also kept in wal)

Effective window: window_ms to 2 × window_ms. An entry inserted at the start of a window lives until the end of the next window. An entry inserted at the end lives for at least window_ms. A 10-second window gives 10–20 seconds of deduplication coverage — sufficient for all retry scenarios.

Memory:

At 100K submissions per 10-second window:
  active:   100K entries × ~50 bytes = ~5MB
  previous: 100K entries × ~50 bytes = ~5MB
  Total:    ~10MB — bounded by submission rate, not pipeline throughput

clear() reuses the HashMap allocation. No GC pressure on flip.

On crash recovery:

Rebuild from WAL replay. For each committed TxMetadata with user_ref != 0 and timestamp within last 2 × window_ms, insert into active or previous based on timestamp. Deduplication window fully restored from WAL after recovery.

New FailReason

#![allow(unused)]
fn main() {
pub const DUPLICATE: Self = Self(7);
// user_ref already committed within deduplication window
}

LedgerConfig additions

#![allow(unused)]
fn main() {
pub struct LedgerConfig {
    // existing fields...
    pub dedup_enabled:    bool,   // default: true
    pub dedup_window_ms:  u64,    // default: 10_000 (10 seconds)
}
}

dedup_enabled: false disables the cache entirely — zero overhead, no check. Useful for internal pipelines where user_ref is not used.

Duplicate Transaction Flow

Transactor detects duplicate user_ref within the window:

TxMetadata {
    tx_id:       new_tx_id,
    entry_count: 0,              // no balance change
    link_count:  1,
    fail_reason: DUPLICATE,
    user_ref:    duplicate_user_ref,
}
TxLink {
    kind:      Duplicate,
    to_tx_id:  original_tx_id,
}

What the caller sees:

submit(op, user_ref=12345) → tx_id = 441100
GetTransactionStatus(441100) → ERROR, fail_reason=DUPLICATE
GetTransaction(441100) → link: Duplicate → to_tx_id=441001
GetTransaction(441001) → original committed transaction

The caller always recovers the original tx_id regardless of how many retries were attempted. The duplicate record in the WAL provides a permanent audit trail of the retry attempt.

Reversal Flow (deferred)

Not yet implemented. The TxLinkKind::Reversal value (1) is reserved in the WAL format so that future reversal records are forward-compatible with WAL files written today. The Operation::Reversal variant, Transactor reversal index, and gRPC Reversal message will be added in a follow-up.

gRPC API (amends ADR-004)

TxLink records included in GetTransactionResponse:

message GetTransactionResponse {
  uint64               tx_id       = 1;
  uint64               user_ref    = 2;
  uint64               timestamp   = 3;
  uint32               fail_reason = 4;
  repeated TxEntryRecord entries   = 5;
  repeated TxLinkRecord  links     = 6;
}

message TxLinkRecord {
  uint64   to_tx_id = 1;
  LinkKind kind     = 2;
}

enum LinkKind {
  DUPLICATE = 0;
  REVERSAL  = 1;
}

CLI Impact (amends ADR-007)

roda-ctl unpack emits TxLink records as JSON Lines:

{"type":"TxMetadata","tx_id":441099,"entry_count":2,"link_count":1,...}
{"type":"TxEntry","tx_id":441099,"account_id":1,"amount":100,"kind":"Credit",...}
{"type":"TxEntry","tx_id":441099,"account_id":0,"amount":100,"kind":"Debit",...}
{"type":"TxLink","tx_id":441099,"kind":"Reversal","to_tx_id":441001}

roda-ctl verify validates:

• link_count in TxMetadata matches actual TxLink records read

Consequences

Positive

• Transaction relationships are first-class WAL records — auditable, covered by CRC
• Duplicate detection protects against network retry and crash recovery resubmission — exactly-once semantics achievable with user_ref + retry
• Caller always recovers original tx_id from a duplicate via TxLink
• Deduplication cache rebuilt from WAL on crash recovery — window survives crash
• Reversal audit trail is explicit in the WAL — no external cross-reference needed (deferred)
• Flip-flop HashMap — exact match, no false positives, bounded memory, O(1) check
• link_count = 0 is the common case — zero overhead for normal transactions
• TxLinkKind 128–255 available for user-defined link kinds

Negative

• link_count added to TxMetadata — breaking format change (alongside ADR-006 changes, one migration covers both)
• New WalEntryKind::Link = 4 — all WAL readers and replay paths must handle it
• Operation::Reversal deferred — requires Transactor reversal index (future work)
• Deduplication cache lost on crash — rebuilt from WAL replay (startup cost)
• dedup_window_ms and dedup_enabled added to LedgerConfig

Neutral

• TxLink is 40 bytes — consistent with all other WAL record types
• user_ref = 0 skips deduplication check — no overhead for operations without key
• Deduplication memory bounded by submission rate, not pipeline throughput (~10MB)

Alternatives Considered

Store to_tx_id as a field in TxMetadata Rejected — only one link per transaction, not extensible. Separate TxLink records allow multiple links and are consistent with the entry model.

Circular buffer for deduplication cache Rejected — hash collisions cause false positives (legitimate transactions rejected). Flip-flop HashMap gives exact match with no false positives at comparable memory cost.

Enforce reversal entries match original Rejected — the system now handles reversal entries generation automatically, ensuring they always correctly mirror the original transaction.

Separate deduplication ADR Rejected — deduplication, TxLink, and reversal are coupled: deduplication emits a TxLink on rejection, reversal is expressed as a TxLink. A single ADR covering all three is more coherent than three separate ADRs with circular references.

Links on simple operations Rejected — Deposit, Withdrawal, and Transfer are standalone by design. Links are only meaningful on Complex operations where the caller controls the full transaction structure.

References

• ADR-001 — TxMetadata, TxEntry, WalEntry, CRC coverage (amended by this ADR)
• ADR-003 — Operation enum, ComplexOperation, FailReason (amended by this ADR)
• ADR-004 — gRPC interface (amended by this ADR)
• ADR-006 — WAL segment lifecycle, Seal thread, storage layout
• ADR-007 — CLI tools (amended by this ADR)
• ADR-008 — Transaction index, GetTransaction response
• src/entities.rs — TxLink, TxLinkKind, updated WalEntry, updated TxMetadata
• src/dedup.rs — DedupCache (flip-flop HashMap)
• src/transactor.rs — duplicate detection, DedupCache integration

ADR-010: Synchronous Submit with Wait Levels

Status: Proposed
Date: 2026-04-06
Author: Taleh Ibrahimli

Amends:

• ADR-004 — adds SubmitAndWait RPC, WaitLevel enum, SubmitResult
• ADR-003 — adds submit_and_wait() to Ledger API

Context

submit() is the primary and recommended submission method in roda-ledger. It is fire-and-forget — returns tx_id immediately, never blocks, and allows the pipeline to run at full throughput. For high-throughput workloads, internal pipelines, and batch processing, submit() is the correct choice.

However, not every caller needs maximum throughput. Some callers have simpler requirements:

• Rejection feedback — know immediately if the transaction was rejected (insufficient funds, duplicate) without a polling loop.
• Durability confirmation — know the transaction is durable on disk before returning a receipt to an end user.
• Balance consistency — know that GetBalance reflects the transaction before querying it.

These callers can implement polling on top of submit() — but this places the burden of timeout handling, polling frequency, and pipeline stage reasoning on every integrator. submit_and_wait() is an ergonomic alternative that handles this internally, at the cost of lower throughput per calling thread.

submit() remains the primary API. submit_and_wait() makes roda-ledger accessible to a broader range of use cases without compromising the core design.

Decision

Introduce submit_and_wait() as an ergonomic alternative to submit() for callers that prioritize simplicity over throughput. The caller specifies a WaitLevel — the pipeline stage to wait for — and blocks until that stage has processed the transaction, receiving a complete SubmitResult in return.

submit() remains the primary API and the recommended choice for high-throughput workloads. submit_and_wait() trades per-thread throughput for ease of use.

ASYNC is not a valid WaitLevel. Fire-and-forget is served by submit(). A WaitLevel with no waiting is a contradiction — and returning a meaningful SubmitResult is impossible before any pipeline stage has processed the transaction.

WaitLevel

#![allow(unused)]
fn main() {
pub enum WaitLevel {
  Processed,    // Transactor has applied the transaction
  // caller knows immediately: accepted or rejected
  // transaction is already queued for WAL write — it will be
  // persisted durably very soon after this notification
  // the only loss scenario is a crash in the very narrow window
  // between Transactor output and WAL fdatasync completing
  // in practice this window is measured in microseconds

  Committed,    // WAL has been flushed to disk (fdatasync)
  // durable — survives crash
  // standard guarantee for financial transactions

  Snapshotted,  // Snapshotter has applied entries to Vec<AtomicI64>
  // GetBalance is guaranteed to reflect this transaction
  // highest latency — pipeline depth after Committed
}
}

Latency profile

Processed:    nanoseconds — Transactor processes in pipeline order
              first to respond, lowest latency
              transaction queued for WAL immediately — durable within microseconds
              in normal operation. loss only if crash occurs in that narrow window.
              useful for: rejection detection, internal high-throughput pipelines

Committed:    ~676µs avg (WAL fsync latency, group commit amortized)
              durable on disk
              useful for: external payments, receipts, user-facing confirmations

Snapshotted:  Committed latency + Snapshotter pipeline lag
              slightly higher than Committed
              useful for: callers who immediately query GetBalance after submit
                          eliminates read-your-own-writes consistency bugs

Guarantee matrix

Level          Survives crash    GetBalance consistent    Rejection known
─────────────────────────────────────────────────────────────────────────
Processed      almost always*    no                       yes ← immediately
Committed      yes               no                       yes
Snapshotted    yes               yes ← guaranteed         yes

* transaction queued for WAL immediately after Processed notification.
  loss only if crash occurs in the microsecond window before WAL fdatasync.

Ledger API (amends ADR-003)

#![allow(unused)]
fn main() {
pub struct SubmitResult {
  pub tx_id:       u64,
  pub status:      TxStatus,
  pub fail_reason: Option<FailReason>,
}

pub enum TxStatus {
  Committed,
  Error,
}

// existing — unchanged
pub fn submit(&self, op: Operation) -> u64;

// new — blocks until WaitLevel reached
pub fn submit_and_wait(&self, op: Operation, level: WaitLevel) -> SubmitResult;
}

submit_and_wait() blocks the calling thread until the specified pipeline stage has processed the transaction. On error (rejection by Transactor), returns immediately with TxStatus::Error and the appropriate FailReason regardless of the requested WaitLevel — there is no point waiting for Committed or Snapshotted if the transaction was rejected at Processed.

Wait Mechanism

Two distinct waiting paths

Ledger API (submit_and_wait) — blocking, for non-async callers:

Calls submit() then wait_for_transaction(tx_id, level) internally. wait_for_transaction is a blocking call that uses the pipeline’s existing wait_strategy — the same adaptive spin/yield/park used by pipeline stages via PressureManager. No extra threads, no tokio involvement.

#![allow(unused)]
fn main() {
pub fn submit_and_wait(&self, op: Operation, level: WaitLevel) -> SubmitResult {
  let tx_id = self.submit(op);
  self.wait_for_transaction(tx_id, level);
  self.get_transaction_status(tx_id)
}
}

When no transactions are advancing, wait_strategy backs off using:

#![allow(unused)]
fn main() {
self.config.wait_strategy.wait_strategy(retry_count);
retry_count += 1;
}

This prevents CPU spinning at 100% when the pipeline is idle or slow — the same adaptive pressure management used throughout the pipeline.

gRPC server — async, tokio-native:

The gRPC server does not use wait_for_transaction. Instead, a single background tokio task continuously polls last_compute_id(), last_commit_id(), and last_snapshot_id(). When any value advances, it wakes all waiters whose tx_id is now satisfied at their requested level.

This means:

• One poller drives all concurrent gRPC waiters
• Waiters sleep until woken — zero CPU between notifications
• Pipeline stages are completely untouched
• wait_strategy applies to the poller when no progress is detected — preventing the poller from spinning at 100% when the pipeline is idle

Pipeline stages untouched

last_compute_id(), last_commit_id(), and last_snapshot_id() are read-only observations of existing monotonic counters already maintained by each pipeline stage. No changes to Transactor, WAL, or Snapshotter internals. The wait mechanism is entirely in the Ledger and gRPC layers.

gRPC API (amends ADR-004)

enum WaitLevel {
  PROCESSED   = 1;
  COMMITTED   = 2;
  SNAPSHOTTED = 3;
}

message SubmitOperationRequest {
  Operation  operation  = 1;
  WaitLevel  wait_level = 2;   // if omitted → use existing SubmitOperation (async)
}

message SubmitOperationResponse {
  uint64    tx_id       = 1;
  uint32    fail_reason = 2;   // 0 if status == COMMITTED
}

// New RPC — blocks until WaitLevel reached
        rpc SubmitAndWait(SubmitOperationRequest) returns (SubmitOperationResponse);

// Existing RPC — unchanged, fire and forget
        rpc SubmitOperation(SubmitOperationRequest) returns (SubmitOperationResponse);

The existing SubmitOperation RPC returns immediately with tx_id only. SubmitAndWait blocks the gRPC stream until the specified WaitLevel is reached, then returns the full SubmitResult.

gRPC handler calls submit_and_wait() on a spawn_blocking thread — the blocking call must not park a tokio async task.

Batch Submit

submit_batch_and_wait() submits multiple operations and waits until all reach the specified WaitLevel:

#![allow(unused)]
fn main() {
pub fn submit_batch_and_wait(
  &self,
  ops:   Vec<Operation>,
  level: WaitLevel,
) -> Vec<SubmitResult>;
}

Each operation in the batch gets its own PendingWait. The caller blocks until all are notified. At Committed level, group commit naturally covers entire batches under a single fdatasync() — efficient.

rpc SubmitBatchAndWait(SubmitBatchRequest) returns (SubmitBatchResponse);

message SubmitBatchRequest {
  repeated Operation operations = 1;
  WaitLevel          wait_level = 2;
}

message SubmitBatchResponse {
  repeated SubmitOperationResponse results = 1;
}

Usage Patterns

User-facing payment — Committed

#![allow(unused)]
fn main() {
// external payment, return receipt to user
let result = ledger.submit_and_wait(
Operation::Transfer { from: 1, to: 2, amount: 500 },
WaitLevel::Committed,
);

let success = result.fail_reason != 0;
}

Immediate balance read — Snapshotted

#![allow(unused)]
fn main() {
// caller needs consistent balance immediately after submit
let result = ledger.submit_and_wait(
Operation::Deposit { account: 1, amount: 1000 },
WaitLevel::Snapshotted,
);

// guaranteed: GetBalance reflects this deposit
let balance = ledger.get_balance(1);
}

High-throughput with fast rejection — Processed

#![allow(unused)]
fn main() {
// internal pipeline — want fast rejection feedback, durability not critical
let result = ledger.submit_and_wait(
Operation::Transfer { from: 1, to: 2, amount: 999_999 },
WaitLevel::Processed,
);

if result.status == TxStatus::Error {
// know immediately — no polling needed
handle_rejection(result.fail_reason);
}
}

Consequences

Positive

• Callers choose the durability/latency tradeoff explicitly per submission
• No polling loop needed — submit_and_wait() blocks until the right stage
• Committed level maps directly to the standard financial transaction guarantee
• Snapshotted eliminates read-your-own-writes consistency bugs for balance queries
• Processed gives fast rejection feedback without durability overhead
• On rejection — all wait levels return immediately, no wasted pipeline time
• Group commit at Committed level batches multiple waiters under one fdatasync()
• submit() unchanged — existing fire-and-forget callers unaffected
• Pipeline stages completely untouched — wait mechanism is in Ledger and gRPC layers
• wait_strategy prevents CPU spinning when pipeline is idle — adaptive backoff
• One background tokio task drives all gRPC waiters — zero per-waiter polling

Negative

• submit_and_wait() blocks the calling thread — not suitable for async contexts directly, use the gRPC path for async callers
• Snapshotted latency is non-deterministic — depends on Snapshotter pipeline lag
• Timeout handling not defined — caller responsible for wrapping in a timeout

Neutral

• WaitLevel has no ASYNC variant — submit() covers fire-and-forget
• Batch submit follows the same pattern — no new mechanism needed
• gRPC SubmitOperation unchanged — existing clients unaffected
• last_compute_id(), last_commit_id(), last_snapshot_id() are read-only observations — no pipeline stage modifications required

Alternatives Considered

Polling-based approach (existing) Insufficient for general use — callers must implement polling, handle timeouts, and reason about pipeline stage semantics. submit_and_wait() internalizes this.

Single blocking level (Committed only) Rejected — Processed is genuinely useful for high-throughput pipelines that want fast rejection feedback without durability cost. Snapshotted eliminates a real class of consistency bugs. Three levels cover the three meaningful pipeline boundaries.

yield_now() per waiter Rejected — 10K concurrent waiters each calling yield_now() in a loop is effectively 10K busy-waiters. One background poller driving all waiters via waker notifications is strictly better.

Modifying pipeline stages for notification Rejected — pipeline stages are the performance-critical core. Adding notification logic couples the wait mechanism to the pipeline internals. Reading existing monotonic ID counters from outside the pipeline is the correct separation.

References

• ADR-003 — Ledger API, Operation enum, submit() (amended by this ADR)
• ADR-004 — gRPC interface, SubmitOperation RPC (amended by this ADR)
• ADR-006 — WAL Sync thread, group commit, fdatasync
• ADR-008 — SnapshotMessage queue, mpsc::sync_channel(1) oneshot pattern
• src/ledger.rs — submit_and_wait(), PendingWait, pending_waits DashMap
• src/transactor.rs — Processed notification, rejection notification
• src/storage/wal.rs — Committed notification after fdatasync
• src/snapshot.rs — Snapshotted notification after balance update

ADR-014: WASM Function Registry and Function Operation Execution

Status: Proposed
Date: 2026-04-16
Author: Taleh Ibrahimli

Amends:

• ADR-003 — removes Operation::Composite, implements Operation::Function execution path
• ADR-001 — adds WalEntryKind::FunctionRegistered (kind=5)
• ADR-006 — adds function_snapshot_{N}.bin to storage layout
• ADR-001, ADR-003, ADR-009, ADR-010 — renames fail_reason to status across all structs, APIs, and documentation (see Decision: Rename fail_reason to status)

Context

Operation::Function was reserved in ADR-003 and ADR-004 as a forward-compatible extension point for user-defined logic. The placeholder existed but had no implementation.

Operation::Composite was the interim escape hatch — a caller-defined sequence of Credit and Debit steps executed atomically. It solves the immediate problem but has significant limitations:

• Anonymous — no name, no audit trail beyond entries
• Inline — defined per-call, not reusable
• No versioning — no way to know which logic produced which entries
• No registry — cannot be inspected, listed, or managed
• Bypasses the registry entirely — invisible to operators

With a proper Function registry, Composite has no remaining purpose. It is removed by this ADR; the proto field number is reserved and no in-process variant remains.

The missing piece was a safe, high-performance runtime for user-defined logic that:

• Executes exactly once in the Transactor — consistent with ADR-001 entries-based model
• Cannot escape the ledger’s invariants — sandboxed API surface
• Is registered by name and versioned — auditable, reusable
• Survives crash recovery — WAL-recorded, snapshot-backed
• Supports any language that compiles to WASM — Rust, AssemblyScript, C, Zig
• Executes atomically — failure causes full rollback, same as all other operations

Decision

Runtime: wasmtime

wasmtime is the WASM runtime. Rationale:

• Bytecode Alliance backed — production-grade, actively maintained
• Used by Fastly for untrusted code execution at the edge — strongest safety story
• Component Model support — clean interface types between host and WASM
• Best-in-class Rust API — Engine, Module, Store, Linker
• Hardware-accelerated JIT — minimal per-execution overhead

wasmer rejected — API instability, weaker safety guarantees for untrusted code.

Decision: Rename fail_reason to status

fail_reason is renamed to status across the entire project. This is a breaking change applied as part of this ADR.

Rationale: fail_reason implies failure is the primary case. status is more accurate — it represents the outcome of a transaction, which may be success (0) or a specific condition code. This aligns with how the field is actually used: success is status = 0, any non-zero value is a condition that caused the transaction not to proceed.

Scope of rename:

• TxMetadata.fail_reason → TxMetadata.status
• FailReason type → Status
• FailReason::NONE → Status::NONE
• FailReason::INSUFFICIENT_FUNDS → Status::INSUFFICIENT_FUNDS
• All other FailReason constants follow the same pattern
• gRPC fail_reason fields → status
• All ADR references updated
• transaction.fail_reason → transaction.status

The u8 representation, value ranges (0=success, 1-127 standard, 128-255 user-defined), and all existing constant values are unchanged. This is a rename only.

Host API Surface

The WASM function can only call three host functions. Nothing else is exposed:

#![allow(unused)]
fn main() {
// namespace: "ledger"
fn credit(account_id: u64, amount: u64);   // guaranteed, no return
fn debit(account_id: u64, amount: u64);    // guaranteed, no return
fn get_balance(account_id: u64) -> i64;
}

credit and debit return nothing. They are guaranteed operations — they update the Transactor’s balance Vec directly. They cannot fail individually. The zero-sum invariant is enforced after execution completes, as with all operations.

Negative balance handling is the responsibility of user code. If the function requires overdraft protection, it must call get_balance before or after applying credits/debits and return a non-zero Status to trigger rollback. The ledger does not enforce negative balance checks inside WASM functions — this is intentional, as some use cases (liability accounts, internal system accounts) legitimately require negative balances.

Integer-only interface — no memory sharing, no pointers, no strings. All WASM value types map directly to native types.

The host functions close over the Transactor’s balance Vec during execution — same mutable reference used by all built-in operations.

Function Entry Point

Every WASM function must export a single entry point with exactly 8 i64 parameters:

#![allow(unused)]
fn main() {
// Rust
#[no_mangle]
pub extern "C" fn execute(
   param1: i64, param2: i64, param3: i64, param4: i64,
   param5: i64, param6: i64, param7: i64, param8: i64,
) -> u8;
}

// AssemblyScript
export function execute(
    param1: i64, param2: i64, param3: i64, param4: i64,
    param5: i64, param6: i64, param7: i64, param8: i64,
): u8;

Why exactly 8 params: Fixed arity eliminates the need for linear memory pointer/length passing. 8 i64 values cover all practical financial use cases — account IDs, amounts, rates, flags, timestamps. Unused params are passed as 0. This keeps the ABI simple, safe, and verifiable at registration time.

Return value: 0 = success. Any non-zero value is a Status — the same u8 type used throughout the system. Values 0–127 map to standard Status constants. Values 128–255 are user-defined custom reasons, consistent with the existing design.

Atomicity: Functions execute atomically. If the function returns a non-zero status, all credits and debits applied during execution are rolled back. The Transactor’s rollback mechanism is identical to that used for built-in operations.

Example: Fee Deduction

#![allow(unused)]
fn main() {
// Rust — compile with: cargo build --target wasm32-unknown-unknown --release
#[link(wasm_import_module = "ledger")]
extern "C" {
    fn credit(account_id: u64, amount: u64);
    fn debit(account_id: u64, amount: u64);
    fn get_balance(account_id: u64) -> i64;
}

// param1 = sender account
// param2 = receiver account
// param3 = fee account
// param4 = amount
// param5 = fee_rate (basis points, e.g. 50 = 0.5%)
#[no_mangle]
pub extern "C" fn execute(
    param1: i64, param2: i64, param3: i64, param4: i64,
    param5: i64, _: i64, _: i64, _: i64,
) -> u8 {
    let sender   = param1 as u64;
    let receiver = param2 as u64;
    let fee_acct = param3 as u64;
    let amount   = param4 as u64;
    let fee      = (amount * param5 as u64) / 10_000;

    unsafe {
        credit(sender, amount + fee);
        debit(receiver, amount);
        debit(fee_acct, fee);
    }
    0 // success
}
}

// AssemblyScript — compile with: asc function.ts -o function.wasm --optimize
@external("ledger", "credit")
declare function credit(account_id: u64, amount: u64): void;

@external("ledger", "debit")
declare function debit(account_id: u64, amount: u64): void;

@external("ledger", "get_balance")
declare function get_balance(account_id: u64): i64;

export function execute(
    param1: i64, param2: i64, param3: i64, param4: i64,
    param5: i64, _p6: i64, _p7: i64, _p8: i64,
): u8 {
    const sender   = param1 as u64;
    const receiver = param2 as u64;
    const feeAcct  = param3 as u64;
    const amount   = param4 as u64;
    const fee      = (amount * (param5 as u64)) / 10_000;

    credit(sender, amount + fee);
    debit(receiver, amount);
    debit(feeAcct, fee);

    return 0; // success
}

Determinism

Functions must be pure and deterministic — same params always produce the same credits/debits. This is enforced architecturally:

• Host API exposes no randomness, no wall-clock time, no I/O
• No WASM threads, no atomics
• wasmtime fuel metering can limit execution cycles

Future Raft replication: when distributed replication is added, only the leader will execute WASM functions. Entries produced by execution will be replicated as binary WAL data to followers. Followers will apply entries directly without re-executing the function. Determinism is required for this model to be correct — non-deterministic functions would produce divergent entries across replicas.

Registration Path

gRPC RegisterFunction(name, binary)
  → validate WASM binary (wasmtime::Module::validate)
  → verify binary exports execute(i64×8) -> i32
  → compute CRC32C of binary
  → write to disk: functions/{name}_v{N}.wasm
  → write WAL record FunctionRegistered directly to active segment (bypasses Transactor)
  → Snapshot stage receives WAL record, updates FunctionRegistry
  → Pipeline atomicLastFunctionSequence incremented
  → Transactor detects sequence change on next cycle, clones registry from Arc<RwLock>
  → return: version number, crc32c

Registration bypasses the Transactor. It is not a financial transaction — no entries, no zero-sum invariant. WAL write goes directly to the active segment, same path used by crash recovery markers.

WAL Record: FunctionRegistered (40 bytes)

#![allow(unused)]
fn main() {
#[repr(C)]
#[derive(Copy, Clone, Debug, Pod, Zeroable)]
pub struct FunctionRegistered {
    pub entry_type: u8,        // 1 @ 0  — WalEntryKind::FunctionRegistered (5)
    pub _pad0:      u8,        // 1 @ 1
    pub version:    u16,       // 2 @ 2  — monotonic per name, starts at 1, max 65535
    pub crc32c:     u32,       // 4 @ 4  — CRC32C of WASM binary; 0 = unregistered
    pub name:       [u8; 32],  // 32 @ 8 — snake_case, UTF-8, null-padded, max 32 bytes
}                              // total: 40 bytes
}

Name constraints: snake_case, ASCII alphanumeric + underscore, must start with a letter. Validated at registration time. Max 32 bytes — sufficient for any meaningful Rust function name. Null-padded to fill the field. No length field needed — scan to first null byte.

crc32c = 0 signals unregistered. Two signals always agree:

• WAL record has crc32c = 0
• File on disk is 0 bytes (empty file)

On recovery, if they disagree, the WAL record is the source of truth.

File path is derived deterministically from name and version:

functions/{name}_v{version}.wasm

No path stored in WAL — always reconstructible from name and version.

WasmRuntime

WasmRuntime is Arc<WasmRuntime> in the pipeline — shared between the Ledger facade, Snapshot stage, and Transactor. All wasmtime concerns are encapsulated in src/wasm_runtime.rs.

#![allow(unused)]
fn main() {
pub struct WasmRuntime {
    engine:     wasmtime::Engine,
    handlers:   RwLock<HandlerMap>,        // name → FunctionCaller
    update_seq: AtomicU32,                 // starts at 0, increments on every change
}

/// Carries both the CRC32C of the WASM binary and the callable handler.
/// CRC32C is embedded in TxMetadata.tag at execution time — single map lookup
/// gives both the audit identifier and the callable, zero extra lookups.
pub struct FunctionCaller {
    pub crc32c:  u32,
    pub handler: Arc<dyn Fn([i64; 8]) -> u8 + Send + Sync>,
}

pub type HandlerMap = HashMap<String, FunctionCaller>;
}

Public API:

#![allow(unused)]
fn main() {
impl WasmRuntime {
    // Validates binary — checks execute(i64×8)->u8 export is present and well-formed.
    // Called by Ledger before writing binary to disk during register_function.
    pub fn validate(&self, binary: &[u8]) -> Result<(), std::io::Error>;

    // Validates CRC32C and binary content, compiles, inserts FunctionCaller into HandlerMap.
    // Increments update_seq. crc32c is stored in FunctionCaller for use in TxMetadata.tag.
    // Called by Snapshot stage when WalEntry::FunctionRegistered is committed.
    pub fn load_function(&self, name: &str, binary: &[u8], crc32c: u32) -> Result<(), std::io::Error>;

    // Removes handler from HandlerMap. Increments update_seq.
    // Called by Snapshot stage when WalEntry::FunctionRegistered { crc32c: 0 } is committed.
    pub fn unload_function(&self, name: &str) -> Result<(), std::io::Error>;

    // Returns current update sequence number.
    pub fn last_update_seq(&self) -> u32;

    // Returns owned clone of handler map. Called by Transactor on seq change.
    pub fn get_handler_map(&self) -> HandlerMap;
}
}

WasmRuntime has no knowledge of disk, WAL, or pipeline. It only manages compiled handlers in memory. validate is called by Ledger before disk write. load_function is called by the Snapshot stage after WAL commit — it validates CRC and binary, then compiles and stores the handler with its CRC32C for embedding in TxMetadata.tag during execution.

One Engine per Ledger — shared, thread-safe, JIT compiled once per function at registration. One Store per execution — created inside FunctionCaller, dropped immediately after.

Engine is Send + Sync. WasmRuntime is Send + Sync. FunctionCaller is Send + Sync.

Function storage is handled by src/storage/functions.rs — separate from WasmRuntime. WasmRuntime receives binary data; it does not read or write disk directly.

Transactor Hot Cache

The Transactor holds an owned local copy of HandlerMap. It never acquires the RwLock during transaction execution — zero lock contention on the hot path.

On every cycle (not every step) the Transactor checks wasm_runtime.last_update_seq(). If the seq has advanced, it calls wasm_runtime.get_handler_map() which acquires a read lock, clones the map, and returns. The clone is rare — only on function registration or unregistration.

#![allow(unused)]
fn main() {
// Transactor cycle (simplified)
fn run_step(&mut self, ctx: &TransactorContext) {
    // Per-cycle check — single atomic load, no lock
    let seq = ctx.wasm_runtime().last_update_seq();
    if seq != self.local_handler_seq {
        self.local_handlers = ctx.wasm_runtime().get_handler_map(); // owned clone
        self.local_handler_seq = seq;
    }

    // Process transactions using local_handlers — no locks, no Arc overhead
    // ...
}
}

Registration Flow

ledger.register_function(name, binary, override)        ← Ledger owns this flow
  → if function exists and override=false → return error
  → wasm_runtime.validate(binary) → check execute(i64×8)->u8 export
  → storage/functions.rs: write functions/{name}_v{N}.wasm to disk
  → push WalEntry::FunctionRegistered onto pipeline WAL input queue
  → return version number to caller

Snapshot stage receives WalEntry::FunctionRegistered
  → storage/functions.rs: read functions/{name}_v{version}.wasm from disk → binary
  → wasm_runtime.load_function(name, binary, entry.crc32c)
      → verify CRC32C(binary) == crc32c → reject if mismatch
      → validate binary (execute(i64×8)->u8 export)
      → compile via Engine
      → insert FunctionCaller { crc32c, handler } into HandlerMap
      → increment update_seq

Transactor picks up on next cycle — update_seq changed → clone handlers → ready.

Terminology:

• register_function — Ledger operation: validation + disk write + WAL entry. Async.
• load_function — WasmRuntime operation: compiles binary, inserts handler into memory. Called by Snapshot on commit.

Execution Flow

Operation::Function { name, params: [i64; 8], user_ref }
  → Transactor receives
  → dedup check on user_ref (same as all operations)
  → lookup FunctionCaller from local_handlers by name
  → if not found → Status::INVALID_OPERATION
  → build tag: [b'f', b'n', b'w', b'\n', crc32c[0..4]]
      caller.crc32c identifies exactly which function version executed
      cross-reference with FunctionRegistered WAL records for full audit
  → meta(tx_id, tag, user_ref, timestamp)
  → (caller.handler)(params) → returns u8
      → WASM calls credit/debit/get_balance via host functions
      → host functions update self.balances directly
      → entries accumulate in Transactor as normal
  → if return != 0 → rollback, Status::from_u8(return_value)
  → zero-sum invariant verified (as always)
  → entries written to WAL

Tag format in WAL output (roda-ctl unpack):

{"type":"TxMetadata","tx_id":441001,"tag":"fnw\\n4a2f1c3d",...}

"fnw\n" — human-readable marker identifying a Function/WASM call. 4a2f1c3d — CRC32C of the WASM binary, identifies exact version executed. Auditors cross-reference with FunctionRegistered records to resolve name and version.

Unregister Flow

ledger.unregister_function(name)                        ← Ledger owns this flow
  → storage/functions.rs: truncate functions/{name}_v{N}.wasm to 0 bytes
  → push WalEntry::FunctionRegistered { crc32c: 0, name, version+1 } onto pipeline WAL input queue
  → return  ← handler still active until WAL entry is committed

Snapshot stage receives WalEntry::FunctionRegistered { crc32c: 0 }
  → wasm_runtime.unload_function(name)
      → remove FunctionCaller from HandlerMap
      → increment update_seq

Transactor picks up on next cycle — handler gone → subsequent Function { name } → Status::INVALID_OPERATION.

Terminology:

• unregister_function — Ledger operation: disk truncation + WAL entry. Async.
• unload_function — WasmRuntime operation: removes handler from memory. Called by Snapshot on commit.

Two signals always agree after unregistration:

• WAL record crc32c = 0
• File on disk is 0 bytes — truncated, not deleted, audit trail preserved

WAL record is the source of truth on recovery if signals disagree.

Override flag on RegisterFunction:

message RegisterFunctionRequest {
  string name     = 1;
  bytes  binary   = 2;   // WASM binary, max 4MB
  bool   override = 3;   // if true, adds new version even if name exists
}

Without override=true, registering an existing active name returns an error.

Function Snapshot

Separate file from balance snapshot. Same Seal thread, same snapshot_frequency trigger, same LZ4 compression and sidecar CRC pattern.

data/
  function_snapshot_{N}.bin
  function_snapshot_{N}.crc

Format:

Header (36 bytes):
  magic(4)          0x46554E43 "FUNC"
  version(1)        1
  segment_id(4)
  function_count(8) number of function records
  pad(19)

Body (per function, 40 bytes each, LZ4 compressed):
  name(32)          snake_case, UTF-8, null-padded
  version(2)        latest version number, u16
  crc32c(4)         CRC32C of WASM binary; 0 = unregistered
  _pad(2)

Sidecar .crc (16 bytes):
  file_crc32c(4)
  file_size(8)
  magic(4)          0x46554E43

Recovery:

1. Load function_snapshot_{N}.bin → for each function record with crc32c != 0:
   • Read functions/{name}_v{version}.wasm from disk → binary
   • wasm_runtime.load_function(name, binary, snapshot.crc32c) → verifies CRC and binary internally, aborts recovery on mismatch
2. Replay WAL FunctionRegistered records after snapshot:
   • If crc32c != 0 → read binary from disk → wasm_runtime.load_function(name, binary, entry.crc32c)
   • If crc32c == 0 → wasm_runtime.unload_function(name)
3. update_seq reflects final state after all loads/unloads

Operation::Composite Removal

Composite is removed by this ADR. The Operation enum no longer has a Composite variant, and proto field number 4 is permanently reserved on SubmitOperationRequest and SubmitAndWaitRequest.

Migration: any Composite operation is expressible as a WASM function. The Rust SDK provides a template (examples/custom_transaction.rs).

gRPC API Additions

service Ledger {
  rpc RegisterFunction(RegisterFunctionRequest)     returns (RegisterFunctionResponse);
  rpc UnregisterFunction(UnregisterFunctionRequest) returns (UnregisterFunctionResponse);
  rpc ListFunctions(ListFunctionsRequest)           returns (ListFunctionsResponse);
}

message RegisterFunctionRequest {
  string name     = 1;
  bytes  binary   = 2;   // WASM binary, max 4MB (enforced by max_message_size_bytes)
  bool   override = 3;
}

message RegisterFunctionResponse {
  uint32 version = 1;
  uint32 crc32c  = 2;
}

message UnregisterFunctionRequest {
  string name = 1;
}

message UnregisterFunctionResponse {
  uint32 version = 1;   // version that was unregistered
}

message ListFunctionsRequest {}

message ListFunctionsResponse {
  repeated FunctionInfo functions = 1;
}

message FunctionInfo {
  string name    = 1;
  uint32 version = 2;
  uint32 crc32c  = 3;
}

Operation::Function already exists in the proto (ADR-004) — no change needed for invocation.

Storage Layout Addition

data/
  ...existing files...
  function_snapshot_000004.bin    ← new
  function_snapshot_000004.crc    ← new

functions/
  fee_calculation_v1.wasm
  fee_calculation_v2.wasm
  compliance_check_v1.wasm

The functions/ directory lives alongside data/. Path: {storage.data_dir}/../functions/ or configurable via StorageConfig.

Source Layout

src/
  wasm_runtime.rs          ← WasmRuntime, FunctionCaller, HandlerMap
  storage/
    functions.rs           ← read/write .wasm files, CRC32C, truncate on unregister
    ...existing files...
  ...existing files...

Cargo.toml additions:
  wasmtime = { version = "...", features = ["cranelift"] }

Consequences

Positive

• User-defined financial logic without recompiling or redeploying the ledger
• Sandboxed — WASM cannot escape the host API
• Auditable — every registration is a WAL record with CRC, every invocation produces standard entries
• Versioned — multiple versions per function, override path, clean unregistration
• Language agnostic — Rust, AssemblyScript, C, Zig, any WASM target
• Atomicity guaranteed — failure causes full rollback, identical to built-in operations
• Transactor hot path lock-free — local cache, sequence check only
• Future Raft compatible — leader executes, entries replicated, followers apply directly
• Composite removed — one extensibility model, not two
• Single WAL source of truth — function registration ordered relative to transactions
• Dual-signal unregistration — WAL crc32c=0 + 0-byte file, WAL wins on conflict
• No name_len or status flag fields — simpler struct, less surface area
• 32-byte name — sufficient for any meaningful snake_case Rust function name
• fail_reason → status rename — clearer semantics project-wide

Negative

• wasmtime JIT compilation at registration — ~10-100ms per function (one-time cost)
• Store creation per Function operation — small allocation on hot path
• 8-param limit — covers all practical cases but not unlimited flexibility
• Removing Composite is a breaking API change for any caller still using it
• functions/ directory adds operational surface to manage
• fail_reason → status is a breaking rename across all structs, APIs, and proto fields

Neutral

• Registration bypasses Transactor — correct, consistent with crash recovery markers
• Function snapshot is a separate file — consistent with balance snapshot separation
• WasmRuntime is a separate struct/module — clean separation of concerns
• Unregistered function files truncated to 0 bytes — audit trail preserved, no deletion

References

• ADR-001 — entries-based execution model, single execution in Transactor
• ADR-003 — Operation enum, Composite removal, Named reserved
• ADR-004 — gRPC interface, Function operation proto
• ADR-006 — WAL segment lifecycle, storage layout, snapshot format
• ADR-009 — Status (formerly FailReason), user-defined range 128-255
• ADR-011 — WAL write/commit separation, active segment direct write
• ADR-013 — transaction-count-based segments, active window
• wasmtime — https://github.com/bytecodealliance/wasmtime
• Bytecode Alliance — https://bytecodealliance.org
• WASM spec linear memory — https://webassembly.github.io/spec/core/syntax/modules.html#memories

ADR-018: Fault Injection Framework

Status: Proposed Date: 2026-05-27 Author: Taleh Ibrahimli

Context

roda-ledger’s correctness story rests on a stack of testing layers, each effective at a different scope:

• Pure state-machine tests in crates/raft (crates/raft/tests/common/mod.rs) — the deterministic simulator covers election safety, log matching, and leader completeness with no IO or network at all.
• Single-component integration tests in crates/ledger and crates/storage (crates/ledger/tests/corruption_tests.rs, crates/ledger/tests/tail_partial_recovery_test.rs) — drive a real Ledger against a real disk, but only the deliberate faults the test author constructs by hand (renaming files, byte-editing a WAL).
• In-process cluster tests (crates/cluster/tests/idle_cpu_test.rs uses ClusterTestingControl) — bring up a real 1-to-N node cluster on threads of the test process. Real consensus, real replication, real-enough IO. No way to force a specific fault pattern.
• E2E scenarios (ADR-012) — real processes, real disks, real network. Slowest signal; faults surface only when timing happens to align.

The structural gap: nothing between layers 2 and 3 lets a test deterministically subject the live consensus + replication stack to adversarial IO or network conditions. As the system matures, every new class of failure that gets discovered tends to manifest first at the e2e layer — slow to detect, slow to reproduce, slow to fix. We need a deterministic harness that makes failure modes a first-class input to a test, not an accident of timing.

The decision: introduce a fault-injection framework at two layers (Ledger and Cluster), plus a MockNode family that lets tests drive the real consensus / replication code against programmable IO and network. Goal: every failure mode the system should survive has a corresponding fast, deterministic integration test that forces it.

Decision

Three independent additions, each behind the workspace fault-injection Cargo feature so production builds compile to the same machine code as today:

1. LedgerFaultInjector — a seam in crates/storage that wraps disk IO with fault hooks. Lets tests drive each disk call on the axis { Normal, Error, Slow, Stuck }, plus out-of-band file-level events (lose a file, corrupt a sealed segment, overwrite a byte range) at the exact call sites that handle WAL, snapshot, and raft-durable data.

2. ClusterFaultInjector — an injection layer in crates/cluster that intercepts (a) outbound replication-driver messages between leader and followers and (b) the role-state transitions inside the consensus driver. Lets tests drop / truncate / reorder / duplicate / delay messages and force role-flip moments.

3. MockNode family — MockClusterNode, MockLeaderNode, MockFollowerNode types in crates/cluster/src/testing/mock/, alongside the existing ClusterTestingControl. Each is a stripped-down node that runs the real consensus / replication-driver logic against a programmable event source instead of real disk + network. Builds a deterministic harness for scenarios that today require spinning a full process cluster.

All three are gated by a single workspace fault-injection feature. CI runs cargo test --features fault-injection once to exercise the union.

Architecture

Ledger-level (LedgerFaultInjector)

A pluggable injector installed on Storage (or equivalently on Segment) that the IO call sites consult before / after touching the disk. The hooks are conditionally compiled — when the fault-injection feature is off they collapse to nothing.

Injection points — exactly the call sites that touch disk for a distinct data class, no more:

• Segment::write_pending_entries — the WAL writer’s append.
• Segment::sync — fdatasync on the active segment.
• WalTailer::tail — the replicator’s read of its own WAL.
• TermStorage::append / VoteStorage::append — raft durability.

Picking these higher-level entry points (rather than every File::write_at / read_at) keeps production code unchanged and the test surface explicit. Adding a new injection point is a deliberate ADR amendment, not an accidental side effect of refactoring storage internals.

Fault model: each IO call has an outcome on the axis { Normal, Error, Slow, Stuck }. Stuck is “block until the test explicitly unstucks it” — distinct from Slow, which completes on its own. Data-at-rest can additionally be corrupted (in-place mutation between calls) or lost (file-level removal). The two axes are orthogonal: a write can be Stuck and the file it would have written to can be Lost from underneath it.

Per-operation faults

Every injection point supports these outcomes. The “operation” column names the natural method on the injector trait the first implementation PR will land:

StuckWrite / StuckSync use a control channel exposed by the injector: the test holds a handle and calls unstuck(call_id) to release a parked operation. This is what lets a test set up “leader’s WAL writer is parked on fdatasync; now kill the leader; restart it; assert the unfinished write was discardable” without racing wall-clock timing.

File-level events (out of band)

These are not per-IO faults — they’re test-driven mutations of the on-disk state between calls. The injector exposes them as direct methods the test invokes from the outside:

WalFileCorrupted and SegmentDataOverride take a precise byte- level intent so a regression test can pinpoint the exact site of any past corruption shape we want to keep guarding against.

Reproducibility

Every injector takes a seeded StdRng. A scenario spec is a Vec<(call_count_predicate, FaultDecision)> plus an ordered list of file-level events keyed by call-count milestones, so e.g. “return WriteCorruption on the 7th WalTailer::tail call and StuckSync on the 3rd Segment::sync until call 12” is expressible and deterministic across runs.

Trait shape and exact signatures are deferred to the first implementation PR; this ADR commits only to where the hooks live and what classes of fault they must support.

Cluster-level (ClusterFaultInjector)

ClusterFaultInjector composes LedgerFaultInjector — it owns one and exposes it as cluster_injector.ledger() so any disk-level fault is reachable from a cluster-scope test without juggling two handles. A cluster scenario like “lose wal.bin while leader is mid-election” is one configuration on a single injector.

Five fault buckets, all configurable per-node and behind the same workspace fault-injection feature. The first three reuse the { Normal, Error, Slow, Stuck } axis from the Ledger layer; the last two are message- and consensus-shaped.

1. DiskAccessFault — raft-durable IO

Term-log and vote-log writes share their own fault bucket because their failure modes have consensus-level meaning (a vote that isn’t durable before the RPC reply violates §5.1). Backed by the Ledger injector’s TermStorage::append / VoteStorage::append hooks; surfaced at cluster scope as a single named bucket so tests don’t need to know which file backs which side.

2. Leader push faults

Wraps run_peer_sender in crates/cluster/src/consensus/replication_driver.rs.

3. Follower replication faults

Wraps apply_wal_update (and the surrounding receive loop) on the follower side.

4. Message-layer faults (MessageInjector)

Wraps the outbound side of run_peer_sender and the inbound side of apply_wal_update at the message envelope, before the apply hook in bucket 3.

5. Consensus-state faults (ConsensusInjector)

Wraps Consensus::self_advance, RaftNode::election_start, and the role-transition path. Can force a role change, suppress a heartbeat, or inject a stale term — the “make the leader think it’s still leader for one tick longer than reality” class of fault.

Includes a QuorumStaleMatchIndex knob: force a peer’s slot in Quorum to report a stale match_index for one tick, so the cached-majority’s monotonicity (fetch_max) is exercised under bad input.

6. Election / handshake faults

Election and handshake have different correctness invariants than steady-state replication (§5.1, §5.3, §5.4) and earn their own bucket so tests don’t have to express them through generic MessageInjector rules.

7. Liveness / health (PingInjector)

Ping (proto/node.proto) is the operational health-check RPC and lives outside the consensus hot path. Dropping or delaying pings creates “node looks dead to monitors but is alive internally” scenarios that the rest of the framework can’t express.

8. Network partitions (asymmetric)

A named primitive on top of MessageInjector::Drop filtered by (peer, direction). Real partitions are often one-way at the application layer, and composing them by hand from Drop rules gets verbose fast.

Tests pair an injector that causes a fault with a recorder that observes what the system did, so a single test can assert both “the fault was injected” and “the system responded correctly.” Production calls compile to nothing.

Time and process-control primitives

Cross-cutting helpers that don’t belong inside a single injector bucket but are needed to make scenarios deterministic:

• ClockSkew(node, d) — applies a fixed per-node offset to every Instant::now() read through a wrapped clock. Election timers, heartbeat intervals, and the WAL committer’s pacing are all wall-clock-driven; without per-node skew the framework can’t deterministically stress timer-ordering edge cases.
• ClockJump(node, d) — one-shot jump forward by d, used to trip a timer once without leaving the clock skewed.
• KillMidFsync / KillMidElectionRound / KillMidHandshake — named helpers that install the corresponding Stuck* fault, wait until the system enters it, then process::abort the target. Doable by hand today via LedgerFaultInjector::StuckSync + external abort, but a named primitive makes the scenario readable.

Implementation uses tokio::time::pause plus a per-node MockClock injected into the Consensus / Ledger time-read paths. Both helpers are no-ops without the fault-injection feature.

Future buckets (placeholders, not v1)

• InstallSnapshotChunkLoss / Delay — once the InstallSnapshot handler in crates/proto/proto/node.proto moves past UNIMPLEMENTED, snapshot-transfer faults get the same { Drop, Truncate, Delay } treatment as the steady-state message layer.

MockNode family

Three layers, each a thin shim around the real component:

• MockClusterNode — owns a real Consensus + a real Ledger but replaces the network layer with a controllable shim and optionally swaps the disk layer for an in-memory or fault-injected variant. Drives scenarios like “leader sends one WalUpdate then dies; bring up a fresh node; assert the WAL is recoverable.”

• MockFollowerNode — drives apply_wal_update directly, bypassing the gRPC layer. The test owns a mpsc::Sender that feeds synthetic ReplicationLeaderMessages. Lets a test construct exact wire-byte sequences (well-formed, malformed, reordered, duplicated) and observe how the receive path responds.

• MockLeaderNode — drives run_peer_sender against a crafted WAL the test sets up directly. Lets a test answer “given leader’s WAL with this specific layout, what bytes does the tailer ship to the follower?” without spinning up the full pipeline.

Reuse what already exists:

• The raft Sim (crates/raft/tests/common/mod.rs) is the state-machine layer beneath the MockNodes — they sit on top of RaftNode, the same way Consensus sits on top of RaftNode in production.
• mem_persistence.rs (crates/raft/tests/common/mem_persistence.rs) is the template for in-memory raft persistence.
• ClusterTestingControl::cluster(N) stays as the higher-tier harness for cases that genuinely need real processes. MockNodes are an additional tier, not a replacement.

Test patterns the framework unlocks

The framework enables a library of deterministic scenarios that today either don’t exist or live only as flaky e2e probes. A non-exhaustive catalogue, sorted by which layer drives them:

Ledger-only (LedgerFaultInjector)

• WAL append fails part-way; restart; assert recovery + balance.
• fdatasync returns EIO; assert leader exits cleanly rather than acking a non-durable commit.
• Active segment is FileLost while the ledger is running; assert recovery on next start either rebuilds from snapshot or fails loudly.
• A sealed segment is corrupted at a precise byte offset (DataLoss::SegmentDataOverride); assert the per-segment CRC detects it on next load.
• SlowSync raises sync latency to 200 ms under sustained writes; assert the WAL committer back-pressures the writer instead of silently dropping work.
• StuckWrite parks the WAL writer; concurrent reads must observe the pre-stuck snapshot; unstuck; assert the writer resumes without corruption.

Cluster-only (ClusterFaultInjector)

• MessageInjector::Drop on every other AppendEntries; assert the leader’s retry path eventually converges quorum.
• MessageInjector::Reorder swaps two consecutive WalUpdates; assert the follower either rejects out-of-order delivery or applies them in the correct sequence.
• MessageInjector::Truncate cuts a WalUpdate mid-batch; assert the follower handles partial delivery (reject, retry, or carry-over) without corrupting its on-disk state.
• ConsensusInjector forces a stale term on the next outbound message; assert the receiving leader steps down cleanly.
• LeaderPushStuck parks a peer’s push; assert the cached quorum-majority continues to advance from other peers, and unstuck restores normal replication without state divergence.
• FollowerReplicationError rejects every other WalUpdate; assert the leader’s retry/backoff converges and next_tx doesn’t skip records.
• DiskAccessFault::Stuck on vote.log during an election round; unstuck after the candidate timed out; assert no two leaders ever appear in the same term.
• RequestVoteLie::HigherTerm against a candidate; assert candidate steps down to Initializing and observes the higher term durably.
• HandshakeReject(LogMismatch) returned to a leader’s first contact; assert the leader follows the §5.3 truncate-after watermark and re-handshakes.
• PingDrop continuously; assert the cluster’s gRPC handler still serves submits (Ping isn’t on the commit path) and the monitor’s “node down” judgement is reachable from another signal.
• Partition::OneWay { from: leader, to: peer }; assert the leader’s view stalls quorum on that peer while the peer can still pull from the other live peer; heal; assert convergence.

Time / crash-point primitives (cross-cutting)

• ClockSkew two followers’ clocks 100 ms apart; assert their randomised election timers still defeat split-vote storms (i.e. the randomisation range exceeds the skew).
• ClockJump past the next heartbeat deadline; assert the leader re-arms the wakeup correctly rather than spinning on a stale past-deadline (the class of bug we caught in next_pending_wakeup).
• KillMidFsync while the leader has acked a commit; restart; assert the on-disk WAL is recoverable and the surviving cluster agrees on the highest committed tx_id.

MockNode-driven cross-layer

• Craft a MockFollowerNode that receives wire bytes from a scripted source; kill+restart between deliveries; assert recovery + balance hold across every interleaving.
• MockClusterNode plus LedgerFaultInjector::StuckSync on the leader: trigger a leader election while the previous leader’s fsync is parked; unstuck after the election; assert no two leaders ever commit conflicting tx_ids.

Each scenario is a deterministic unit / integration test runnable on every CI push. The library accumulates over time so every class of failure the system has been shown to survive is locked in by an explicit test.

This is the inverse of the e2e strategy in ADR-012: e2e proves the system holds together under realistic conditions; fault injection proves the protocol holds together under adversarial ones. The two are complementary.

Open questions

To be resolved in the first implementation PR:

• Production overhead. The ADR commits to “production builds compile to the same machine code as today.” The implementation chooses the mechanism (trait monomorphisation, cfg-guarded matches, etc.) that demonstrably achieves this — verified by reading the generated cargo asm output for at least the WalTailer::tail and Segment::write_pending_entries paths.

• Determinism boundary inside Mock tests. Recommend tokio::time::pause so timing-dependent state transitions (e.g. the consensus loop’s tokio::time::sleep_until) are reproducible without depending on wall-clock. To be confirmed when the first MockNode test that depends on timer behaviour lands.

• gRPC interception layer. ClusterFaultInjector currently sits above the gRPC layer (deciding what ReplicationLeaderMessages to send / accept). Sitting below (corrupting bytes on the wire, frame-level faults) is needed for tonic-level bugs. Recommendation: start above, defer below until motivated by a concrete need.

Out of scope

• Production chaos engineering (Pumba, ChaosMesh equivalents). That sits at the e2e layer and is covered — or not — by ADR-012.
• Generative fuzzing (proptest / bolero) of full scenarios. This framework is the substrate that would make such fuzzing practical; the fuzz harness itself is its own ADR.
• Compile-time invariant checking (Loom, Shuttle). Same answer — the substrate is here; the harness is separate.
• Library-grade fail / failpoints integration. We adopt the same design pattern but a project-local injector keeps the surface narrow and tied to roda-ledger’s specific call sites.
• Memory / resource exhaustion (allocator failure injection, RSS caps). The test harness can’t safely intercept alloc without unsafe global state, and forcing OOM mid-run reduces to external ulimit + process kill — better expressed at the e2e layer.

ADR-019: Transaction Ring — Single-Writer Inter-Stage Transport

Status: Accepted
Date: 2026-06-05
Author: Taleh Ibrahimli

Amended by:

• ADR-020 — removes the in-ring metadata back-fill (patch) step described below; the metadata is now assembled complete and published with a single push as the transaction’s trailing commit record.
• ADR-021 — moves the ring releaser from the snapshot to the WAL (release-on-ingest) and has the snapshot tail the durable WAL instead of reading the ring; the ring becomes single-reader (SPSC) and reclamation is gated on ingest, not durability. The “Durability-gated reclamation” and “snapshot is the sole releaser” sections below are superseded.

Amends:

• ADR-006 — replaces the queue-based hand-off between the transactor, WAL, and snapshot stages, and ties ring-slot reclamation to the durability watermark.
• ADR-011 — the WAL no longer forwards committed entries to the snapshot stage; both stages read the shared ring independently.

Context

Before this change the pipeline moved WAL records between stages with two bounded queues:

• transactor → WAL — the transactor staged a transaction’s records in a temporary buffer, then enqueued them; the WAL stage dequeued and persisted them.
• WAL → snapshot — after a record became durable, the WAL stage re-enqueued it onto a second queue that the snapshot stage consumed.

This shape had three structural costs:

1. Copies and staging. Every record was copied at least twice (into the transactor→WAL queue, then into the WAL→snapshot queue), and the transactor had to build each transaction in a separate staging buffer before it could enqueue — an allocation and a copy per transaction.
2. Coupling. The snapshot stage could only observe records the WAL stage chose to forward. The two consumers were chained rather than independent, so the snapshot’s view was a function of the WAL’s forwarding cadence.
3. No single reclamation invariant. Durability (WAL fdatasync) and consumption (snapshot indexing) were tracked in different places, with the queues acting as the only backpressure signal.

The transactor produces a strictly ordered stream of records, and the WAL and snapshot stages each need to read every record, at their own pace, for different purposes (persistence vs. indexing). That is a single-producer, multiple-reader problem — a shared ring fits it far better than a chain of copying queues.

Decision

Introduce a single shared, lock-free transaction ring as the one transport between the transactor and the read stages, replacing both queues. The transactor also uses the ring directly as its per-transaction scratch space, eliminating the staging buffer.

Roles around the ring

              writes (sole producer)            copy-out reads
  transactor ───────────────►  TX RING  ──────────────────────────►  WAL (persist + fsync)
                                  ▲   │  ──────────────────────────►  snapshot (index)
                                  │   │
                  reclaims slots  │   └── release frontier (slot reuse)
                  (sole releaser) └────────────────────────────────  snapshot

• One producer — the transactor is the only writer. It appends records into ring slots and publishes them.
• Many copy-out readers — the WAL and snapshot stages each read the published stream independently, each at its own position. They copy records out of slots rather than borrowing them, because a slot may be reused once it is released.
• One releaser — the snapshot stage is the sole party that advances the reclamation point, and it does so only up to the durability watermark.

Absolute addressing

Positions in the ring are absolute, monotonically increasing indices starting at zero (mapped to physical slots only on access), rather than wrapping counters. A write returns the absolute index it landed at; the transactor uses that index to operate on the records of the in-flight transaction, and each reader tracks its own absolute cursor. Absolute indices make “where is X” and “how far has each party progressed” unambiguous across wraps.

The transactor builds transactions in place

Instead of staging a transaction in a side buffer, the transactor writes its metadata and sub-items directly into the ring’s uncommitted region — the slots it owns exclusively and that no reader can see yet. Within that region it:

• validates the transaction (e.g. the zero-sum invariant) by walking its own uncommitted records,
• back-fills the metadata once the sub-item count and checksum are known, and
• on success, publishes the whole transaction atomically by advancing the write frontier in a single step.

On failure it discards the uncommitted tail, so a rejected or partial transaction is never visible to any reader. Readers therefore only ever observe complete, validated transactions — publish-on-commit visibility.

Durability-gated reclamation (the core invariant)

A slot’s lifecycle is:

1. the transactor writes a record into the slot (uncommitted, private to the writer);
2. the transactor commits the transaction → the slot becomes visible to readers;
3. the WAL reads it, persists it, and fdatasyncs → the durability watermark advances;
4. the snapshot reads it (only once it is at or below the durability watermark) and indexes it;
5. the snapshot, as releaser, advances the reclamation point past the slot → the transactor may reuse it.

Because the releaser only advances to the durability watermark, the frontiers are always ordered write ≥ persisted ≥ durable ≥ indexed = reclaimed. A slot is reused only after the record it held is durable on disk and has been consumed by every reader. This makes “never overwrite a record that isn’t yet durable” a single, structural property of the transport rather than something each stage must enforce.

Caller-driven, non-blocking backpressure

The producer never sleeps inside the ring. It reserves a window of free slots, fills it, and reclaims more space as the releaser frees it. When no space is available the transactor’s own retry/wait-strategy loop decides how to back off, and it can abort cleanly on shutdown. Backpressure is thus expressed at the call site, not hidden inside the transport — consistent with the pipeline’s non-blocking design.

Consequences for the read stages

• The WAL stage reads the ring and persists; it no longer owns an input queue and no longer forwards anything to the snapshot.
• The snapshot stage reads the ring, indexes for query serving, and drives reclamation; its remaining queue carries only query requests, not records.
• The intermediate message types that wrapped records for the two queues are removed.

Consequences

Positive

• Fewer copies, no per-transaction staging. Records are written once into the ring; each reader copies out only what it consumes. The transactor’s separate staging buffer is gone.
• In-place transaction assembly. Validation, checksumming, and rollback all operate on the ring’s uncommitted region — the ring is the transactor’s scratch space for the in-flight transaction.
• Independent readers. The WAL and snapshot consume the same published stream at their own pace; the snapshot no longer depends on the WAL’s forwarding cadence.
• One reclamation invariant. Slot reuse is gated on durability and consumption in a single place, so the “don’t overwrite undurable data” guarantee is structural.
• Backpressure stays caller-driven and the producer never blocks inside the transport.

Negative

• Concentrated unsafe code. A lock-free ring with hand-managed memory ordering and a strict window discipline is inherently unsafe; correctness rests on those invariants and warrants careful, isolated review.
• A transaction must fit in the ring. Because a transaction is published atomically, the ring capacity must be at least the largest possible single transaction; otherwise the writer cannot make progress.
• Readers cannot hold references across reclamation. Consumers must copy records out; they cannot retain borrows into ring slots past the release boundary.
• Durability-coupled backpressure. If durability stalls, reclamation stalls, and the producer eventually blocks once the ring fills. This is the same coupling any bounded transport has, now explicitly tied to the durability watermark.

Neutral

• Replaces the transactor→WAL and WAL→snapshot queues and their wrapper message types; the snapshot’s query queue is unchanged in purpose.
• The reclamation party is fixed to the snapshot stage by design — it is the last reader and the one that observes the durability watermark.

References

• ADR-001 — Entries-based execution model (the record types the ring carries)
• ADR-006 — WAL, Snapshot, and Seal Durability (durability lifecycle; amended)
• ADR-011 — WAL Write/Commit Separation (WAL→snapshot forwarding; amended)
• ADR-005 — Adaptive Pipeline Execution Mode (pipeline staging and backpressure context)
• ADR-010 — Sync Submit (submit-path backpressure semantics)

ADR-020: Trailer Metadata — Commit-Record WAL Transaction Layout

Status: Accepted
Date: 2026-06-06
Author: Taleh Ibrahimli

Amends:

• ADR-006 — changes the on-disk record order within a transaction and the recovery scan
• ADR-019 (Transaction Ring) — removes the in-ring metadata back-fill (patch) step
• ADR-001 — record ordering of the entries-based execution model

Context

Each atomic transaction is a TxMetadata record followed by its sub-items (TxEntry, TxLink, TxTerm, FunctionRegistered). Every record is a fixed 40-byte POD struct whose first byte is the WalEntryKind discriminant. Today the metadata is written first:

TxMetadata, TxEntry, TxEntry, …      (header layout)

The metadata carries two fields that can only be known once the whole transaction has been built: sub_item_count (how many followers belong to it) and crc32c (a checksum over the metadata plus every follower). Because the metadata is written before its followers exist, the writer must go back and patch it after the fact:

#![allow(unused)]
fn main() {
// finalize_committed_meta — transactor.rs
let sub_item_count = head - meta_idx - 1;
self.tx_ring_pusher.patch(meta_idx, |m| { m.sub_item_count = …; m.crc32c = 0; });
self.tx_ring_pusher.walk(meta_idx, head, |entry| { /* fold CRC over meta + followers */ });
self.tx_ring_pusher.patch(meta_idx, |m| { m.crc32c = digest; });   // back-patch again
}

Two structural costs follow from “metadata first”:

1. Write-side back-patching. The metadata is touched three times — written, then patched for the count, then patched again for the CRC computed by walking the followers (transactor.rs:263-290). This is the sole reason TxRingWriter::patch exists (mutate an already-written, not-yet-published slot). The checksum simply cannot be finalized until the followers are on the ring.

2. Trusting an unverified count on read. Recovery reads the leading metadata, takes its sub_item_count on faith to know how far to scan, and only then checks that that many valid followers are present and that the CRC matches (recover.rs:118-180). Detecting a partial tail means asking “did all the followers the header promised actually arrive?” — but the header is the very thing being validated.