Staff-level drill on steps 1–5 of the big-picture tx flow — one Radix tx, one shard, production path to Pending on every validator that admits it.
| Big flow step | This phase |
|---|---|
| 1–2 User sign & submit | § Pipeline steps 1–2 |
| 3 Node receives | § 3–4 RPC + crossbeam + step(SubmitTransaction) |
| 4 Cross-shard determination | Shard set from declared_reads ∪ declared_writes (single-shard = one shard) |
| 5 Mempool | § 8–9 validation + admit_internal |
| 6+ BFT / execution | Phase 2+ |
ShardId in code). Example config only: e.g. 26 shards × 50–100 validators per shard—not fixed by the protocol.ShardLoop + NodeStateMachine (BFT + mempool + execution) + storage + network impl.hyperscale-mempool), not one shared shard mailbox.hyperscale-network-libp2p (libp2p gossipsub over QUIC between validators—separate from the wallet’s HTTP POST in step 2). Simulation uses hyperscale-network-memory — noted only where it differs.Read this before steps 3–9. RPC (step 2) only hits one validator; gossip and mempool answer “how do the rest of the committee get the tx, and what happens on this machine before block building?”
The mempool is not the crossbeam queue, not “the network inbox,” and not where RPC writes when you POST. Those are ingress/scheduling layers (steps 3–5).
Role after validation (steps 8–9): on each validator, MempoolCoordinator holds txs that are cryptographically valid and policy-eligible as Pending candidates. That pool is the local staging area for consensus:
ready_transactions() when building the next block—not from RPC or from “the shard’s global pool.”Pending so they can vote on a block that includes it, track expiry/tombstones, and stay consistent with peers.involves_local_shard)—no entry in its pool.So: gossip = spread bytes; mempool = per-validator “ready to be proposed/voted on” set once this node trusts the signature and Radix rules.
“Batch” in Phase 1 is not “the ShardLoop dequeues many crossbeam messages in one turn.” Production still does one NodeInput → one step per loop iteration (step 4).
| “Batch” means | What gets grouped | Why |
|---|---|---|
| TransactionGossip batch (outbound) | Many txs → one TransactionGossip pubsub message per destination shard |
Bursty RPC/gossip ingress would spam one packet per tx. enqueue_tx_for_gossip appends to a per-shard BatchAccumulator; when tx_gossip_max (e.g. 200) or tx_gossip_window is hit, flush_tx_gossip_batch publishes once (step 6). |
| Validation batch (local crypto) | Many txs → one DispatchPool::TxValidation job |
Separate accumulator (validation_batch); amortizes Ed25519/Radix work off the pinned thread. Flush on cap/deadline or flush_expired_batches. |
Loop flush (flush_expired_batches) |
Time-based wakeups in run_shard_loop |
Even if no new NodeInput arrives, the pinned loop wakes at nearest_batch_deadline() so gossip/validation batches do not sit past their window. |
Inbound gossip can already be batched on the wire: one libp2p message may carry several txs; the handler loops and enqueues one TransactionGossipReceived per tx on crossbeam—still one step per event, each tx then joins the same validation batch accumulator as a local submit.
In-repo proof — outbound gossip batching is per-shard accumulator, not crossbeam mux
// crates/node/src/shard/mod.rs (field docs)
/// Per-destination-shard outbound TransactionGossip accumulators.
/// ... fills until its count cap or time window expires, then flushes
/// as a single batched gossip message.
// crates/node/src/shard/mempool/validation.rs — enqueue_tx_for_gossip
if batch.push(tx, now) { self.flush_tx_gossip_batch(shard); }
// flush → TransactionGossip::new(txs); network.broadcast_to_shard(shard, &gossip);Pending here or on peers, but bandwidth was still spent—by design.
Proof snippets below are trimmed from the hyperscale-rs tree. Ellipsis (...) hides boilerplate; follow data-file links for full sources.
Walk through one transaction the way a person would: sign on the phone, hit a validator’s website API, let that machine copy bytes to its peers and check signatures, then land in a local waiting room (Pending) until Phase 2 proposes a block. Each numbered step tells the story first; the Rust excerpts underneath are receipts.
On the phone or browser, a wallet (or dApp) builds a Radix NotarizedTransactionV1: the transaction intent plus a user signature that proves the account owner authorized it. That user signature uses Ed25519, which is the normal Radix end-user signing scheme—fast, one signature per transaction, verified later by TransactionValidation in the node.
This is not the same cryptography validators use among themselves. Committee votes and quorum certificates use BLS (aggregatable signatures for many validators). Confusing the two is a common onboarding mistake: Phase 1 follows the user tx bytes; Phase 2 follows validator BLS votes and QCs. See also cryptography module for the full split.
The wallet submits the signed transaction to a validator using ordinary HTTP, not the validator-to-validator P2P stack. In production the usual entrypoint is POST /api/v1/transactions on the node’s RPC server (implemented in the production crate). The body carries the transaction as hex or SBOR bytes—the wire encoding the RPC handler decodes before any consensus logic runs.
A phone or browser is not expected to open a libp2p connection or publish to gossipsub topics. Wallets talk HTTP to one (or a few) validators they trust for submission; after the RPC handler accepts the payload, the node forwards work internally. Validator replicas still meet on libp2p later in this phase—that is how peers share the same tx bytes, not how the user’s app first delivers them.
On the HTTP-serving side of the node, the handler decodes the body (hex or SBOR into a typed transaction) and may say “not now” if the machine is overloaded, far behind on sync, or under cross-shard pressure. That is politeness and capacity—not yet “this signature is valid.”
Malformed payloads can fail here; proving the wallet’s Ed25519 signature waits for a worker pool later (step 8). The handler does not write into the mempool directly—it drops a small message on an internal queue so the dedicated consensus thread can pick it up safely without races.
In-repo proof — decode + crossbeam only; comments name the async path (no mempool here)
// crates/production/src/rpc/handlers.rs — submit_transaction_handler
if let Some(rejection) = check_backpressure(&state) { return rejection; }
let transaction = match decode_transaction(&request.transaction_hex) { ... };
// Submit directly to ShardLoop via crossbeam channel.
// ShardLoop will: 1. Gossip 2. Queue batch validation 3. Mempool after validation
state.tx_submission_tx.send(NodeInput::SubmitTransaction { tx: tx_arc }) ...;
// Return immediately - validation and gossip happen asyncDeep inside the node, a single pinned “shard-loop” thread owns BFT, mempool orchestration, and execution hooks. HTTP, libp2p, and worker pools do not touch that state directly—they send notes through three prioritized mailboxes: timers first, then “background job finished,” then everything else (your submitted tx, gossip, votes).
Each lap of the loop reads at most one note, handles it completely, then may flush gossip or validation batches that became due. If a timer, a finished validation, and your submit are all waiting, the timer wins this lap; the others follow in later laps—fast, but never two logical events in one turn.
That is different from “batching txs” in §0b: batching groups many transactions for one network publish or one crypto worker job; it does not merge unrelated internal events into one turn.
In-repo proof — one event per loop iteration; timer > callback > consensus
// crates/production/src/runner.rs — run_shard_loop
// Drains the three crossbeam channels via try_recv in priority order
// (timer_rx > callback_rx > consensus_rx). ...
if let Ok(e) = config.timer_rx.try_recv() { break 'recv Some(e); }
if let Ok(e) = config.callback_rx.try_recv() { break 'recv Some(e); }
if let Ok(e) = config.consensus_rx.try_recv() { break 'recv Some(e); }
// ...
if let Some(event) = event {
let output = shard_loop.step(event); // exactly one NodeInput per iteration
for op in output.timer_ops { timer_mgr.process_op(op); }
}
shard_loop.flush_expired_batches(wall_clock_local());When the consensus thread handles your submission, it kicks off two parallel stories (see §0b for why both matter). Neither finishes in this instant—both play out in later loop turns.
Story 1 — share the bytes. From which shards the tx touches, the node queues outbound copies for each shard’s validator mesh. Sends are batched (count and time limits) so libp2p is not spammed one message per tx; the actual publish is step 6.
Story 2 — verify locally. If this hash is not already being checked, the node queues it for Ed25519 + Radix rules on a worker. Failure comes back as “validations failed”—no mempool. Success comes back as “validated”—then step 8 admits it only if this machine’s shard is involved.
Peers that receive gossip later run the same verification story; relaying does not skip crypto.
Multi-shard nuance: One submission can fan out copies to every shard the tx touches while this machine checks signatures locally. If local checks fail, gossip already sent is not recalled—bad txs become harmless noise that peers reject too.
If checks pass here: this validator keeps the tx only when its shard participates; otherwise it helped relay. Sibling shards admit on their own machines after their own gossip (steps 6–7). Putting txs into blocks is Phase 2; cross-shard execution is Phase 4.
In-repo proof — Track A + B in one handler; validation async via event_sender
// crates/node/src/shard/mempool/validation.rs — handle_submit_transaction
for shard in shards { // reads ∪ writes → shard_for_node
self.enqueue_tx_for_gossip(shard, Arc::clone(&tx));
}
if !self.pending_validation.contains(&tx_hash) && !self.caches.tx_store.contains(&tx_hash) {
self.locally_submitted.insert(tx_hash);
self.pending_validation.insert(tx_hash);
self.queue_validation(tx);
}
// flush_validation_batch — DispatchPool::TxValidation (off pinned thread)
self.dispatch.spawn(DispatchPool::TxValidation, move || {
let results: Vec<bool> = batch.iter()
.map(|tx| validator.validate_transaction(tx).is_ok()).collect();
if valid {
let _ = event_tx.send(NodeInput::TransactionValidated { tx });
} else {
let _ = event_tx.send(NodeInput::TransactionValidationsFailed { hashes: ... });
}
});Gossip publish — batched per shard; may flush on cap or deadline
// same file — flush_tx_gossip_batch
let gossip = TransactionGossip::new(txs);
self.network.broadcast_to_shard(shard, &gossip);When a TransactionGossip batch is ready (queued in step 5, flushed by the network layer or batch timing), the production network stack publishes it to other validators on each involved shard. Under the hood that is libp2p gossipsub over a QUIC transport—validator-to-validator replication, separate from the wallet’s HTTP POST in step 2.
The purpose of gossip here is to copy the same signed transaction bytes to the rest of the committee so each peer can validate and, if appropriate, admit the tx into its own mempool. A common misconception is that gossip “writes into the destination mempool” or that RPC somehow updates remote pools directly. It does neither: every validator still runs validation and admit_internal locally after it receives TransactionGossipReceived (step 7).
In-repo proof — libp2p handler forwards bytes; mempool unchanged here
// crates/node/src/process/network_handlers.rs
self.network.register_gossip_handler::<TransactionGossip>(TopicScope::Shard,
move |gossip: TransactionGossip| -> GossipVerdict {
for transaction in gossip.transactions {
let _ = tx.send(NodeInput::TransactionGossipReceived { tx: transaction });
}
GossipVerdict::Accept
},
);When another validator receives the gossip, libp2p delivers NodeInput::TransactionGossipReceived into its ShardLoop, which feeds the same batched validation pipeline used for locally submitted transactions—unless the hash is already in TxStore or tombstoned (terminal, must not re-admit). Peers do not skip signature or Radix checks just because the tx originated elsewhere.
In-repo proof — gossip ingress uses same queue_validation as submit
// crates/node/src/shard/mempool/validation.rs — handle_gossip_received_tx_for_validation
if !self.caches.tx_store.contains(&tx_hash) && !self.state.mempool().is_tombstoned(&tx_hash) {
self.pending_validation.insert(tx_hash);
self.queue_validation(tx);
}When the worker reports success, the hot thread learns “this tx is cryptographically sound.” That is the first moment the big state machine trusts the payload—not when HTTP returned 200.
Admission into the local waiting pool happens only if this validator’s shard is actually involved in the tx. Others may have relayed and validated the same bytes for the network’s health, but they will not store it in their own mempool—that is validate-and-forward, not “everyone keeps a copy.”
In-repo proof — worker success → feed_event(TransactionValidated)
// crates/node/src/shard/mempool/validation.rs — handle_transaction_validated
self.feed_event(ProtocolEvent::TransactionValidated { tx, submitted_locally });
// crates/node/src/state/participation/transactions.rs — on_transaction_validated
if !self.topology.snapshot().involves_local_shard(&tx) {
return vec![]; // validate-and-relay only
}
self.mempool.on_transaction_gossip(...)If the tx is valid, not expired, not a duplicate, and not tombstoned, this validator keeps it in its in-memory pool as Pending—ready for a future leader to pick (§0b). That pool is staging for Phase 2, not “the blockchain’s mempool in the cloud.”
Typical reject reasons:
validity_range.end is before the node’s current chain time (WeightedTimestamp from committed QCs, not the machine’s wall clock).If admission succeeds, the tx is inserted as Pending in this validator’s BTreeMap<TxHash, PoolEntry> (with ready/deferred tracking for proposer selection later). New pool content can set a latch via queue_ready_proposal(), which tells BFT that fresh material exists—but building and voting on a block still happens in Phase 2, not in this step.
In-repo proof — admission gate is admit_internal; success emits TransactionsAdmitted
// crates/mempool/src/coordinator.rs — on_transaction_gossip
match self.admit_internal(topology, &tx, submitted_locally, now) {
Some(_) => vec![Action::Continuation(ProtocolEvent::TransactionsAdmitted {
txs: vec![tx],
})],
None => vec![],
}Single-shard outcome: For a transaction that only touches one shard, every validator in that shard’s committee that receives and successfully validates the gossip should end with the same hash in Pending inside its own BTreeMap<TxHash, PoolEntry>—there is still no shared global mempool object. A future proposer on that shard will later choose from ready_transactions(); inclusion in a block is not part of Phase 1.
Rough crate map only—no file links. Use the repo tree when you are ready to set breakpoints.
| Concern | Crate (under crates/) | Role in this phase |
|---|---|---|
| RPC ingress | production | HTTP POST /api/v1/transactions; enqueues SubmitTransaction for the ShardLoop—does not write the mempool directly. |
| Thread boundary | production + node | crossbeam Sender<NodeInput> into timer_rx / callback_rx / consensus_rx; pinned loop receives with timer → callback → consensus priority. |
| Orchestrator | node (ShardLoop) | One NodeInput handled per step on the pinned consensus thread. |
| State machine | node (NodeStateMachine) | Composes MempoolCoordinator and BFT; routes ProtocolEvent after validation. |
| Mempool | mempool | Per-validator pool: PoolEntry, ready/deferred sets, tombstones, shared TxStore. |
| Tx validation | engine | Radix rules and signature checks; in production usually on a worker pool, result fed back as events. |
| Gossip payload | types / messages | TransactionGossip message type; batched before publish. |
| Network transport | network, network-libp2p | Trait API (broadcast_to_shard, etc.); production uses gossipsub over QUIC for validator-to-validator replication—not the wallet HTTP path. node registers inbound handlers. |
| Chain time | types | WeightedTimestamp from committed QCs drives validity and tombstones—not NTP on each machine. |
Is there a distinct “RPC client” in the protocol?
No. Any HTTP client can POST. Validators are servers; wallets are clients. Validators also gossip on libp2p to each other.
Where does the validator verify the user Ed25519 signature?
On a DispatchPool::TxValidation worker, after queue_validation / batch flush—not on the HTTP RPC thread and not inside admit_internal.
Flow: RPC decodes and enqueues SubmitTransaction (step 3) → pinned step(SubmitTransaction) schedules validation (step 5) → worker runs TransactionValidation → success returns as TransactionValidated on callback_rx (step 8) → only then mempool admission (step 9). Forged or wrongly signed txs fail in validation; they never reach Pending.
Validator BLS signatures (votes, QCs) are a different path—Phase 2, not user tx ingress.
Why crossbeam instead of pushing straight into the mempool?
Because the mempool is not a public inbox that any thread can append to. On each validator, MempoolCoordinator lives inside NodeStateMachine and is updated only when the pinned ShardLoop thread runs step on a NodeInput or ProtocolEvent. That design keeps BFT, mempool, and execution state in one place without mutexes on every hot path.
The HTTP RPC handler runs on a different thread (Tokio/async world alongside libp2p and validation workers). When a wallet POSTs a transaction, the handler can decode the body and apply backpressure, but it must not call admit_internal directly—that would race with the consensus thread mutating the same maps.
Instead the handler sends a small message on a crossbeam channel: NodeInput::SubmitTransaction { tx }. The ShardLoop thread receives it, runs step(SubmitTransaction), gossips and schedules validation, and only later—after TransactionValidated—does the state machine call mempool admission. So the channel is a thread-safe handoff of work, not “the mempool queue.” The actual pool is still updated synchronously on the owner thread, in the right order with gossip and validation.
If you imagined “RPC writes straight into mempool,” you would need either heavy locking around the whole pool (slow, easy to get wrong) or a second code path that bypasses validation ordering. Hyperscale-rs avoids both by funneling everything through NodeInput → step → events.
Why several crossbeam channels instead of one Sender<NodeInput>?
One FIFO queue would force you to either process strictly in arrival order (bad for liveness when gossip bursts ahead of a fired proposal timer) or scan/reorder on every dequeue. Production splits producers into timer_rx, callback_rx, consensus_rx so run_shard_loop can apply a fixed policy: timer → callback → consensus on each receive.
That matches how work is created: Tokio timer tasks, validation/execution pools, and network/RPC tasks all run elsewhere; each sends a small NodeInput when ready. The pinned thread stays the sole mutator of BFT + mempool + execution orchestration.
What is an ShardLoop::step?
One atomic turn of the validator’s orchestrator on the pinned shard-loop thread. The production runner dequeues exactly one NodeInput from crossbeam (timer, callback, or consensus priority), then calls ShardLoop::step(that_event) in shard/mod.rs.
Inside that single step, the loop does one of two things:
step without entering the big state machine yet. Examples: SubmitTransaction (schedule gossip + validation), TransactionGossipReceived, TransactionValidated (then feed mempool admission).ProtocolEvent (often via NodeInput::Protocol(...)) and call NodeStateMachine::handle, then run every returned Action (network publish, timers, storage, continuations).step returns a small StepOutput (status emissions, timer ops, action counts). It does not drain the whole crossbeam queue. Housekeeping such as flush_expired_batches may run in the same outer run_shard_loop iteration after step, but that is not a second NodeInput.
Mental model: one NodeInput = one “something happened” message; one step = “process that one thing and emit whatever the node must do next.”
Can one round of run_shard_loop process multiple crossbeam events?
No—one outer-loop iteration handles one dequeued NodeInput and one ShardLoop::step. After step, the same iteration may run post-step housekeeping (flush_expired_batches, metrics, optional JVT GC)—that is not consuming another channel message.
If many inputs are pending, the loop may spin through iterations quickly (timer, then callback, then consensus, …), but each iteration still advances the state machine with one logical event. Throughput comes from fast step handlers and worker pools, not from batching unrelated NodeInputs into a single step call.
Does “gossip batch” mean crossbeam delivers many txs in one ShardLoop turn?
No. Crossbeam still delivers one NodeInput per loop iteration. “Gossip batch” means many transactions are packed into one outbound TransactionGossip message per shard via BatchAccumulator (count cap + time window), so libp2p is not flooded with one publish per tx. A separate validation batch groups txs for one worker job—see §0b table.
If the tx only touches my shard, do we skip gossip?
No. Peers still need the bytes. You also validate locally → TransactionValidated without waiting for your own gossip echo.
admit_internal rules on its own MempoolCoordinator—so “arriving at the shard” really means “copied to every committee member who then admits independently.”
Can an attacker submit someone else’s signed tx?
Yes (replay/spam/propagation). Forgery fails TransactionValidation. Censorship: one RPC endpoint can drop traffic — mitigate with multi-submit and honest gossip; protocol needs quorum, not one gateway.
Do validators gossip transactions over libp2p on QUIC in production?
Yes, in production. After a validator packages a TransactionGossip message, the network-libp2p stack publishes it to peers using libp2p gossipsub on a QUIC transport. That is validator-to-validator replication on shard-scoped topics—not the wallet’s HTTP RPC path from step 2.
In simulation and tests, the same message types flow through an in-memory network adapter (fake latency queues, no real QUIC handshake on the validator mesh). The protocol logic and SBOR payloads are the same; only the transport layer changes. When you read traces, “gossip left the node” in prod means gossipsub/QUIC between validators; in sim it means the harness delivered bytes to peers on a deterministic schedule. Neither path is the wallet’s HTTP submit from step 2.
Wallets and dApps can use a single HTTPS URL—easy to document and for users (“send tx to this RPC”).
How mitigation works in practice: the wallet sends the same already-signed bytes to more than one validator RPC (e.g. three endpoints run by different operators). Each POST /api/v1/transactions is independent. If validator A drops your request, B or C may still accept it, run SubmitTransaction, and gossip to the shard. You only need one honest ingress; gossip spreads the tx to the rest of the committee.
Typical patterns: a primary plus fallback RPC list in the wallet; parallel background POSTs; an app backend that fans out; or public infra that load-shares across operators. No resigning is required—the signature is already on the payload.
One URL only: if the user never leaves a censoring RPC, that node never helps—but another endpoint can, without a new signature from the user.
Captured endpoint list: if every RPC URL the wallet knows is run by colluding operators who all drop the same tx, ingress fails. That is curated trust in RPC providers, not a protocol-level fix.
No locks on BFT/mempool; clear happens-before.
All ingress is serialized on one thread: each loop iteration runs one step, so a slow handler delays the next dequeue (timer, callback, or consensus).
Mitigation: Heavy work on worker pools; completion returns as NodeInput on callback_rx. Validation batches flush on deadlines without merging multiple unrelated inputs into one step.
Peers stay consistent; matches pubsub reality.
Bandwidth vs hypothetical direct fanout RPC.
Mitigation: Batched TransactionGossip; receiver dedup.
No central shard bottleneck; parallel admission.
Brief drift until gossip converges.
Mitigation: Same rules + eventual inclusion in a block.
NotarizedTransactionV1 with Ed25519 (wallet/dApp—not validator BLS)POST /api/v1/transactions accepted (hex/SBOR body; optional: submit to several validators)SubmitTransaction on crossbeam → pinned stepTransactionValidatedMempoolCoordinator: Pending on each admitting validatorPass threshold 70%. Read the gotchas and Q&A — several questions are deliberately pedantic.