E2E harnesses (simulation + production)

Part I — Simulation: SimCluster, EventKey, and single_shard_tx_sim. Part II — Production: per-shard shard-loop, RPC ingress, production E2E (same module—read both before hands-on labs).

Learning objectives

• Trace single_shard_tx_sim from SimCluster through ShardLoop::step to commit and execution
• Trace runner_boots_listens_and_shuts_down_cleanly through ProductionRunner::build, genesis, and the pinned event loop
• State how production differs from simulation (wall-clock, validator QUIC/gossipsub vs wallet HTTP, RocksDB, Tokio + pinned thread, batch flushing)
• Explain what a pinned shard-loop thread is, how run_shard_loop relates to it, and how CPU affinity is applied with core_affinity::set_for_current

Simulation vs production (same core, different harness)

Shared core: Simulation uses SimCluster + NodeHost with deterministic ShardEvent scheduling; production uses per-shard ShardLoop::step on dedicated threads. Both drive the same NodeStateMachine—so SubmitTransaction, shard consensus, BlockCommitted, and execution hooks match once an event is delivered. The harness decides how events arrive and when time advances.

Pinned thread, run_shard_loop, and CPU affinity

What “pinned thread” means here

Below is the actual production entrypoint: a named std::thread, optional CPU affinity via core_affinity::set_for_current, then run_shard_loop. Read the code first; captions spell out what each part testifies to.

Snippet context: spawn_shard_loop — pinned-thread entry (file in study table).

pub fn spawn_shard_loop(
    shard_loop: ProdShardLoop,
    config: PinnedLoopConfig,
) -> std::thread::JoinHandle<()> {
    std::thread::Builder::new()
        .name("shard-loop".to_string())
        .spawn(move || {
            // Try to pin to core 0
            if let Some(core_ids) = core_affinity::get_core_ids() {
                if let Some(&core_id) = core_ids.first() {
                    if core_affinity::set_for_current(core_id) {
                        info!(?core_id, "Pinned shard-loop thread to core");
                    } else {
                        warn!("Failed to pin shard-loop thread to core 0");
                    }
                }
            }

            run_shard_loop(shard_loop, config);
        })
        .expect("failed to spawn shard-loop thread")
}

• Thread (not Tokio): std::thread::Builder + .name("shard-loop") + .spawn — this is an OS thread whose closure only calls run_shard_loop after the optional pin.
• Affinity + syscall: set_for_current(core_id) runs inside that new thread; the info!(…) line means the affinity syscall succeeded, not that the core is yours alone.

Snippet context: Tokio task after spawn_shard_loop — metrics and shutdown select! (runner file in study table).

        // ── 3. Spawn pinned thread ───────────────────────────────────────
        let loop_handle = spawn_shard_loop(shard_loop, pinned_config);

        // ── 4. Metrics + shutdown loop ───────────────────────────────────
        let mut metrics_tick = tokio::time::interval(Duration::from_secs(1));
        metrics_tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
        let mut shutdown_rx = self.shutdown_rx.take().expect("shutdown_rx already taken");

        loop {
            tokio::select! {
                biased;
                _ = &mut shutdown_rx => {
                    info!("Shutdown signal received");
                    break;
                }
                _ = metrics_tick.tick() => {
                    self.collect_metrics();
                }
            }
        }

The same binary keeps a Tokio loop here (metrics, shutdown); libp2p/RPC/timers live on that runtime too. NodeInput for consensus reaches the shard-loop thread through crossbeam channels (timer_rx / callback_rx / consensus_rx in PinnedLoopConfig) read inside run_shard_loop — see the next subsection.

Diagram: logical CPUs, pinned target, and what runs where

Schematic for one validator process on a typical multi-core host. Affinity affects only the shard-loop std::thread; Tokio worker threads are usually unpinned and the OS may move them across logical CPUs.

Pinning does not reserve a core exclusively for this process unless you add stronger isolation (cgroups, isolated CPUs, etc.).

What run_shard_loop actually does

Read in-repo: run_shard_loop (spawned from spawn_shard_loop in the same module) — path in study table.

It is the main engine loop for advancing the ShardLoop while the node is up. The very first thing inside the function (before the big loop) is to apply genesis-time timer ops once:

• Where they come from: On the Tokio side, ProductionRunner::maybe_initialize_genesis() builds the genesis block, calls initialize_genesis on the state machine, runs handle_actions / flush, drains timer_ops from that step, then steps StateCommitComplete at height 0 (syncing BFT with the real JVT root) and merges any more timer ops. That vector becomes PinnedLoopConfig.initial_timer_ops. (Implementation on the production-runner side — study table.)
• What “apply once” means: run_shard_loop does for op in take(initial_timer_ops) { timer_mgr.process_op(op); } a single time at startup. Each TimerOp (commonly Set for the initial proposal timer) tells ProdTimerManager to spawn a Tokio sleep; when it fires, it sends a timer NodeInput on crossbeam timer_rx. So “once” is one drain at loop entry, not “timers only fire once.”

After that, the loop repeats until shutdown:

• Check for shutdown on the shutdown channel.
• Set wall-clock time on the loop (shard_loop.set_time(now)).
• Receive the next NodeInput with a priority order on crossbeam channels (timer, then callback, then consensus); if nothing is ready, block on crossbeam::select! with a timeout derived from nearest_batch_deadline().
• For each event: shard_loop.step(event), then apply TimerOps from the step (Tokio-backed sleeps that re-inject timer events into the timer channel).
• flush_expired_batches(wall_clock_duration()) — see Why simulation flushes everything; production batches below.
• Periodic metrics, channel depth export, optional JVT GC (run_jvt_gc).

Why simulation flushes everything; production batches until deadlines

Read in-repo: flush_expired_batches, flush_all_batches, nearest_batch_deadline, and step caller-protocol docs — study table.

The ShardLoop holds batch accumulators (notably validation and committed-header verification) with time-based deadlines. The two harnesses choose different flush policies after step:

• Simulation — flush_all_batches() flushes all pending batches and related paths every time, ignoring whether a deadline has arrived. The runner is single-threaded and fully owns simulated time. Flushing aggressively makes the state after each step “fully settled,” which simplifies assertions and avoids subtle ordering bugs when run_until jumps the clock—work is not left sitting in a batch whose expiry might fall awkwardly between synthetic events. Throughput of the fake network is not a goal, so paying the cost of an immediate flush every step is cheap and predictable.
• Production — flush_expired_batches(now) only calls flush_validation_batch / flush_committed_header_verifications when now has passed each batch’s deadline. Reason: real validators see bursty traffic (gossip, RPC); batching until a short deadline amortizes verification work, reduces spikes on the pinned thread and worker pools, and acts as a form of backpressure—the pipeline does not have to react to every packet edge individually. The same file exposes nearest_batch_deadline() so run_shard_loop can set the crossbeam::select! timeout: the loop wakes when a batch must flush even if no new NodeInput arrived, keeping latency bounded without flushing blindly after every event.

flush_all_batches() still exists on ShardLoop for paths that need an immediate drain (e.g. shutdown); production’s steady-state path prefers deadline-driven flushes.

For a broader ladder of multitasking / threading terms mapped to hyperscale-rs (processes, async vs blocking, pools, backpressure), see the Basic module State Machines & Event-Driven Architecture (section on threading and concurrency).

1. Simulation E2E: single_shard_tx_sim

2. Production E2E-style: runner_boots_listens_and_shuts_down_cleanly

3. Mental model: one picture

Where to go next

• Transaction Flow — full user→finality diagram aligned with these steps.
• Performance Measurement — hyperscale-sim binary vs production metrics (different from these integration tests).
• More scenarios (cross-shard, conflict cycles, crash): other integration tests in the simulation crate (e.g. cross_shard_tx_sim in the same scenarios target) use the same SimCluster pattern with different configs/assertions — paths in study table.
• Hands-on: Single-Shard Simulation E2E — improved/innovative test case — edit the same test, add trace markers, assert synchronized heights, and correlate stats with simulated time.

Quiz

Pass threshold: 70%.

Reading module (3/4): what the production harness uses, where user / RPC input begins, and what canonical production E2Es can show today—after the simulation pair.

Where user input begins (production)

What runner_boots_listens_and_shuts_down_cleanly shows: RocksDB + libp2p + pinned loop + shutdown—not a full signed tx path (hands-on module next adds your markers).

Learning objectives

• Map RPC and tx_submission_sender ingress to consensus_rx
• Trace runner_boots_listens_and_shuts_down_cleanly and run_shard_loop
• State what production E2E can observe without submitting a transaction

Simulation vs production (same core, different harness)

Shared core: Simulation uses SimCluster + NodeHost with deterministic ShardEvent scheduling; production uses per-shard ShardLoop::step on dedicated threads. Both drive the same NodeStateMachine—so SubmitTransaction, shard consensus, BlockCommitted, and execution hooks match once an event is delivered. The harness decides how events arrive and when time advances.

Pinned thread, run_shard_loop, and CPU affinity

What “pinned thread” means here

Below is the actual production entrypoint: a named std::thread, optional CPU affinity via core_affinity::set_for_current, then run_shard_loop. Read the code first; captions spell out what each part testifies to.

Snippet context: spawn_shard_loop — pinned-thread entry (file in study table).

pub fn spawn_shard_loop(
    shard_loop: ProdShardLoop,
    config: PinnedLoopConfig,
) -> std::thread::JoinHandle<()> {
    std::thread::Builder::new()
        .name("shard-loop".to_string())
        .spawn(move || {
            // Try to pin to core 0
            if let Some(core_ids) = core_affinity::get_core_ids() {
                if let Some(&core_id) = core_ids.first() {
                    if core_affinity::set_for_current(core_id) {
                        info!(?core_id, "Pinned shard-loop thread to core");
                    } else {
                        warn!("Failed to pin shard-loop thread to core 0");
                    }
                }
            }

            run_shard_loop(shard_loop, config);
        })
        .expect("failed to spawn shard-loop thread")
}

• Thread (not Tokio): std::thread::Builder + .name("shard-loop") + .spawn — this is an OS thread whose closure only calls run_shard_loop after the optional pin.
• Affinity + syscall: set_for_current(core_id) runs inside that new thread; the info!(…) line means the affinity syscall succeeded, not that the core is yours alone.

Snippet context: Tokio task after spawn_shard_loop — metrics and shutdown select! (runner file in study table).

        // ── 3. Spawn pinned thread ───────────────────────────────────────
        let loop_handle = spawn_shard_loop(shard_loop, pinned_config);

        // ── 4. Metrics + shutdown loop ───────────────────────────────────
        let mut metrics_tick = tokio::time::interval(Duration::from_secs(1));
        metrics_tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
        let mut shutdown_rx = self.shutdown_rx.take().expect("shutdown_rx already taken");

        loop {
            tokio::select! {
                biased;
                _ = &mut shutdown_rx => {
                    info!("Shutdown signal received");
                    break;
                }
                _ = metrics_tick.tick() => {
                    self.collect_metrics();
                }
            }
        }

The same binary keeps a Tokio loop here (metrics, shutdown); libp2p/RPC/timers live on that runtime too. NodeInput for consensus reaches the shard-loop thread through crossbeam channels (timer_rx / callback_rx / consensus_rx in PinnedLoopConfig) read inside run_shard_loop — see the next subsection.

Diagram: logical CPUs, pinned target, and what runs where

Schematic for one validator process on a typical multi-core host. Affinity affects only the shard-loop std::thread; Tokio worker threads are usually unpinned and the OS may move them across logical CPUs.

Pinning does not reserve a core exclusively for this process unless you add stronger isolation (cgroups, isolated CPUs, etc.).

What run_shard_loop actually does

Read in-repo: run_shard_loop (spawned from spawn_shard_loop in the same module) — path in study table.

It is the main engine loop for advancing the ShardLoop while the node is up. The very first thing inside the function (before the big loop) is to apply genesis-time timer ops once:

• Where they come from: On the Tokio side, ProductionRunner::maybe_initialize_genesis() builds the genesis block, calls initialize_genesis on the state machine, runs handle_actions / flush, drains timer_ops from that step, then steps StateCommitComplete at height 0 (syncing BFT with the real JVT root) and merges any more timer ops. That vector becomes PinnedLoopConfig.initial_timer_ops. (Implementation on the production-runner side — study table.)
• What “apply once” means: run_shard_loop does for op in take(initial_timer_ops) { timer_mgr.process_op(op); } a single time at startup. Each TimerOp (commonly Set for the initial proposal timer) tells ProdTimerManager to spawn a Tokio sleep; when it fires, it sends a timer NodeInput on crossbeam timer_rx. So “once” is one drain at loop entry, not “timers only fire once.”

After that, the loop repeats until shutdown:

• Check for shutdown on the shutdown channel.
• Set wall-clock time on the loop (shard_loop.set_time(now)).
• Receive the next NodeInput with a priority order on crossbeam channels (timer, then callback, then consensus); if nothing is ready, block on crossbeam::select! with a timeout derived from nearest_batch_deadline().
• For each event: shard_loop.step(event), then apply TimerOps from the step (Tokio-backed sleeps that re-inject timer events into the timer channel).
• flush_expired_batches(wall_clock_duration()) — see Why simulation flushes everything; production batches below.
• Periodic metrics, channel depth export, optional JVT GC (run_jvt_gc).

Why simulation flushes everything; production batches until deadlines

Read in-repo: flush_expired_batches, flush_all_batches, nearest_batch_deadline, and step caller-protocol docs — study table.

The ShardLoop holds batch accumulators (notably validation and committed-header verification) with time-based deadlines. The two harnesses choose different flush policies after step:

• Simulation — flush_all_batches() flushes all pending batches and related paths every time, ignoring whether a deadline has arrived. The runner is single-threaded and fully owns simulated time. Flushing aggressively makes the state after each step “fully settled,” which simplifies assertions and avoids subtle ordering bugs when run_until jumps the clock—work is not left sitting in a batch whose expiry might fall awkwardly between synthetic events. Throughput of the fake network is not a goal, so paying the cost of an immediate flush every step is cheap and predictable.
• Production — flush_expired_batches(now) only calls flush_validation_batch / flush_committed_header_verifications when now has passed each batch’s deadline. Reason: real validators see bursty traffic (gossip, RPC); batching until a short deadline amortizes verification work, reduces spikes on the pinned thread and worker pools, and acts as a form of backpressure—the pipeline does not have to react to every packet edge individually. The same file exposes nearest_batch_deadline() so run_shard_loop can set the crossbeam::select! timeout: the loop wakes when a batch must flush even if no new NodeInput arrived, keeping latency bounded without flushing blindly after every event.

flush_all_batches() still exists on ShardLoop for paths that need an immediate drain (e.g. shutdown); production’s steady-state path prefers deadline-driven flushes.

For a broader ladder of multitasking / threading terms mapped to hyperscale-rs (processes, async vs blocking, pools, backpressure), see the Basic module State Machines & Event-Driven Architecture (section on threading and concurrency).