State Machines & Event-Driven Architecture

Learning Objectives

• Understand the state machine pattern
• Learn event-driven design principles
• Explore Event and Action types in hyperscale-rs
• Understand determinism and why it matters
• See how state machines compose in hyperscale-rs
• Map general threading and concurrency ideas to how hyperscale-rs runs the state machine in production vs simulation

For a first-pass map of message sources and destinations (RPC, gossip, mempool, BFT, execution, timers), transports, typical in-memory stores, how each path is drained, and handlers / callbacks (including the full register_* table), see Transaction flow → Messaging & transports inventory (registry + four sub-tables).

1. What is a State Machine?

Definition

A state machine is a computational model that consists of:

• States - The possible configurations of the system
• Transitions - Rules for moving from one state to another
• Inputs - Events that trigger transitions
• Outputs - Actions produced by transitions

Simple Example

Light Switch State Machine:
  States: {ON, OFF}
  Transitions:
    - If state is OFF and event is "flip" → state becomes ON
    - If state is ON and event is "flip" → state becomes OFF

Why State Machines?

• Deterministic - Same state + same event = same result
• Testable - Easy to test all state transitions
• Predictable - Behavior is well-defined
• Composable - Can combine multiple state machines

2. The StateMachine Trait

Core Interface

In hyperscale-rs, everything implements the StateMachine trait:

trait StateMachine {
    fn handle(&mut self, event: Event) -> Vec<Action>;
    fn set_time(&mut self, now: Duration);
    fn now(&self) -> Duration;
}

Key Properties

• Synchronous - No async, no .await
• Pure - No I/O, no locks, no side effects
• Deterministic - Same inputs always produce same outputs
• Simple - Just state + event → actions

Exploring the Trait

Open crates/core/src/traits.rs and find the StateMachine trait. Notice:

• It's very simple - just three methods
• The handle method takes an event and returns actions
• Time is managed explicitly (for determinism)

3. Events: What Happens

What are Events?

Events are passive data structures that describe something that happened. They flow into state machines.

Event Types

Open crates/core/src/protocol_event.rs to see all event types. Examples:

enum ProtocolEvent {
    // Timers (examples)
    ViewChangeTimer,
    CleanupTimer,

    // Consensus traffic
    BlockHeaderReceived { .. },
    BlockVoteReceived { vote: BlockVote },

    // Pipeline callbacks
    ProposalBuilt { .. },
    QuorumCertificateFormed { .. },

    // ... see crates/core/src/protocol_event.rs
}

Event Sources

• Network - Messages from other validators
• Timers - Scheduled events (view-change liveness, cleanup)
• Internal - Generated by other state machines
• Client - User-submitted transactions

Event Priority

Events at the same timestamp are processed by priority:

1. Internal (0) - Consequences of prior processing
2. Timer (1) - Scheduled timers
3. Network (2) - External messages
4. Client (3) - User submissions

4. Actions: What to Do

What are Actions?

Actions are commands that describe what the state machine wants to do. They flow out of state machines and are executed by runners.

Action Types

Open crates/core/src/action.rs to see all action types. Examples:

enum Action {
    // Network
    BroadcastToShard { shard: ShardId, message: OutboundMessage },
    BroadcastStateVote { shard: ShardId, vote: StateVoteBlock },

    // Timers
    SetTimer { id: TimerId, duration: Duration },

    // Internal
    EnqueueInternal { event: Event },

    // Storage
    CommitBlock { block: Block, qc: QuorumCertificate },

    // ... many more
}

Action Execution

Actions are executed by runners:

• SimulationRunner - Deterministic simulation
• ProductionRunner - Real networking, storage, async

5. Event-Driven Flow

The Flow

1. Runner receives network message
2. Runner creates Event::BlockHeaderReceived
3. Runner calls state_machine.handle(event)
4. State machine processes event, updates state
5. State machine returns Vec<Action>
6. Runner executes actions (send network, store, etc.)
7. Actions may generate new events
8. Loop continues...

Example: Block Proposal Flow

1. New txs/QCs/provisions arrive → node latches try_propose; separately ViewChange timer fires → ProtocolEvent::ViewChangeTimer for round timeout
2. State machine handles → Checks if this node is proposer
3. If proposer → Builds block, returns Action::BroadcastToShard
4. Runner executes → Sends block header to network
5. Other nodes receive → Create Event::BlockHeaderReceived
6. They vote → Return Action::BroadcastToShard (vote)
7. Votes collected → Event::QuorumCertificateFormed
8. QC processed → Chain state updated (latest_qc), commit rule checked (Phase 2; shard crate)

6. Determinism: Why It Matters

What is Determinism?

Deterministic means: given the same initial state and same sequence of events, you always get the same final state and same sequence of actions.

Why Determinism Matters

• Testing - Can reproduce bugs exactly
• Debugging - Can replay execution step-by-step
• Simulation - Can test in controlled environment
• Consistency - All nodes produce same results

What Breaks Determinism?

• ❌ Random number generation (without seed)
• ❌ System time (use explicit time parameter)
• ❌ Network timing (simulate with delays)
• ❌ Thread scheduling (single-threaded in state machine)

How Hyperscale-rs Maintains Determinism

• ✅ Explicit time parameter (set_time)
• ✅ No random number generation in state machine
• ✅ Synchronous execution (no async in state machine)
• ✅ Deterministic event ordering (priority system)

7. Composing State Machines

NodeStateMachine

The NodeStateMachine composes multiple sub-state machines:

NodeStateMachine
├── BeaconCoordinator        (beacon chain: epochs, topology)
├── ShardCoordinator         (per-shard HotStuff-2)
├── ExecutionCoordinator     (execution + cross-shard waves; ConflictDetector inside)
├── MempoolCoordinator       (transaction pool)
├── ProvisionCoordinator     (cross-shard provisions)
├── OutboundProvisionTracker   (ACK until execution certs)
└── RemoteHeaderCoordinator  (remote headers)

How They Compose

1. NodeStateMachine receives an event
2. Routes event to appropriate sub-state machines
3. Each sub-state machine processes the event
4. Actions from all sub-machines are collected
5. Sub-machines can generate internal events for each other

Exploring Composition

Open crates/node/src/state/mod.rs and find the handle / dispatch helpers. Notice how it:

• Routes events to different sub-machines
• Collects actions from all sub-machines
• Coordinates between sub-machines

8. Knowledge Check Quiz

9. Practical Assignment

Threading & concurrency ladder (Hyperscale-rs context)

Your StateMachine::handle is synchronous and pure; a real node still runs in an OS full of threads and async runtimes. Below: one sentence each, from foundations toward details that show up in hyperscale-rs (especially production). For a code-level walkthrough of the production shard-loop thread and run_shard_loop, see E2E harnesses — production Part II.

1. Process vs thread — A process is an isolated program; threads share memory—hyperscale uses a dedicated thread for the hot ShardLoop path and separate Tokio/async work for network and timers.
2. Multitasking / time-sharing — The OS preempts threads; long synchronous work inside ShardLoop::step still blocks every consensus event on that thread until step returns.
3. Concurrency vs parallelism — Concurrency interleaves many tasks (events + channels); parallelism is simultaneous CPUs—hyperscale runs the state machine sequentially on one thread while pools (PooledDispatch) parallelize crypto, validation, and execution where offloaded.
4. Cooperative vs preemptive — The OS preempts threads preemptively; between iterations the pinned loop yields to the scheduler, but a single heavy step is still one uninterrupted chunk of work on that thread unless work is split or offloaded.
5. Async (Tokio) vs blocking — Tokio runs async tasks on a pool and awaits I/O; hyperscale’s ShardLoop uses synchronous step on the pinned thread while libp2p/RPC/timer code runs async and sends NodeInput in via channels.
6. Message passing (channels) — Threads communicate by sending NodeInput instead of ad-hoc shared mutable state; the pinned loop is the single writer for the hot state machine progression path.
7. Locking / data races — Locks serialize shared data; hyperscale reduces races by isolating hot state on one thread and using Arc + channels for handoff (storage bridges like SharedStorage still require careful design).
8. CPU affinity (“pinning”) — Affinity tells the OS which cores may run a thread; production tries to pin the shard-loop thread for steadier scheduling, not because the loop is test-only code.
9. Thread pools / work distribution — Pools run many units of work across workers; hyperscale’s dispatch pools handle heavy work so it need not all run inside step, with results coming back as callback channel events.
10. Deterministic simulation vs production — Simulation drives the same ShardLoop::step from one driver thread with fake time and fake network; production drives it from a pinned thread with wall-clock time and real libp2p gossipsub over QUIC between validators (wallet submit stays HTTP, separate from that mesh)—same logic, different harness.
11. Backpressure — When producers outpace consumers, queues grow or you drop/block; production uses batch deadlines and flush_expired_batches so validation and I/O do not grow without bound.
12. NUMA / cache effects (niche) — On large servers, which core and which memory node matter for cache and NUMA; pinning the hot thread is a small step toward predictable behavior, not a full NUMA story by itself.