Performance Measurement

Learning Objectives

• Understand how performance is measured today in hyperscale-rs (simulation and production)
• Know what simulation metrics can miss compared to real deployment
• Identify gaps and get concrete recommendations for additional performance measures

After the hands-on lab, continue to libp2p then cryptography. Optional reading: Rust optimizations ties measurements to implementation idioms.

How performance is measured today

Hyperscale-rs measures performance in two main contexts: deterministic simulation (simulator + workload) and production (Prometheus metrics). The same consensus and execution logic runs in both; the difference is I/O (simulated time and network vs real libp2p and RocksDB) and what is instrumented.

Simulation

The simulator (hyperscale-sim) runs a controlled workload for a fixed duration (e.g. --seed 42 -s 2 -v 4 -d 60 --tps 1000). It collects:

• Throughput: Submitted / completed / rejected counts; average TPS and peak TPS over the run (from completion counts per sample window).
• Latency: Per-transaction latency from submit to finalization; P50, P90, P99, max, avg (from an HDR histogram in the simulator metrics module — study table).
• Outcomes: Retries, rejections, and in-flight at end (transactions not yet finalized when the run stops).
• Lock contention: Peak locked nodes, deferred count, and contention ratio from the mempool (cross-shard backpressure).

Reports are produced by MetricsCollector / SimulationReport and printed at the end (and optionally a livelock report for stuck transactions). The simulation uses simulated time and a network simulator (configurable latency, no real validator QUIC/gossipsub mesh and no wallet HTTP—just in-memory delivery with optional fake latency).

Production (metrics crate + Prometheus)

In production, hyperscale_metrics defines a MetricsRecorder trait; hyperscale_metrics_prometheus implements it and is installed at startup. Recorded metrics include:

• Consensus: Blocks committed, block commit latency, block height, round, view changes.
• Transactions: Finalized count (with latency histogram and cross_shard label), mempool size.
• Backpressure: In-flight count, backpressure active, txs with commitment proof.
• Infrastructure: Network messages sent/received, signature verification latency (by type), execution latency, speculative execution latency and cache hit/miss/late-hit/invalidated.
• Pools and channels: Queue depths for consensus, crypto, tx validation, execution, RPC, sync, and request channels.
• Storage: RocksDB read/write latency, storage operation latency (by operation), batch size, votes/certificates/blocks persisted.
• Sync and fetch: Blocks downloaded/received/buffered/filtered/verified/applied, fetch started/completed/failed/latency/items.
• Ingress and errors: Tx rejected (syncing, pending limit, reason), invalid message, livelock cycle/deferral, broadcast failure/retry/drop, backpressure events.

So in production you get real wall-clock latency, real network and disk behavior, and rich counters/gauges/histograms for debugging and capacity planning.

What simulation can miss vs real life

Simulation is excellent for correctness and deterministic regression, but it does not fully reflect production performance. Gaps include:

• Real network behavior: Simulation typically uses a simple latency model and in-memory message passing. Production validators use libp2p gossipsub over QUIC for peer replication (separate from wallet HTTP submit). Real meshes have variable RTT, packet loss, reordering, and congestion; fanout, mesh, and duplicate-suppression behavior are not reproduced in sim; neither are slow or flaky peers.
• Real I/O and CPU contention: Simulated time advances in steps; there is no real disk latency, no RocksDB compaction stalls, and no contention with other processes. Production runs with real RocksDB, real signature verification (CPU-bound), and real execution (Radix Engine) on real hardware—so tail latencies and throughput under load can differ a lot.
• Concurrency and scheduling: Simulation may use a single-threaded or simplified runner; production uses thread pools (dispatch-pooled), async I/O, and channel backpressure. Queue depths, lock contention, and thread starvation in production are not mirrored in the sim.
• Scale and topology: Sim runs are often small (few shards, few validators). Real deployments have more nodes, more shards, and more cross-shard traffic; scaling behavior (e.g. broadcast amplification, sync load) is not extrapolated from small sims.
• Failure and recovery: Sim can inject crashes and partitions, but real failures (node restarts, network splits, disk full, OOM) and recovery (sync, catch-up, reconnection) have timing and resource usage that are not fully captured.
• End-to-end user experience: Simulation measures submit-to-finalize latency inside the system. Real users see RPC latency, client retries, and propagation delay to their node; these are not in the sim report.

So: use simulation for throughput and latency trends and regression (e.g. “no slowdown after this change”), but always validate with production or production-like benchmarks (e.g. spammer against a local cluster with real network and storage).

Recommendations for additional performance measures

To close gaps and support both operations and development, consider the following.

Simulation

• Configurable network model: Add optional packet loss, jitter, and per-link or per-shard latency distributions so sim can stress-test under worse-than-ideal conditions.
• Larger and multi-shard runs in CI: Add periodic or nightly sim runs with more shards and validators and cross-shard workload to catch scaling and contention issues.
• Time-to-finality by shard and cross-shard: Already have cross_shard on transaction_finalized in production; in sim, report P50/P90/P99 and TPS split by single-shard vs cross-shard to track cross-shard overhead.
• Round and view-change stats: In sim, record rounds per height and view-change count; report averages and max so that timeouts and liveness can be correlated with throughput.

Production

• End-to-end latency from client view: From RPC submit to a “finalized” or “included in block” response (or callback). This can be done in the RPC layer or a sidecar that submits and polls for status.
• Per-shard and per-validator breakdown: Where useful, add shard_id / validator_id (or similar) to key metrics (e.g. commit latency, finalized count) to spot hot shards or slow validators.
• Resource usage: CPU and memory (e.g. process or container) and, if possible, disk I/O (e.g. RocksDB compaction). Correlate with blocks/sec and TPS to understand cost per tx.
• Alerting and SLOs: Define SLOs (e.g. P99 finalization latency < Xs, TPS > Y) and wire metrics into alerting so regressions and outages are visible in production.

Shared

• Unified vocabulary: Keep naming aligned between sim report (e.g. “completed”, “rejected”, “in-flight”) and production metrics (e.g. transactions_finalized, rejection reasons) so that runbooks and dashboards map clearly from sim to prod.
• Export sim report to structured format: Optionally export SimulationReport (and livelock report) to JSON or similar for CI dashboards and trend analysis (e.g. TPS and P99 over commits).

Step-by-step: Instrument and measure with simulation

Follow these steps to add or use performance instrumentation and run the simulator to collect throughput and latency.

1. Build and run the simulator. From the hyperscale-rs repo root: cargo build --release, then cargo run --release --bin hyperscale-sim -- --seed 42 -s 2 -v 4 -d 60 --tps 1000. This runs 2 shards, 4 validators per shard, 60 seconds of simulated time, with a target of 1000 TPS. The binary uses SimulationRunner and MetricsCollector; no extra instrumentation is required for built-in metrics.
2. Read the simulation report. At the end of the run, SimulationReport::print_summary() prints: Submitted / Completed / Rejected / Retries / In-flight; Average TPS and Peak TPS; P50 / P90 / P99 / Max / Avg latency; peak lock contention (locked nodes, deferred count, contention ratio). Use this to judge throughput and tail latency for the workload and parameters.
3. Vary parameters to measure behavior. Change -s (shards), -v (validators), -d (duration), and --tps to see how TPS and latency change. For example: -s 1 -v 4 -d 30 --tps 500 for a single-shard baseline; -s 2 -v 4 -d 60 --tps 2000 to stress cross-shard. Compare reports to spot regressions or scaling limits.
4. Add a custom metric in simulation (optional). Extend MetricsCollector in the simulator metrics module (see study table) with a new field and record_*; update SimulationReport and finalize / print_summary. Call your recorder where the sim observes the event.
5. Add a metric used in both sim and production (optional). Define the semantic on the shared MetricsRecorder trait in the metrics crate; implement it in the Prometheus backend; call hyperscale_metrics::record_*(...) from production paths. The simulator’s report uses MetricsCollector — align naming with the study table paths or install a no-op recorder in sim if you call record_* from both worlds.
6. Optional: livelock report. After run_for, call analyze_livelocks() on the runner and print_summary() on the returned report to see stuck or deferred transactions and potential cycles. This helps interpret high in-flight or low completion rate.

Summary: run hyperscale-sim with chosen -s -v -d --tps, read the printed report for TPS and latency, vary parameters to compare; optionally extend MetricsCollector or the shared MetricsRecorder and wire calls from the right code paths.

What “done” looks like

• Two sim summaries pasted or summarized with numbers (not “it looked fine”).
• A short comparison paragraph for Lab 2.
• Sim vs prod gap: at least one metric in each column from Lab 3.