Deep Dive: JMT & Shard State Storage

Segment 1 — What JMT is (and what it is not)

Network
  Shard 0 validators ──► RocksDB + JMT (version 0…H)  → state_root in shard-0 headers
  Shard 1 validators ──► separate RocksDB + JMT         → state_root in shard-1 headers
  …

Cross-shard txs touch multiple shard trees independently; atomicity is provision-based (Phase 4), not one global JMT.

Segment 2 — Two layers on each validator (Node DB + JMT)

Every substate slot has the same logical address in both layers, but different on-disk encoding:

In-repo contract — owner-prefixed leaf keys (state_key.rs)

/// Owner-major JMT leaf key: high 16 bytes = blake3(routing_node),
/// low 16 bytes = blake3(full storage_key). routing_node is the global
/// owner for internal nodes (vaults, KV stores) or the node itself for globals.
pub fn jmt_leaf_key(storage_key: &[u8], owner: Option<NodeId>) -> [u8; 32] { … }

// Provision proof verification uses the same rule via ProvisionEntry::owned_nodes.

Takeaway: execution reads/writes Node DB; consensus headers carry the JMT root so everyone agrees on the map’s fingerprint without shipping every account on every block.

Segment 3 — Keys in real life (entity, partition, sort_key)

A token balance (or any substate row) is not “Alice” as a string. It is a triple:

Byte layout (shard/keys.rs)

// Key layout: [node_key][partition_num (1B)][sort_key]
pub fn to_storage_key(partition_key: &DbPartitionKey, sort_key: &DbSortKey) -> Vec<u8> {
    let mut key = Vec::with_capacity(partition_key.node_key.len() + 1 + sort_key.0.len());
    key.extend_from_slice(&partition_key.node_key);
    key.push(partition_key.partition_num);
    key.extend_from_slice(&sort_key.0);
    key
}

Execution emits DatabaseUpdates: nested map of entity_key → partition → sort_key → Set|Delete. Storage flattens that into JMT work items:

// tree/mod.rs — put_at_version (simplified)
let storage_key = make_storage_key(entity_key, partition_num, &sort_key.0);
let jmt_key = hash_storage_key(storage_key, owner_map);  // → jmt_leaf_key(...)
let jmt_value = value.map(hash_value);         // BLAKE3(raw bytes) or None = delete

Toy diagram note: paths like 1010 / 1101 in the block-fields walkthrough are 4-bit teaching cartoons. Production uses the full 256-bit path implied by the 32-byte jmt_key.

Segment 4 — Genesis: first INSERT (empty → Alice & Bob funded)

Simulation and production fund accounts at genesis via initialize_genesis_with_balances. That produces DatabaseUpdates with DatabaseUpdate::Set on keys that **did not exist** — pure **inserts**.

Genesis install (storage-rocksdb/shard/core.rs)

impl GenesisCommit for RocksDbShardStorage {
    fn install_genesis(&self, merged: &DatabaseUpdates) -> StateRoot {
        Self::commit_substates_only(self, merged);   // 1) write raw bytes @ version 0
        Self::finalize_genesis_jmt(self, merged)     // 2) build first JMT @ version 0
    }
}

Teaching cast (shard 0)

After genesis — Node DB vs JMT

JMT v0 (R₀) — INSERT leaves only
              R₀
             /  \
      path→1010   path→1101
      Alice=100    Bob=50

Node DB (state CF):
  entity_alice|vault|GOLD  →  bytes(100)
  entity_bob|vault|GOLD    →  bytes(50)

Segment 5 — Real transfer timeline (simulation path)

This is the same rhythm as single_shard_tx_sim and the storage test test_list_substates_for_node_at_height_returns_historical_data — simplified to Alice paying Bob 30 GOLD.

Critical timing: the transfer tx sits in block 1’s transaction_root, but the state_root bump appears on block 2 when that execution’s FinalizedWave is stapled. See block 40 → 41 example.

Receipts carry the writes execution observed:

// types/receipt/consensus.rs — successful receipt includes:
database_updates: DatabaseUpdates  // Set(entity_alice, …, 70), Set(entity_bob, …, 80)

Segment 6 — UPDATE walk (same keys, new values)

The storage integration test test_list_substates_for_node_at_height_returns_historical_data is the cleanest in-repo proof of insert-then-update on the **same** storage_key:

In-repo test (storage-memory/shard/tests.rs)

// Block height 1: INSERT value [100]
let updates1 = make_mapped_database_update(1, 0, vec![10], vec![100]);
let root_v1 = commit_with(&storage, &updates1, &block1, &qc1);

// Block height 2: UPDATE same sort_key → [200]
let updates2 = make_mapped_database_update(1, 0, vec![10], vec![200]);
let root_v2 = commit_with(&storage, &updates2, &block2, &qc2);
assert_ne!(root_v1, root_v2, "roots must differ after overwrite");

Map test → Alice/Bob transfer

Rehash cost (toy vs production)

UPDATE — same jmt_key paths, new leaf hashes
              R₁  (was R₀)
             /  \
      path→1010   path→1101
      Alice=70     Bob=80     ← same paths as genesis, new values

Historical reads: at JMT version 1 you still see Alice=100; at version 2 you see 70 — the test asserts exactly that pattern with list_substates_for_node_at_height.

Segment 7 — prepare_block_commit at propose time

When the proposer builds block 2, it does not guess the root. It replays certificates against the parent state:

// storage/src/shard/chain_writer.rs (trait docs)
// 1. prepare_block_commit → (state_root, jmt_snapshot, prepared)
// 2. Runner stores prepared closure keyed by block hash
// 3. At QC commit → invoke prepared(SyncHint, certified_block, …)
// 4. If no prepared closure (sync) → commit_block recomputes from scratch

Inside prepare: merge all database_updates from finalized wave receipts → flatten to (storage_key, optional_value) → put_at_version → new root + collected JMT nodes for the batch write.

parent_state_root R₀  +  body.certificates (FinalizedWave₁)
        │
        ├─ merge DatabaseUpdates from receipts
        ├─ for each key: Node DB read (or base_reads cache) + JMT path update
        └─► state_root R₁  +  JmtSnapshot  +  PreparedCommit closure

Peers verify R₁ before BlockVote; after QC, closure persists v2 to RocksDB.

Code map (study order)

Lab tie-in: after the simulation E2E lab, set a breakpoint in storage commit or grep logs for state_root across two consecutive heights during single_shard_tx_sim.