MIMD-0019 - State Replication

[bmuddha]

Status: Draft
Created: 2026-02-16
Updated: 2026-03-13

---

Abstract

State replication enables standby nodes to stay in sync with the primary node by streaming transactions via NATS JetStream. The magicblock-replicator crate introduces a producer component (primary) that publishes transactions and block metadata, and a consumer component (standby) that replays them against their local AccountsDb. Standbys can catch up from any point using NATS message replay, with periodic snapshots enabling fast recovery.

---

Motivation

Currently, each validator node maintains its own independent state. This works for single-node deployments but becomes problematic when:

1. Scaling read traffic: Multiple nodes serving read requests need consistent state
2. High availability: Hot spares should be able to take over instantly
3. Geographic distribution: Edge nodes need to stay synced with the source of truth
4. Testing/Staging: Environment parity without re-processing the entire ledger

Transaction-level replication via NATS JetStream gives us durable messaging, automatic replay, and leader election out of the box.

---

Specification

Architecture Overview

                    NATS JetStream
                    ┌─────────────────────────────────────┐
                    │  Stream: TRANSACTIONS               │
                    │  Bucket: REPLICATOR_LOCK            │
                    │  ObjectStore: SNAPSHOTS             │
                    └──────────┬─────────────┬────────────┘
                               │             │
         ┌─────────────────────┘             └─────────────────────┐
         ▼                                                         ▼
┌─────────────────┐                                         ┌─────────────────┐
│  Primary Node   │                                         │  Standby Node   │
│  (Producer)     │                                         │  (Consumer)     │
└────────┬────────┘                                         └────────┬────────┘
         │                                                           │
         ▼                                                           ▼
  TransactionScheduler                                        TransactionScheduler
         │                                                           │
         ▼                                                           ▼
    AccountsDb                                                   AccountsDb

Protocol Messages

All messages use bincode serialization with a 4-byte discriminator prefix:

Type	Discriminator	Payload
Transaction	0x01	(slot: u64, index: u32, tx: SanitizedTransaction)
Block	0x02	(slot: u64, blockhash: Hash)
SuperBlock	0x03	(slot: u64, checksum: Hash)

Leader Election

Primary role is determined via NATS KV distributed lock:

1. Node attempts to acquire lock with TTL
2. If successful → becomes Primary (Producer)
3. If failed → becomes Standby (Consumer)
4. Primary refreshes lock periodically
5. Standby monitors lock; on timeout → promotes to Primary

Message Flow

Primary (Producer)                         Standby (Consumer)
       │                                           │
       │  ┌─ Acquire leader lock                   │
       │  └─ Publish to TRANSACTIONS stream        │
       │                                           │
       ├────── Transaction ──────────────────────► │
       ├────── Transaction ──────────────────────► │ Replay via Scheduler
       ├────── Block ───────────────────────────► │
       ├────── Transaction ──────────────────────► │
       ├────── SuperBlock ──────────────────────► │ Verify checksum
       │                                           │
       │  (upload snapshot to ObjectStore)         │ (retrieve snapshot for recovery)

Producer (Primary)

1. Acquires distributed lock via NATS KV
2. Publishes transactions to TRANSACTIONS stream as they're processed
3. Sends Block delimiters when slot transitions occur
4. Sends SuperBlock every N blocks with AccountsDb checksum
5. Uploads snapshots to NATS ObjectStore periodically
6. Refreshes lock every second; releases on shutdown

Consumer (Standby)

1. Subscribes to TRANSACTIONS stream
2. Retrieves snapshot from ObjectStore for initial recovery
3. Replays messages starting from last acknowledged position:

Message	Action
Transaction	Replay via TransactionScheduler.replay()
Block	Verify blockhash matches local observation, transition slot
SuperBlock	Compute local AccountsDb checksum, compare with received

4. Acknowledges processed messages
5. Monitors leader lock for primary liveness
6. Promotes to primary on leader timeout

Transaction Replay

Standbys replay transactions using the existing TransactionScheduler.replay() path. This ensures:

• Same SVM semantics as primary
• Account locking and conflict resolution
• Program cache consistency
• Sysvar updates (Clock, SlotHashes)

Replayed transactions update AccountsDb but don't persist to ledger.

---

Safeguards

Validation Checks

1. Blockhash verification: Standby ensures its observed blockhash matches Block delimiter
2. Checksum verification: SuperBlock contains AccountsDb hash for state validation
3. Transaction ordering: NATS preserves message ordering within stream
4. Gap detection: Missing messages detected via sequence numbers

Failure Modes

Scenario	Detection	Response
Replay failure	TransactionError from scheduler	Log, halt, restart from snapshot
Checksum mismatch	SuperBlock hash differs	Alert, flag state divergence
Primary lock timeout	Lock TTL expires	Standby promotes to primary
NATS disconnect	Connection loss	Reconnect, resume from last ack
Snapshot unavailable	ObjectStore miss	Start from stream beginning

---

Open Questions

1. Checksum algorithm: SHA256 of AccountsDb? Merkle root? Something faster like xxHash?

2. Authentication: TLS for NATS connections? NATS user accounts? Or trust internal network?

3. Backpressure: What if standby can't keep up with stream? NATS rate limiting? Consumer pacing?

4. Snapshot frequency: How often should primary upload snapshots? Trade-off between recovery time and upload overhead.

5. Replay failure policy: If a transaction fails to replay on standby (e.g., due to program differences), what should happen? Panic? Retry? Alert operator?

---

Implementation Phases

Phase 1: Protocol and NATS Integration ✅

• Message type definitions with bincode serialization
• NATS JetStream broker connection and stream/bucket initialization
• Producer with leader lock acquisition
• Consumer with stream subscription
• Snapshot upload/retrieval to ObjectStore

Phase 2: Service Layer ✅

• Automatic primary/standby role transitions
• Leader election via NATS KV lock
• Lock refresh and timeout handling
• Standby promotion on leader timeout

Phase 3: Filesystem Watcher ✅

• Monitor AccountsDb snapshot archives
• Trigger snapshot uploads on creation

Phase 4: Validation and Monitoring 📋

• Blockhash verification
• SuperBlock checksum computation
• Metrics (lag, throughput, errors)
• Health checks

Phase 5: Testing and Hardening 📋

• Integration tests (primary + standby via NATS)
• Network failure simulation
• Large state sync tests
• Performance benchmarks