The Question

High-Consistency Ledger System Design

Design a highly available and strictly consistent financial ledger system capable of processing 5,000 transactions per second. The system must ensure double-entry integrity, provide absolute auditability, and handle high-contention accounts (hot spots) without compromising on ACID properties. Discuss the schema design, idempotency strategy, and how the system remains resilient during database failovers.

PostgreSQL

Redis

Kafka

gRPC

ACID

Double-Entry Bookkeeping

CDC

Questions & Insights

Clarifying Questions

Scale & Performance: What is the expected Peak Transactions Per Second (TPS)? Are we looking at 100 TPS or 100,000 TPS?

Consistency Model: Is absolute strong consistency (ACID) required for all transactions, or can balance updates be eventually consistent for read-only views?

Audit Requirements: Does the system need to support point-in-time recovery and immutable audit logs?

Multi-Currency: Do we need to handle currency conversions and exchange rates within the ledger, or is that handled upstream?

Assumptions for MVP:

Scale: Target 5,000 TPS (Transactions Per Second).

Consistency: Strict ACID compliance is mandatory for the ledger to ensure "no money is created or lost."

Auditability: Every change must be recorded in an immutable journal.

Scope: Focus on the core double-entry accounting engine (Accounts, Journal, Entries).

Thinking Process

Core Engine: Implement a strict Double-Entry Bookkeeping model. Every transaction must have at least one debit and one credit that sum to zero.

Data Integrity: Use a Relational Database (RDBMS) with serializable or repeatable read isolation levels to prevent race conditions during balance updates.

Idempotency: Use idempotent keys for every transaction request to prevent duplicate entries from network retries.

Scaling Strategy: Start with a single highly-available RDBMS instance. For 10x scale, move to database sharding based on AccountID.

Bonus Points

Immutable Ledger: Use a "Write-Once-Read-Many" (WORM) pattern for journal entries where records are never updated or deleted, only reversed with new entries.

Hot-Account Handling: Implement a "sharded counter" or "slotting" strategy for system-level accounts (e.g., a "Revenue" account) that experience extreme write contention.

Deterministic Processing: Ensure the ledger logic is deterministic to allow for replaying the journal from Genesis to reconstruct state in case of catastrophic corruption.

Design Breakdown

Functional Requirements

Core Use Cases:

Create and manage accounts (Asset, Liability, Equity, Revenue, Expense).

Record double-entry transactions (Journal Entries).

Retrieve real-time account balances.

Fetch transaction history/audit logs.

Scope Control:

In-scope: Transaction execution, balance management, audit logs.

Out-of-scope: KYC/AML, payment gateway integration, currency exchange rate forecasting.

Non-Functional Requirements

Scale: Support 5,000 TPS and 1 Billion+ transactions per year.

Latency: P99 transaction latency < 100ms.

Availability & Reliability: 99.999% availability (Five 9s) given it handles financial data.

Consistency: Strong consistency for financial integrity.

Fault Tolerance: Multi-AZ deployment with synchronous replication.

Security: Strict RBAC, encryption at rest/transit, and full audit trails.

Estimation

Traffic Estimation:

Average QPS: 5,000 writes/sec.

Peak QPS: 15,000 writes/sec.

Read QPS: ~10,000 (balance checks + history).

Storage Estimation:

1 Journal entry ≈ 500 bytes.

5,000 tx/sec * 86,400 sec/day ≈ 432 million tx/day.

432M * 500 bytes ≈ 216 GB/day.

1 Year ≈ 78 TB. (Archiving strategy required).

Bandwidth:

Inbound: 5,000 * 1 KB ≈ 5 MB/s.

Blueprint

Concise Summary: A centralized ledger service backed by a high-consistency relational database using a double-entry accounting model.

Major Components:

Ledger API Service: Stateless service handling transaction validation, idempotency, and orchestration of the double-entry logic.

PostgreSQL: The primary source of truth for accounts and journal entries, utilizing ACID transactions.

Redis: Used exclusively for idempotency key storage and distributed locking for high-contention accounts.

Simplicity Audit: This design avoids complex distributed transaction protocols (like 2PC or Sagas) by centralizing the core ledger in a single ACID-compliant database cluster, which is sufficient for the MVP scale.

Architecture Decision Rationale:

Why this?: Financial systems prioritize correctness over horizontal scalability. RDBMS provides the necessary primitives (Transactions, Foreign Keys, Check Constraints) to ensure zero data loss.

Functional Satisfaction: Meets the need for double-entry and balance tracking.

Non-functional Satisfaction: Synchronous DB replication ensures high availability and durability.

High Level Architecture

Sub-system Deep Dive

Service

Topology & Scaling: Stateless instances deployed in a Multi-AZ Auto-Scaling Group. Scaling is based on CPU and request count.

API Schema Design:

POST /v1/transactions: Creates a journal entry.

Protocol: gRPC (Internal) / REST (External).

Request:

{ idempotency_key: UUID, entries: [{ account_id: ID, amount: 100, type: DEBIT }, { account_id: ID, amount: 100, type: CREDIT }] }

Idempotency: Returns 201 for new, 200 for existing key.

Resilience & Reliability:

Circuit Breaker: If the DB latency spikes, reject incoming requests to prevent cascading failure.

Retries: Client-side retries with exponential backoff for 5xx errors.

Storage

Access Pattern: Heavy write-append (Journal), Point-read (Balance), Range-read (History).

Database Table Design:

Accounts: id (PK), name, type (Asset/Liability), balance (Decimal), version (Optimistic Locking).

JournalEntries: id (PK), transaction_id (FK), account_id (FK), amount (Decimal), direction (Debit/Credit), created_at.

Idempotency: key (PK), response_payload, status, created_at.

Technical Selection: PostgreSQL.

Rationale: Strong ACID compliance, support for DECIMAL types (avoiding floating-point errors), and mature tooling for WAL (Write-Ahead Logging).

Distribution Logic: For MVP, a single primary with synchronous replicas. If scaling is needed, shard by AccountID.

Cache

Purpose & Justification: Redis is used for idempotency and distributed locking.

Key-Value Schema:

idem:{key} -> JSON(Response). TTL: 24 hours.

lock:account:{id} -> ThreadID. TTL: 5 seconds (to prevent deadlocks).

Failure Handling: If Redis is down, the system falls back to the DB Idempotency table (slower but safe).

Messaging

Purpose & Decoupling: Kafka is used to broadcast "Transaction Finalized" events to downstream systems (e.g., Notification Service, Analytics, Data Warehouse).

Event / Topic Schema: ledger.transactions.v1. Payload: Full journal entry details.

Throughput & Partitioning: Partitioned by TransactionID to ensure downstream systems receive events in order.

Technical Selection: Kafka for high throughput and replayability.

Wrap Up

Advanced Topics

Trade-offs: We prioritize Consistency (C) over Availability (A) in the CAP theorem. If the primary database and its synchronous replica are both unreachable, the system will reject writes to prevent data divergence.

Hot Spot Shards: If a system "Revenue" account becomes a bottleneck, we use a Slotting Technique: Split the account into Revenue_1 to Revenue_N for writes and sum them for reads.

Security & Privacy:

Encryption: AES-256 for data at rest.

Audit: Database triggers or CDC (Change Data Capture) to move data to an immutable S3 bucket for long-term compliance.

Optimization: For fast balance lookups, maintain a balance column in the Accounts table updated in the same transaction as the journal entry, rather than summing entries on every read.