blog.dopana

Back

In financial technology (Fintech) and large-scale ecommerce, payment systems demand absolute correctness, high fault tolerance, and an immutable, tamper-proof audit trail. A minor rounding discrepancy, a double-charge during network timeout, or an unrecorded transaction can cost millions of dollars and lead to severe regulatory sanctions.

Traditional CRUD systems—where account balances are modified in-place using SQL UPDATE statements—fall catastrophically short in financial domains. When a customer disputes a charge, an updated row in a database cannot prove how that balance arrived at its current number or which intermediary states occurred.

Combining Event Sourcing with CQRS (Command Query Responsibility Segregation) is the industry gold standard adopted by financial leaders like Stripe, Wise, and PayPal to build resilient, self-healing payment architectures.

1. Explain Like I’m 10 (ELI5)#

Think of a payment system as a high-security bank vault equipped with an un-erasable paper ledger and a surveillance camera:

  • The Traditional CRUD Approach: The bank teller opens the vault door, takes out an eraser, rubs out “10,000"writtenonthedoor,andwrites"10,000" written on the door, and writes "7,000”. If an auditor asks “Where did the $3,000 go?”, the teller has to search through scratchpads or rely on memory. If the number was written incorrectly, no one knows who changed it or when.
  • Event Sourcing + CQRS: The teller never erases anything. Every single movement of money must be recorded on a signed, sequentially numbered deposit or withdrawal slip:
    1. Slip #1: Customer deposited $10,000 into account.
    2. Slip #2: Withdrew $3,000 to pay electricity bill invoice INV-999.

Outside the vault sits an electronic scoreboard (Read Model / CQRS). This display reads the committed slips and calculates the balance in real-time: “Current Balance: $7,000”. Customers quickly check their balance on the scoreboard, while the physical slips inside the Event Store remain permanently locked, immutable, and fully verifiable.

[!NOTE] In an Event Sourced payment system, money is not a static number in a table. Money is the sum of every financial event that has ever happened to that account.

2. Microservices Architecture for Payment Processing#

The system is decoupled into specialized microservices that communicate asynchronously through an event streaming backbone (such as Apache Kafka) to guarantee zero message loss during peak traffic spikes.

Submit PaymentRedis Idempotency CheckCreatePayment CommandAppend EventPublish EventConsume EventUpdate ProjectionConsume EventCall External GatewayWebhook CallbackClient / MerchantPayment Gateway ServiceRedis CachePayment Ledger Service(Write Service)Event StorePostgreSQL / EventStoreDBApache Kafka(Message Broker)Payment Query Service(Read Service)Read DatabasePostgreSQL / ElasticClient Balance / History QueryPayment PSP WorkerPartner Bank / Card Network

The core microservices include:

  1. Payment Gateway Service: The perimeter ingress service handling client requests, cryptographic signature validation (HMAC/mTLS), rate limiting, and distributed idempotency checks via Redis before dispatching commands.
  2. Payment Ledger Service (Write Service): The core engine of the Event Sourcing architecture. It accepts commands, evaluates domain invariants (e.g., sufficient wallet balance), enforces optimistic concurrency, and appends new immutable events to the Event Store.
  3. Payment Query Service (Read Service): Serves customer-facing queries, balance lookups, and transaction statements from a read-optimized, pre-projected database with sub-millisecond response times.
  4. Payment PSP Worker: Consumes PaymentInitiated events and invokes external banking APIs or card networks (Stripe, Visa, Adyen). Once the partner bank responds, the worker issues completion commands back to the Ledger.
  5. Reconciliation Service: A continuous background auditing engine that compares internal Event Store streams against daily clearing and settlement statements (e.g., ISO 20022 or CAMT.053 files) from banks to detect any discrepancies down to the cent.

3. Database Design: Event Store vs. Read Model#

Event Sourcing enforces a strict separation between the authoritative Single Source of Truth (Event Store) and query-optimized materialized views (Read Model).

A. Event Store (Write DB) - Immutable Event Stream#

The Event Store allows only INSERT operations—UPDATE and DELETE queries are strictly forbidden. This can be built on top of PostgreSQL (leveraging monthly declarative table partitioning) or dedicated engines like EventStoreDB.

schema-event-store.sql
CREATE TABLE payment_events (
    event_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_id UUID NOT NULL,
    version INT NOT NULL,
    event_type VARCHAR(64) NOT NULL,
    event_data JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    CONSTRAINT uq_aggregate_version UNIQUE (aggregate_id, version)
);

-- Optimized index for chronological event replay per aggregate
CREATE INDEX idx_payment_events_aggregate ON payment_events (aggregate_id, version ASC);
sql
Column NameData TypeDescription
event_id (PK)UUIDUniversally unique identifier for each event occurrence.
aggregate_idUUIDAggregate root entity ID (e.g., transaction_id or wallet_id).
versionINTStrictly increasing sequence number for Optimistic Concurrency Control.
event_typeVARCHARThe event descriptor (PaymentInitiated, PaymentAuthorized, PaymentCompleted, PaymentFailed).
event_dataJSONBFull structured payload containing financial figures, currencies, and metadata.
created_atTIMESTAMPHigh-precision timestamp recording when the event was appended.

Example event_data payload for PaymentInitiated:

payload-payment-initiated.json
{
  "amount": 5000.00,
  "currency": "USD",
  "source_wallet": "wallet-abc-123",
  "destination_merchant": "merchant-xyz-999",
  "idempotency_key": "idem-uuid-9999",
  "payment_method": "CREDIT_CARD"
}
json

B. Read Model (Read DB) - Fast Denormalized Views#

The read database is asynchronously populated by projection consumers subscribed to Kafka topics. Because it serves user-facing dashboards and queries, tables can be denormalized across PostgreSQL, MySQL, or Elasticsearch.

schema-read-model.sql
CREATE TABLE transactions (
    transaction_id UUID PRIMARY KEY,
    user_id UUID NOT NULL,
    source_wallet VARCHAR(64) NOT NULL,
    destination_merchant VARCHAR(64) NOT NULL,
    amount DECIMAL(18, 2) NOT NULL,
    currency VARCHAR(8) NOT NULL,
    status VARCHAR(32) NOT NULL, -- PENDING, SUCCESS, FAILED
    failure_reason TEXT,
    updated_at TIMESTAMP WITH TIME ZONE NOT NULL
);

CREATE INDEX idx_transactions_user_status ON transactions (user_id, status);
sql

4. End-to-End Data Flow#

The lifecycle of a single payment follows a coordinated five-stage pipeline:

123456789101112131415POST /payments (with Idempotency-Key)Check Idempotency KeyDispatch CreatePaymentCommandVerify Balance & Append PaymentInitiatedAppend Success (Version: 1)Publish PaymentInitiated EventProjector writes record (Status: PENDING)Worker consumes PaymentInitiatedInvoke External Payment APIPayment Settled (Success)Dispatch CompletePaymentCommandAppend PaymentCompleted (Version: 2)Publish PaymentCompleted EventProjector updates record (Status: SUCCESS)Query Transaction Status: Receives SUCCESSCustomer ClientPayment GatewayRedis CachePayment Ledger ServiceEvent StoreApache KafkaRead DatabasePSP WorkerPartner BankCustomer ClientPayment GatewayRedis CachePayment Ledger ServiceEvent StoreApache KafkaRead DatabasePSP WorkerPartner Bank
  1. Command Submission: The client clicks “Pay”. Payment Gateway authenticates the token, verifies the mandatory Idempotency-Key in Redis, and dispatches a CreatePaymentCommand.
  2. State Validation & Event Append: The Ledger Service replays previous events to rehydrate the wallet balance. If valid, it writes a new PaymentInitiated event into the Event Store.
  3. Event Broadcasting: The committed event is published to Apache Kafka (typically using the Transactional Outbox Pattern to guarantee at-least-once delivery).
  4. Projection Materialization: The Payment Query Service consumes the event from Kafka and inserts a PENDING record into the transactions table.
  5. Third-Party Settlement & Completion: The PSP Worker processes the event and contacts the banking network. Upon success, a CompletePaymentCommand is issued to the Ledger, producing a PaymentCompleted event (version 2) that advances the projection status to SUCCESS.

5. Critical Payment Edge Cases & Solutions#

Preventing Duplicate Charges (The Idempotency Pattern)#

Network timeouts and impatient users frequently trigger duplicate payment requests.

  • Solution: Clients must send an Idempotency-Key HTTP header (UUIDv4) with every payment request.
  • The Payment Gateway uses Redis distributed locking (SET NX EX 86400) to hold the key for 24 hours.
  • If a concurrent duplicate request arrives while processing, the gateway responds with HTTP 409 Conflict. If the transaction has already completed, the cached response payload is immediately returned without re-executing business logic.

Race Conditions & Optimistic Concurrency Control#

What happens if a user with 100attemptstwosimultaneous100 attempts two simultaneous 80 purchases across different browser tabs?

  • Traditional row-level database locks (SELECT ... FOR UPDATE) degrade performance and cause deadlocks under high concurrency.
  • In Event Sourcing, the database enforces a UNIQUE (aggregate_id, version) constraint.
  • When transactions A and B both rehydrate state at version = 5:
    • Transaction A appends first with version = 6 and commits.
    • Transaction B attempts to append with version = 6 and is immediately rejected by PostgreSQL with a unique constraint violation.
    • The Ledger catches this collision, reloads the fresh aggregate state (now depleted by Transaction A), and cleanly rejects Transaction B for insufficient funds.

Reconciliation & Discrepancy Auditing#

External network connections can drop abruptly—for example, a partner bank may charge a customer card but timeout before delivering the HTTP webhook confirmation.

  • Solution: A batch Reconciliation Service runs daily at midnight (T+1), ingesting external bank settlement feeds (ISO 20022 CAMT.053 / BAI2 files).
  • The service conducts Three-Way Matching: comparing internal Event Store streams against Gateway access logs and bank clearing statements.
  • If an unconfirmed transaction was settled by the bank, the reconciliation engine automatically appends an adjustment event (PaymentAutoResolved or CompensatingRefundIssued) to re-align internal balances with actual bank balances.

6. TypeScript Aggregate Implementation#

Here is a clean implementation of the PaymentTransactionAggregate demonstrating state transitions, command invariants, and version tracking:

7. Conclusion#

Combining Event Sourcing and CQRS transforms payment systems from fragile, prone-to-corruption CRUD databases into resilient financial ledgers:

  • Complete Audit Trail: Every monetary event is permanently recorded as an immutable fact.
  • Independent Scalability: Write paths achieve massive throughput via append-only inserts, while read paths deliver instantaneous customer queries.
  • Built-in Resilience: Natural defense against duplicate charges, race conditions via optimistic locking, and automated multi-way bank reconciliation.

References#