Designing Payment Systems with Event Sourcing and CQRS
Architecting a fault-tolerant payment system with immutable audit trails, idempotency, and optimistic concurrency using Event Sourcing, CQRS, and Kafka.
In financial technology (Fintech) and large-scale ecommerce, payment systems demand absolute correctness, high fault tolerance, and an immutable, tamper-proof audit trail. A minor rounding discrepancy, a double-charge during network timeout, or an unrecorded transaction can cost millions of dollars and lead to severe regulatory sanctions.
Traditional CRUD systems—where account balances are modified in-place using SQL UPDATE statements—fall catastrophically short in financial domains. When a customer disputes a charge, an updated row in a database cannot prove how that balance arrived at its current number or which intermediary states occurred.
Combining Event Sourcing with CQRS (Command Query Responsibility Segregation) is the industry gold standard adopted by financial leaders like Stripe, Wise, and PayPal to build resilient, self-healing payment architectures.
1. Explain Like I’m 10 (ELI5)#
Think of a payment system as a high-security bank vault equipped with an un-erasable paper ledger and a surveillance camera:
- The Traditional CRUD Approach: The bank teller opens the vault door, takes out an eraser, rubs out “7,000”. If an auditor asks “Where did the $3,000 go?”, the teller has to search through scratchpads or rely on memory. If the number was written incorrectly, no one knows who changed it or when.
- Event Sourcing + CQRS: The teller never erases anything. Every single movement of money must be recorded on a signed, sequentially numbered deposit or withdrawal slip:
- Slip #1: Customer deposited $10,000 into account.
- Slip #2: Withdrew $3,000 to pay electricity bill invoice INV-999.
Outside the vault sits an electronic scoreboard (Read Model / CQRS). This display reads the committed slips and calculates the balance in real-time: “Current Balance: $7,000”. Customers quickly check their balance on the scoreboard, while the physical slips inside the Event Store remain permanently locked, immutable, and fully verifiable.
[!NOTE] In an Event Sourced payment system, money is not a static number in a table. Money is the sum of every financial event that has ever happened to that account.
2. Microservices Architecture for Payment Processing#
The system is decoupled into specialized microservices that communicate asynchronously through an event streaming backbone (such as Apache Kafka) to guarantee zero message loss during peak traffic spikes.
The core microservices include:
- Payment Gateway Service: The perimeter ingress service handling client requests, cryptographic signature validation (HMAC/mTLS), rate limiting, and distributed idempotency checks via Redis before dispatching commands.
- Payment Ledger Service (Write Service): The core engine of the Event Sourcing architecture. It accepts commands, evaluates domain invariants (e.g., sufficient wallet balance), enforces optimistic concurrency, and appends new immutable events to the Event Store.
- Payment Query Service (Read Service): Serves customer-facing queries, balance lookups, and transaction statements from a read-optimized, pre-projected database with sub-millisecond response times.
- Payment PSP Worker: Consumes
PaymentInitiatedevents and invokes external banking APIs or card networks (Stripe, Visa, Adyen). Once the partner bank responds, the worker issues completion commands back to the Ledger. - Reconciliation Service: A continuous background auditing engine that compares internal Event Store streams against daily clearing and settlement statements (e.g., ISO 20022 or CAMT.053 files) from banks to detect any discrepancies down to the cent.
3. Database Design: Event Store vs. Read Model#
Event Sourcing enforces a strict separation between the authoritative Single Source of Truth (Event Store) and query-optimized materialized views (Read Model).
A. Event Store (Write DB) - Immutable Event Stream#
The Event Store allows only INSERT operations—UPDATE and DELETE queries are strictly forbidden. This can be built on top of PostgreSQL (leveraging monthly declarative table partitioning) or dedicated engines like EventStoreDB.
CREATE TABLE payment_events (
event_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
aggregate_id UUID NOT NULL,
version INT NOT NULL,
event_type VARCHAR(64) NOT NULL,
event_data JSONB NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uq_aggregate_version UNIQUE (aggregate_id, version)
);
-- Optimized index for chronological event replay per aggregate
CREATE INDEX idx_payment_events_aggregate ON payment_events (aggregate_id, version ASC);sql| Column Name | Data Type | Description |
|---|---|---|
event_id (PK) | UUID | Universally unique identifier for each event occurrence. |
aggregate_id | UUID | Aggregate root entity ID (e.g., transaction_id or wallet_id). |
version | INT | Strictly increasing sequence number for Optimistic Concurrency Control. |
event_type | VARCHAR | The event descriptor (PaymentInitiated, PaymentAuthorized, PaymentCompleted, PaymentFailed). |
event_data | JSONB | Full structured payload containing financial figures, currencies, and metadata. |
created_at | TIMESTAMP | High-precision timestamp recording when the event was appended. |
Example event_data payload for PaymentInitiated:
{
"amount": 5000.00,
"currency": "USD",
"source_wallet": "wallet-abc-123",
"destination_merchant": "merchant-xyz-999",
"idempotency_key": "idem-uuid-9999",
"payment_method": "CREDIT_CARD"
}jsonB. Read Model (Read DB) - Fast Denormalized Views#
The read database is asynchronously populated by projection consumers subscribed to Kafka topics. Because it serves user-facing dashboards and queries, tables can be denormalized across PostgreSQL, MySQL, or Elasticsearch.
CREATE TABLE transactions (
transaction_id UUID PRIMARY KEY,
user_id UUID NOT NULL,
source_wallet VARCHAR(64) NOT NULL,
destination_merchant VARCHAR(64) NOT NULL,
amount DECIMAL(18, 2) NOT NULL,
currency VARCHAR(8) NOT NULL,
status VARCHAR(32) NOT NULL, -- PENDING, SUCCESS, FAILED
failure_reason TEXT,
updated_at TIMESTAMP WITH TIME ZONE NOT NULL
);
CREATE INDEX idx_transactions_user_status ON transactions (user_id, status);sql4. End-to-End Data Flow#
The lifecycle of a single payment follows a coordinated five-stage pipeline:
- Command Submission: The client clicks “Pay”.
Payment Gatewayauthenticates the token, verifies the mandatoryIdempotency-Keyin Redis, and dispatches aCreatePaymentCommand. - State Validation & Event Append: The
Ledger Servicereplays previous events to rehydrate the wallet balance. If valid, it writes a newPaymentInitiatedevent into theEvent Store. - Event Broadcasting: The committed event is published to
Apache Kafka(typically using the Transactional Outbox Pattern to guarantee at-least-once delivery). - Projection Materialization: The
Payment Query Serviceconsumes the event from Kafka and inserts aPENDINGrecord into thetransactionstable. - Third-Party Settlement & Completion: The
PSP Workerprocesses the event and contacts the banking network. Upon success, aCompletePaymentCommandis issued to the Ledger, producing aPaymentCompletedevent (version 2) that advances the projection status toSUCCESS.
5. Critical Payment Edge Cases & Solutions#
Preventing Duplicate Charges (The Idempotency Pattern)#
Network timeouts and impatient users frequently trigger duplicate payment requests.
- Solution: Clients must send an
Idempotency-KeyHTTP header (UUIDv4) with every payment request. - The
Payment Gatewayuses Redis distributed locking (SET NX EX 86400) to hold the key for 24 hours. - If a concurrent duplicate request arrives while processing, the gateway responds with HTTP
409 Conflict. If the transaction has already completed, the cached response payload is immediately returned without re-executing business logic.
import type { Request, Response, NextFunction } from 'express';
import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS_URL!);
export async function idempotencyMiddleware(req: Request, res: Response, next: NextFunction) {
const idempotencyKey = req.header('Idempotency-Key');
if (!idempotencyKey) {
return res.status(400).json({ error: 'Missing required Idempotency-Key header' });
}
const cacheKey = `idempotency:${idempotencyKey}`;
const cachedResponse = await redis.get(cacheKey);
if (cachedResponse) {
const parsed = JSON.parse(cachedResponse);
return res.status(parsed.status).json(parsed.body);
}
// Acquire distributed lock to prevent concurrent duplicate submissions
const acquired = await redis.set(`lock:${idempotencyKey}`, 'LOCKED', 'EX', 10, 'NX');
if (!acquired) {
return res.status(409).json({ error: 'Transaction is already being processed' });
}
next();
}typescriptRace Conditions & Optimistic Concurrency Control#
What happens if a user with 80 purchases across different browser tabs?
- Traditional row-level database locks (
SELECT ... FOR UPDATE) degrade performance and cause deadlocks under high concurrency. - In Event Sourcing, the database enforces a
UNIQUE (aggregate_id, version)constraint. - When transactions A and B both rehydrate state at
version = 5:- Transaction A appends first with
version = 6and commits. - Transaction B attempts to append with
version = 6and is immediately rejected by PostgreSQL with a unique constraint violation. - The Ledger catches this collision, reloads the fresh aggregate state (now depleted by Transaction A), and cleanly rejects Transaction B for insufficient funds.
- Transaction A appends first with
Reconciliation & Discrepancy Auditing#
External network connections can drop abruptly—for example, a partner bank may charge a customer card but timeout before delivering the HTTP webhook confirmation.
- Solution: A batch Reconciliation Service runs daily at midnight (T+1), ingesting external bank settlement feeds (ISO 20022 CAMT.053 / BAI2 files).
- The service conducts Three-Way Matching: comparing internal Event Store streams against Gateway access logs and bank clearing statements.
- If an unconfirmed transaction was settled by the bank, the reconciliation engine automatically appends an adjustment event (
PaymentAutoResolvedorCompensatingRefundIssued) to re-align internal balances with actual bank balances.
6. TypeScript Aggregate Implementation#
Here is a clean implementation of the PaymentTransactionAggregate demonstrating state transitions, command invariants, and version tracking:
// 1. Domain Event Definitions
export interface PaymentInitiatedEvent {
type: 'PaymentInitiated';
transactionId: string;
userId: string;
amount: number;
timestamp: string;
}
export interface PaymentCompletedEvent {
type: 'PaymentCompleted';
transactionId: string;
bankRefId: string;
timestamp: string;
}
export interface PaymentFailedEvent {
type: 'PaymentFailed';
transactionId: string;
reason: string;
timestamp: string;
}
export type PaymentEvent = PaymentInitiatedEvent | PaymentCompletedEvent | PaymentFailedEvent;
// 2. Aggregate Root Managing State and Invariants
export class PaymentTransactionAggregate {
public id: string = '';
public userId: string = '';
public amount: number = 0;
public status: 'INITIAL' | 'PENDING' | 'SUCCESS' | 'FAILED' = 'INITIAL';
public version: number = 0;
// Rebuild state by replaying past events
public loadFromHistory(events: PaymentEvent[]): void {
for (const event of events) {
this.applyEvent(event);
this.version += 1;
}
}
private applyEvent(event: PaymentEvent): void {
switch (event.type) {
case 'PaymentInitiated':
this.id = event.transactionId;
this.userId = event.userId;
this.amount = event.amount;
this.status = 'PENDING';
break;
case 'PaymentCompleted':
this.status = 'SUCCESS';
break;
case 'PaymentFailed':
this.status = 'FAILED';
break;
}
}
// Command: Initiate a new payment
public initiatePayment(transactionId: string, userId: string, amount: number): PaymentInitiatedEvent {
if (this.status !== 'INITIAL') {
throw new Error('Transaction has already been initialized');
}
if (amount <= 0) {
throw new Error('Payment amount must be greater than zero');
}
const event: PaymentInitiatedEvent = {
type: 'PaymentInitiated',
transactionId,
userId,
amount,
timestamp: new Date().toISOString()
};
this.applyEvent(event);
return event;
}
// Command: Confirm payment completion from partner bank
public completePayment(bankRefId: string): PaymentCompletedEvent {
if (this.status !== 'PENDING') {
throw new Error(`Cannot complete transaction in ${this.status} state`);
}
const event: PaymentCompletedEvent = {
type: 'PaymentCompleted',
transactionId: this.id,
bankRefId,
timestamp: new Date().toISOString()
};
this.applyEvent(event);
return event;
}
}typescript7. Conclusion#
Combining Event Sourcing and CQRS transforms payment systems from fragile, prone-to-corruption CRUD databases into resilient financial ledgers:
- Complete Audit Trail: Every monetary event is permanently recorded as an immutable fact.
- Independent Scalability: Write paths achieve massive throughput via append-only inserts, while read paths deliver instantaneous customer queries.
- Built-in Resilience: Natural defense against duplicate charges, race conditions via optimistic locking, and automated multi-way bank reconciliation.
References#
- Martin Fowler - Event Sourcing Pattern ↗
- Martin Fowler - CQRS (Command Query Responsibility Segregation) ↗
- Stripe Engineering: Designing Robust and Idempotent Payment APIs ↗
- Designing Data-Intensive Applications - Martin Kleppmann ↗
- Microsoft Architecture: Event Sourcing Pattern ↗
- Enterprise Integration Patterns - Gregor Hohpe & Bobby Woolf ↗