How to Scale a Ledger, Part IV

Note: This post is the fourth chapter of a broader technical paper on How to Scale a Ledger.

Writing To The Ledger For Recording & Authorization

Now that we have introduced the basic data models that comprise a ledger, we can discuss how a ledger fits into a money movement system. There are two common use cases for a ledger:

1. A consolidated record of money movement that happens in other systems. We call this recording.
2. Approving or denying money movement. We call this authorizing.

Most ledger implementations we’ve seen can operate at scale in only one of these use cases. As a result, when new use cases emerge, or new products are developed, companies tend to build multiple ledgers that aren’t interoperable with each other. The most powerful ledgers can operate in both models, allowing clients to choose at the Entry level, which guarantees they need depending on performance and consistency requirements.

Recording

Recording is all about taking financial events that occur on other systems (like your fintech SaaS products or your bank), translating them into the core data models, and making them available for query by both internal and external customers. The ledger is an async copy of these financial events and is optimized for:

• High throughput. The possibility of 1000s of writes per second (or higher).
• Complex queries. The system is optimized for filtering and aggregating the core data models.
• Eventual consistency. Reads from the ledger may be stale for a few seconds.

The ledger should not approve or deny the money movement from a recorded Entry. This limitation prevents inconsistency between the ledger and the external system.

1. Event Source: The source of the money movement. This can be any service that moves money, for example, a bank, a card payment processor, or a payments API like Modern Treasury. For recorded Entries, the event source is the ultimate source of truth.
2. Application Layer: Your application code that processes events. It’s responsible for translating the event source’s data model into your domain objects and Transactions in the Ledger DB.
3. Domain Object DB: Your application data store. This will store any state from the event that doesn’t relate to money movement.
4. Ledger DB: The consolidated store of money movement events.
   

Notice that the Domain Object DB and the Ledger DB are dotted-lined from the Application Layer, indicating that this connection can be asynchronous and eventually consistent. This eventual consistency enables high throughput. You may be asking “how can I move money confidently with eventual consistency?” The key insight is that you can keep a ledger internally consistent even if it is a delayed representation of money movement. The way to achieve this is by supporting client-supplied timestamps and Account balance versions.

Client-supplied timestamps

Since money movement already occurred by the time it’s recorded by the ledger, clients need a way to specify when the money movement actually happened. The Ledgers API has an additional field on the Transaction data model that’s client-supplied to achieve this.

To preserve atomicity, all Entries on a Transaction inherit the Transaction’s effective_at timestamp.

effective_at allows clients to modify historical balances, so that they reflect the balances as they were in an external system. Using effective_at, we can support querying historical balances on accounts.

Typically, the effective_at timestamp sent by the client will be close to the created_at timestamp set by the ledger. The difference between effective_at and created_at is the delay between getting information from the external system into the ledger—usually on the order of seconds or minutes.

Account balance versions

Allowing clients to modify historical balances introduces a problem—if balances in the past can change at any moment, how can we know which transactions correspond to a balance? Consider this race condition:

1. At time t: A client reads Alice's Account balance
2. At time t+1: A Transaction transaction_1 is written to the ledger with effective time t-1
3. At time t+2: A client asks for all Transactions before T. The result will include transaction_1, but the balance read did not. The client will see an inconsistency between the Transactions and the Account balance.

The Ledgers API allows for consistent Transaction and Account balance reads through fields on Account and Entry.

With these new fields, we can know exactly which Entries correspond with a given Account version. After reading the Account posted_balance at the effective time 2022-08-26T20:47:11+00:00, if the Account version is 10, the Entries that correspond to that balance can be found by filtering Entries by:

• Account ID
• Status of posted
• effective_at less than or equal to 2022-08-26T20:47:11+00:00
• Account version less than or equal to 10
• Not discarded

Use cases

These common use cases work well with recording:

• Displaying account details: account UIs for digital wallets, cards, brokerages, and other account-like products typically have a UI where the balance on an Account is displayed along with recent Entries. Using the Account version and Entry account_version fields, we can ensure that the displayed balance and Entries are always consistent.
• Payouts: Marketplaces collect money on behalf of their customers, and pay that money out on a certain cadence (daily, weekly, monthly). These systems must tolerate a few seconds of staleness, because actual money movement is happening through payment processors outside of the ledger. Using effective_at, the ledger ensures that Transactions are in the correct order, even if they are recorded out of order. Account versions enable displaying to a user exactly which Entries correspond to a particular payout.
• Loan servicing: Systems that service loans are complicated, but generally work asynchronously. Accruing interest is very similar to a marketplace payout—take a snapshot of a past-due balance, get the Account version, and accrue interest based on the Entries that comprise that balance. Applying a payment to a balance benefits from client-supplied timestamps—these systems should compute past due status based on when a payment was processed, not when the lending ledger recorded the payment.
• Crypto: By definition, crypto transactions happen outside of your application’s ledger on a blockchain. Often it makes sense to keep a local copy of those transactions in a ledger database for performance reasons. Because the transactions have already happened by the time they are written to the application ledger, they should be recorded with an effective_at timestamp matching when the transaction happened on the blockchain.

Authorizing

Authorizing is about approving or denying Transactions based on the state of the associated Accounts. It’s characterized by:

• Read-after-write consistency. Updates from Entries are instantly applied to the associated Accounts.
• Lower Transaction throughput. Performance will degrade around 100 Entries per second on individual Accounts.
• Assertions on balances. The system is optimized to maintain invariants on Account balances or versions.

Enforcing approval rules on each Transaction requires a strict ordering of Transactions, which also means that Transactions can only be processed one at a time. This limitation results in higher latencies because Transaction recording can no longer be done in parallel on individual Accounts.

There are two types of rules that the Ledgers API supports, one based on Account versions and another based on Account balance.

Version locking

The simplest approval rule that we can enforce while writing a Transaction is on Account versions. Since the Account version is updated every time an Entry is written to the Account, we can ensure that we are the next Entry written by having the client send an Account version along with the request to create a Transaction. The ledger will reject the write if the current Account version in the database is different from the version sent by the client.

This approval rule is similar to the concept of optimistic locking. We can enforce the rule at the database level using transactions following this algorithm:

1. Start a database transaction
2. Write the ledger entry
3. Update the Account version, including a condition on the current Account version
4. If the Account was updated in 3 (the client-provided version matched the one in the database), then commit the database transaction. Otherwise, roll it back.

Balance locking

Version locking is susceptible to “hot accounts,” what we call Accounts that see a high volume of writes. If the Account version is incrementing at a fast enough pace, some Transactions will never be able to commit. To solve this problem, it helps to step back and think about the main use case for locking in the first place. In almost all cases, clients want version locking in order to enforce balance assertions. For example, the client might want to create a pending Transaction for a card authorization only if there is enough available balance on the card.

To address the most common use case for locking, the Ledgers API supports balance locking at the Entry level. To implement this, we add a few new fields to the Entry data model.

BalanceCondition

These new fields allow us to implement balance assertions when writing Transactions. Consider an Account with $100 available balance, with two Transactions written simultaneously on the account, one for $25 and one for $75. With version locking, the flow would be:

$25 Transaction

1. Read Account balance and version, and check that balance is greater than or equal to $25
2. Write Transaction that includes an Entry on the Account with the version read in the previous step.

$75 Transaction

1. Read Account balance and version, and check that balance is greater than or equal to $75
2. Write a Transaction on the Account, which fails because the $25 Transaction was written slightly before
3. Again read Account balance and version
4. Write Transaction

In total, we made a call out to the ledger 6 times. We can do the same in 2 calls with balance locking:

$25 Transaction

Write Transaction with gte: 0  on the relevant Entry.

$75 Transaction

Write Transaction with gte: 0  on the relevant Entry.

Not only are we calling the ledger fewer times, but also the API better matches our intent to prevent a certain Account balance from going negative.

Implementing balance locking so that race conditions are handled properly and so that all possible combinations of locks on different balances are respected is beyond the scope of this paper. The logic is based on how version locking works—within a database transaction, attempt to write each Entry finding an Account with the required balance, and then check whether the update actually went through, rolling back the database transaction if it did not.

Use cases

These common use cases require authorizing:

• Digital wallets: A digital wallet holds a sum of money for a user that can be withdrawn into a bank account (typically by initiating an ACH credit). It’s important that digital wallets ensure that users have sufficient funds to initiate a withdrawal, which necessitates balance locking. Additionally, many digital wallets support closed-loop payments between users. Authorizing Entries enable such instant payments while making sure Account balances can’t go below 0.
• Card authorizations: In order to respond to a card authorization request, a ledger must be able to know the exact available balance of an Account at a point in time. Balance locking enables this, while also preventing double-spending. Implementations can reserve the authorized amount until it is cleared using pending Transactions.

Automatically recording or authorizing

Most ledgers we’ve worked with in the past implement only one of the recording or authorizing models. Because the guarantees and use-cases for each model are so different, it’s easy to see why. As companies add new products that need recording or authorizing ledgers, they build new ledgers that are optimized for each product. The downside of this approach is that a company needs to staff a crew of engineers who are experts in building ledgers on each product team. It’s an expensive strategy.

We’ve worked with some ledger implementations that get closer to being general purpose by designating certain Accounts as exclusively recording or authorizing. Some ledger implementations even allow different modes to be toggled by on-call engineers—if an Account is experiencing high Transaction throughput, an on-call engineer can put the Account into recording mode.

What we’ve realized by working with companies using ledgers for many use cases is that even an Account settings implementation is not sufficient. For the Ledgers API to be truly multi-purpose and multi-product, the model must be determined at the Entry level. Here are two examples where the same Account needs to both authorize and record depending on the situation.

• Digital wallets: We’ve already covered why digital wallets need authorizing for withdrawals and closed-loop payments. Some Transactions on digital wallets need to be executed regardless of Account state, however. One example is pay-ins—a user adding money to their wallet from a funding source should be simply recorded. Another is financial charges / credits. If the digital wallet is an interest-bearing account, interest should be added to the balance regardless of the Account state. On the other side, if the Account contains a loan that’s accruing interest, the interest charges should be deducted from the balance even if it would make the Account go negative.
• Card accounts: Ledgering the many possible events from a card network for an Account that tracks a credit or debit card balance is complex and beyond the scope of this paper. But even processing the two simplest events—authorization and clearing—requires a mix of authorizing and recording. As discussed above, card authorizations require a synchronous read of the Account balance. After a successful authorization, the card network will eventually send a clearing event to indicate that the reserved funds should be marked as finalized. This event should be recorded by the ledger regardless of the Account state.

We can infer whether an Entry needs to be recorded or authorized based on whether the Entry contains an Account version or balance lock. Transactions may contain a combination of locked and not locked Entries. For example, here’s a sample card authorization Transaction:

card_entry_1 requirements:

• Corresponds to a purchase on the card holder’s Account.
• Should have a balance lock—it should not be authorized if it would bring card_account_id's balance below 0.
• Needs strong consistency. The balance lock cannot be processed without an up-to-date read of the current Account balance.
• The associated Account will not require high throughput. You can’t swipe a credit card 1,000 times per second.

processor_entry_1 requirements:

• Corresponds to the money that will need to be sent to the card issuer processor as part of daily settlement. This money eventually makes its way to the issuing bank, which fronted the money for Transactions across your card program during the day.
• Should not have a balance lock. The card auth should be allowed to go through regardless of the state of the issuer processor’s Account.
• Needs eventual consistency. The system only needs to read the balance of the issuer processor’s Account at the end of the day during the settlement process.
• The associated Account requires high throughput. Every card authorization in your program will include a credit entry that writes to this Account. As your card program grows, this Account can easily experience 1,000’s of writes per second.

The throughput and consistency requirements for each Entry are very different. Even so, the Transaction that contains both Entries must be processed all-or-nothing. As a result, scalable ledgers must be able to process Entries on the same Transaction with different consistency and throughput requirements. Recorded Entries should be processed asynchronously and in batches, but also should not be written if their containing Transaction was not authorized. Additionally, they should not be present in reads from the ledger until the containing Transaction is authorized.

This interplay between authorizing and recording Entries on the same Transaction is a key reason why double-entry accounting is such a difficult engineering problem. A single-entry ledger completely avoids this problem by processing Entries separately, at the expense of consistency. It would be easy for a system to accidentally approve a card authorization, but not include that authorization in a daily settlement with the issuer processor. With double-entry, that is not possible.

Next Steps

This is the fourth chapter of a broader technical paper with a more comprehensive overview of the importance of solid ledgering and the amount of investment it takes to get right. If you want to learn more, download the paper, or get in touch.

Read the rest of the series:

Part I | Part II | Part III | Part V | Part VI