Bad Software and Expensive Mistakes

Recently, there’s been discussion about an accidental $900 million wire that Citibank sent out while Revlon was restructuring some of its debt. You can read the details here, but in short, the payment escaped controls and was completed, and when the team realized their mistake they asked for their money back. But they couldn’t get it all back.

Although $900 million is an astonishing number, Citibank is not alone. In 2018 Deutsche Bank accidentally wired an exchange $35 billion—$5 billion more than the bank was worth at the time. The worst “fat-finger” mistake I know of happened in Tokyo in 2005, when a trader filed a share order worth $617 billion and sent it to the exchange.

These types of errors happen all the time, and they’re entirely the fault of software, not people.

The Citibank case goes to the heart of why we started Modern Treasury.

In 2015, Sam and I started building a marketplace for individuals to invest in the renovation loans that LendingHome was making. Building a “fractional” marketplace [1] increases the number of transactions in a system. When it really started working, it exploded the number of payments we made—from hundreds to tens of thousands per month—and that’s when we started uncovering the many issues that we now call “payment operations.”

In spite of being the lifeblood of every business, there has been a dearth of modern software built for payment operations in the past half century. So I wanted to share a few of our near misses and provide a perspective on why we believe so passionately that payment operations deserves great software.

Could Citi’s Mistake Have been Prevented?

There are several levels.

Here’s a screenshot of the Flexcube product that the team at Citibank used for this transaction. The team wanted to pay out interest but not loan principal, but in order to do so the software required an internal “wash” account, and a process that involved checking three boxes (and no less: checking only one doesn’t trigger any alerts.) I think we can agree that this product is less than intuitive to use.

In addition to being simple to use, a payment ops product needs to be flexible. For this team, there is no API, so the workflow had been hard-coded years before. It was so out of date that the team had to have a manual to instruct how to work around it. The reason the high amount was being typed in at all was that this “software will only let you pay principal to some lenders if you pretend to pay it to every lender” (source). The workflow is so rigid it’s all or nothing.

Finally, controls. The Citibank team went through three levels of approvals over email. A good payment operations system should dynamically route payments to the right person or persons for release, but only if those users get the information they need to make an informed decision. Over email, that doesn’t happen.

Good people operating bad software will eventually make a mistake.

How Software Like This Gets Built

Act One: The Tech Team Wants to Talk to the Bank

The first realization we had when we set out to solve this problem at LendingHome was that in order to make payments, you must do so through your bank. But banks do not have clean APIs. To compensate, we had to build a custom bank integration.

The US has a single system for wire transfers, the Fedwire system, and that is the common denominator across all banks. But each bank then operates its own “bank core” software on top of the Fedwire system, and that software maintains accounts and ledgers, records transactions, and so on. The bank therefore actually has a separate interface. So we now have two interfaces:

Customer -- Bank -- Fedwire

Building on these interfaces takes investment of time and resources, because bank cores are challenging to integrate with and understand. Some banks use Direct Transmission over SFTP while others provide SOAP APIs. Timings vary. There are different, competing, data exchange formats for things like reporting on errors, and the configurations can vary from banks on identical cores.

When pressed for resources, teams cut corners. It’s hard to justify investing in wonderful design and robust systems for internal processes, so much of what’s out there is half-built, inflexible, and non-intuitive, and rarely gets any attention from the design team.

The result of poor software is workarounds. We had finance colleagues who had Excel wizard skills and a tolerance for manual process pain at which we marveled, but at some point even they would come down to the engineering floor and, in a desperate cry for help, ask us to “JUST COMPUTER IT!”

(We registered that domain a couple of years later. Check out www.justcomputerit.com the next time you’re frustrated with a payment ops problem.)

Act Two: It’s Not One Bank, It’s Many

It’s not just a single bank integration problem. It’s many.

Most companies, once they become somewhat large, operate on more than one bank. So not only does the engineering team have to understand and integrate with one bank, they have to repeat that process for each subsequent bank from scratch, and as we have seen, that’s a scary proposition.

Now we have many integrations:

Customer
-- Bank 1 -- Fedwire
-- Bank 2 -- Fedwire
-- Bank 3 -- Fedwire
-- Bank 4 -- Fedwire

Every bank is a unique filter on the Fedwire system, making it even harder to make it flexible, intuitive software. The challenges of a single bank integration are now proliferating, and the engineering team tasked with this has to grow.

Act Three: The Controller Wants Control

There’s good reasons why bank relationships are sensitive. After all, the executive team has a fiduciary responsibility to make sure company funds are safe and secure at all times. The CFO and their team are concerned about the prospect of the engineering team building an integration to talk to the bank and move money around whenever they want. Not to mention that with each money movement, the accounting team has to track, tag, and reconcile every dollar that moves in and out of every bank account.

“Move fast and break things” is quite literally the worst way to sell software to CFOs.

So now we discover the next layer of issues: a good bank integration is necessary but not sufficient. There must be a smart, easy-to-use, and intuitive app for the controller that allows them to manage the process. Some of the features this app must have include the ability to create rules to manage the API, to monitor what is happening, and to provide context and trigger approvals for actions that are unexpected.

One such surprise happened to us when we were subleasing office space from Salesforce. The accounting team saw a giant payment to Salesforce and came over to us, angry and flustered that someone irresponsible had sent Salesforce several salaries’ worth of cash for what should have been a few CRM seats. We were amused as we had to explain that indeed, the payment was made to Salesforce, yes, that Salesforce, but no, it wasn’t for software. It was rent.

Context matters.

Bonus Act: Idempotency and Reversibility

There are some very specific engineering concerns that anyone building a payment ops system has to keep in mind.

The first one is idempotency, or doing things only once. We put together a post on “Why Idempotency Matters in Payments,” which I highly encourage anyone working in payments to read, because the only thing worse than sending the wrong amount is sending it twice.

We lived this. One day we accidentally double-funded every mortgage: many millions of dollars, paid out twice. That was not a good day. Mercifully, because mortgages are disbursed to escrow, we got all the funds back, but we never forgot idempotency matters in payments after that day.

Wires are not reversible, which makes mistakes particularly scary. There are other payment types, such as ACH, that are. If you sent an ACH and you didn’t mean to, or if someone debits your account without your permission, you can reverse it.[2] Approvals are important, but they’re particularly important for irreversible transactions, such as the wire Citi sent and couldn’t recall after sending.

Thinking About Payment Ops as a Single Continuous Process

Steve Jobs said, “Design is not just what it looks like and feels like. Design is how it works.”

This sums up how we believe payment operations should run. Rather than have many silos for information—from a database to a CSV to a bank portal to an accounting system—we believe in a single piece of payment operations software. This software crosses systems and, perhaps most importantly, it crosses teams. The tech team wants to live in the API, the finance team wants to live in the app, accountants need continuous accounting, and customer service teams need to answer customer requests.

We’ve written at length in the Modern Treasury Journal about these issues. ACH, wire, and paper check account for the vast majority of payments in the US, and yet have seen very little innovation in the last fifty years.

We believe that will change this decade. Join us, build with us, and learn with us.