I left a well-paying, intellectually interesting job at a large financial institution to join a company with fewer than twenty people and less than a year of runway. My colleagues thought I’d lost perspective. My family was diplomatically concerned. Here’s the honest version of the reasoning. ...
Financial systems are natural candidates for event sourcing. Regulators want to know the state of positions at any point in time. Audit trails are not optional. The need to replay a day’s events to debug a pricing anomaly comes up regularly. These requirements — which other domains treat as optional — map directly onto event sourcing’s core properties. That said, event sourcing in production has costs that the enthusiast literature systematically underplays. Here’s an honest accounting. ...
MiFID II went live on January 3, 2018. The preparation started in 2016. Two years for a set of regulatory requirements that, from the outside, looked straightforward: report each trade to a trade repository within 15 minutes of execution. From the inside, “report each trade” requires answering: which trades? From which systems? In what format? To which trade repository? What constitutes a trade for the purposes of reporting vs. booking vs. settlement? What do you do when the reporting service is unavailable? What happens when the trade repository rejects a report? This is the engineering story of building a system to answer those questions. ...
Kafka 0.11 landed with exactly-once semantics and a lot of marketing. We were running trade event pipelines in a regulated environment and the promise was appealing: no duplicate trades in the downstream risk system, no idempotency logic sprinkled through consuming services. After three months with it in production, the honest summary is: EOS (exactly-once semantics) works as advertised within its scope, and that scope is narrower than it sounds. ...
KDB+ is used in risk analytics, trade surveillance, and market data storage across most tier-1 financial institutions. If you work in finance long enough, you will encounter it. Nothing in your Java background prepares you for it. ...
When I moved to the large financial institution, the team I joined managed the market data and trade data storage layer. The engineering problem was deceptively simple to state: store every price tick, every trade execution, and every risk calculation — billions of records per day — and answer analytical queries over them quickly. The existing system was PostgreSQL. It worked, technically. Queries that needed to run in seconds for trading decisions took minutes. Operational costs for storage were climbing. The database team was spending more time running VACUUM than building features. Understanding why required understanding what time-series data actually is and why it’s different. ...
The first time someone told me the risk calculation system was written in Clojure, I assumed it was a prototype or a skunkworks project. It wasn’t. It processed end-of-day risk for a significant portion of the firm’s trading book and had been in production for two years. Here’s why that decision made sense, and what it was actually like to work in. ...
The job spec said “trading systems.” I assumed it would be similar to what I’d been doing — latency-focused, technically aggressive, small team, fast decisions. I was wrong on most counts, and right for reasons I didn’t expect. ...
When I joined the trading firm straight out of university, I could write Java, profile a JVM, and argue about data structures. I had no idea how currency markets worked. My tech lead gave me one piece of advice: “Read everything you can about how the market actually works before you write any code. It’ll save you from building the wrong thing twice.” This is what I wish someone had explained to me on day one. ...