What is Event Sourcing?

Event Sourcing is an architectural pattern that shows up in many mature information systems. This is the fourth in my three-part architecture series.

Transcript

Eric Normand: What is event sourcing? Hi, my name is Eric Normand and these are my thoughts on functional programming. I have another thought on architecture. I had a three-part series and this is the fourth part. I didn't expect a fourth one.

There's an architecture called events sourcing. I thought I'd try to explain what it is and how it relates to functional programming because it's quite a nice, functional approach to architecture. We're trying to keep track of information about things in the world. Facts, things that happened like events, which is data. Data means facts about events.

Those events are things like the user click to button or maybe it captures more intent like the user wants to buy this or the user changed their email address. The user changed their name. We received an event telling us that the shipment was delivered. These are all events and this information about events. We want to capture that.

https://twitter.com/ericnormand/status/1077232877014863872

One thing that we could do, which is the traditional approach — the approach that doesn't have a name — is to have a mutable flag or a mutable field in the database that it's either a no or it has the delivery date. Once we receive the shipment or we know that the shipment has been delivered, we change the database. We change that field to say the date that it was delivered.

If they change their first name, the user submits a form to change their first name. We just overwrite it in the database and we lose the old name. That's fine because in a lot of cases, you don't care about the history. It's just the way it is. You just care about what's the current name.

There's another approach called event sourcing. What event sourcing does is it says, "Wait a second, why are we storing all these derived data?" We get this event that says the user wants to change their name. Then, immediately, we take action which is change their name.

https://twitter.com/ericnormand/status/1080856950336819200

What we're storing is the result of that action. That could change again. If they change their name again, we've lost that.

We've forgotten that there were two events. We just know the last value. What event sourcing does is it says, "Why don't we just store the events instead of taking action right away on the events? We'll just store the events and then, at any point in time, we can replay all the events up to that time and know what the current value would be."

This event log has a lot more information in it than whatever the current value of something is. This is something that happens a lot. The thing that happens a lot is people say, "Well, people can change their name. So, name has to be mutable." The shipment date will be set at some point. It was either delivered or it's not. That's true, but what it's missing is the notion of time.

I might want to know what was the user's first name at this time three weeks ago, or I might want to know the...a really good example is someone's age. If you have an age, of course, the age is changing. It should be mutable. Except, we often want to ask what was your age at the start of the year. It's not about being 21 today. It's about 21 when it was the day that you had the drink.

Are you 18 on the voter registration day? Not 18 today. There's all these questions of time that are ignored when you just say when you just say ages change. Every day, your age is a little different. Same with first name. People's first names change for reasons. Maybe the user is correcting a mistake. My first name was misspelled on my account and I'm changing it.

Maybe, they had a legal name change, and so their actual name has changed in the legal sense. It's not really a correction. It was right before. There's no place to put that semantics when you say this thing is mutable.

In an event log system, an event sourcing system, you consider that event log to be primary. That is the source of truth. You can calculate so many interesting things from it after the fact. Things that you hadn't even thought about wanting to calculate when you developed this model of a user has a mutable first name.

You could ask how many times do users change their names. You can ask how many times do they change their name. They send in an event to change their name, but it hasn't even changed. There's all sorts of little questions you could ask that weren't included in this model of the user can change their name and now, it's different.

Now, this brings up a bunch of challenges. Before I get to the challenge, let me address why this is a good functional architecture. One reason is that it's data that doesn't change. If you have a mutable first name, it's not data anymore. It's just a string that you could change at any time.

If you have an event log, you record everything indelibly. You write it down and now it's a record of that button getting pressed or the user intending to change their name or what have you. It's stored with the time it happened, all the information around it. That is immutable.

Now, further, functional programming has a lot to say about this because what you do to interpret this log is you run a reduction over it. You run a fold. You reduce this long stream, a linear sequential stream of events, into the current state of the system.

You start with an empty state. You have an event. You interpret it. It changes the state. Then, you have another event. You interpret it. It changes the state. You have another event. You interpret it. It changes the state.

At any point in time, you can take that stream starting from the first event and replay it until you get to the time you want to look at and there you go. You have the state at that time because it's a fold and reduction, whatever you want to call it. You can also call it a catamorphism. That's the more technical term, if you want to call it that.

This is a functional idea. You're taking this immutable stream, you're folding it, and you're getting a value from that. There's some optimizations because if your log gets really long, that's a challenge. You don't want to replay that every time. The way to address that challenge is you can cache the reduced value at different points.

https://twitter.com/ericnormand/status/1071435221445345281

Let's say you want to say we're going to remember the state every 100 messages. Every 100 events, we're going to remember that. At any point in time, the most events you have to replay is 100. It's not too bad. It's cheaper than saving it every time, and 100 events is probably not too long, too much calculation to do. To respond to a Web request or something like that.

Another challenge is you want to be able to look something up by ID. All you have is this big events stream. You don't actually have an entity that is indexed and stuff like that. What people do is they derive an index from it. They have a view that is not canonical, but that's listening to the events stream and building this index.

If you have an index that's users on their first name, so users index by first name. Someone issues an event that says I want to change my first name. The index will be listening for that event. When it sees that, I need to look at this kind of event. Some events have nothing to do with first names. It's OK, I'll just ignore them. But this one I have to look at.

I am going to modify my index so that now, the new first name will point to the record. This is very interesting because it means that you have separated out your architecture from your use cases. If you have use cases, it says we need a really fast index because we're going to be looking people up by their first names a lot. That is a use case that you can then develop.

It's not saying that this is primary to our data model just because we have this use case. It's a very interesting property of this architectural pattern. Those are the main challenges. A long event log will mean that it takes a long time to recalculate the current state. You deal with that with caching. You have the other challenge of having these derived indexes on this log.

It's not very convenient for doing fast operations on like we're used to with stuff like a SQL database where you can just index any of the fields. I like it because it has this other property of separating out the architecture from the use cases.

My name is Eric Normand. You can follow me on Twitter @ericnormand. You can also email me any questions you have, eric@lispcast.com. Also find me on LinkedIn, just search for Eric Normand. I'll see you later. Bye.