David blog

CQS, CQRS, eventual consistency, what do this all mean?

With the rise and evolution of microservices architectures, there has been some new concepts being develop in the past few years, and also old concepts that are starting to get traction again.

But what do this concepts actually mean? how are they related to each other?

Here I’m trying to give my vision of this concepts, from my research on how they are used, specially by whom I consider to be some of the leaders on the field.

What is CQS?

CQS is a principle that state that you should either have void methods that change state (commands) but don’t return any value, or query methods that return a value but should not modify the state.

An example of CQS is IEnumerable interface in C#. IEnumerable have a MoveNext() command that modifies the state but returns nothing, and a query Current() method that returns the current element, but doesn’t modify the state of the collection.

An example of something not doing CQS is the typical Queue implementation, where you have a Pop() method that remove the top element of the queue, modifying the state, and it also returns the value of this element.

Check this great Gregg Young talk CQRS and Event Sourcing, specifically minute 45:40 where I got this examples from.

CQRS

CQRS pattern main idea is that responsibility of commands and queries should be in different modules.

Different modules can mean different assemblies in your project, it can also be different services in a microservices architecture or it can just be different classes inside your application, even if it is one singe monolith.

Do you need 2 databases to apply CQRS?

You don’t, the pattern can be applied using the same database, it can even be with the same data model. The pattern only dictates to separate the services where you do the commands and the queries. Now, it is very natural that since the queries and commands are separated, that you also use two different data models for them.

CQRS and eventual consistency

In a microservices application, it is very common to have a service sending commands over a messaging system like RabbitMQ, and later another process picking this message up, writing to a user facing read database, and later have some user execute queries against this database.

In this type of architecture you are already applying CQRS, your commands are being processed by a separate service over the network than the service handling your queries. Also, when using a messaging system, you are naturally using the eventual consistency distributed model.

This type of architecture has become so common that is very easy to associate CQRS with eventual consistency, but this concepts are completely different, and as we saw before, you can apply CQRS in a monolith application over a single, strongly consistent database engine.

So, what is eventual consistency?

It is a distributed system model. In a strong consistent model, after a transaction is executed against the system, and it response back with a success, you are guarantee that the entire system will immediately contain the changes made. This is what can be seen in a SQL database with ACID guarantees, with replicas across the network.

In eventual consistency, when a transaction is committed to the system and it respond back with a success, there is not guarantee that the data will be immediately available in the entire system, there is only a guarantee that the whole system will be consistent at some point in the future.

But this model does provide some great benefits, mainly higher availability and lower latency compared to strong consistent models. Additionally to this, strong consistency techniques are a lot harder to implement, or near impossible in the case of microservices systems where you are dealing with different engines like RabbitMQ and a SQL database. Eventual consistent guarantees are usually good enough for most microservices systems.

But this model can cause issues in determined use cases. An example would be when a query needs to do done in the system immediately after executing a command, since the query may not return the updated state from the previous command.

An example where eventual consistency is not an issue could be when you are importing data from an external system, and updating a read side database. Usually there is no need to have the data available to the user immediately after it is inserted in the system, a delay of a few seconds is perfectly fine for this kind of applications.

And event sourcing?

Event sourcing consist on storing the state of your application as an immutable log of events, instead of (for example) using a relational model. This concept is typically associated with microservices, mostly because it is common to use events to propagate changes in your application across different microservices.

But it is perfectly viable to have a single monolith with one database that use event sourcing to persist the application state.

Microservices techniques: Export data out of the monolith

Going from a monolith to a full microservices architecture is not an easy task, and one of the hardest part is breaking the database, choosing which data should be moved out of the monolith, and what should stay in it.

The new service will own data that previously belong to the original DB, but it will need to share some of it back to the monolith. Likewise the new service will also have dependencies on data still belonging to the monolith.

In this post I will focus on the later problem, accessing monolith data required by the new service, but not owned by it.

For example, in my current project we are creating a new Notifications service. We are moving all related tables to the new notifications database. But the new domain still have dependencies on data not owned by it. For example, we have the concept of favorite in our website and applications, which help us decide who to send some of our notifications, but favorites data is not owned by the notifications domain.

There are different ways to extract this type of data, they all have some pros and cons, some of them are generally better, but in the end it depends on your use case.

There are two main ways to access the data needed from the monolith, you can synchronously request the data when you need it, or you can store a copy of the data (a cache) in your local database, in this case you will have to apply a strategy to synchronize your local DB with the monolith DB.

Aggregates with Marten

In my current project we have been using Marten as an event sourcing database.

Marten is a great library that have saved us a lot of time and pain, but we encounter some small use case where we changed the way we create our aggregates. If you want to generate two or more aggregates on the same stream and same transaction using the Aggregate function you will hit your database multiple times, making it in some case better to just load your stream of events once and them create your aggregates from there.

Lets see how we improved this by using a fantasy quests example, similar to the one in marten documentation for the Event Store.

Each stream represents a quest, and it can hold the following events: QuestStarted, QuestEnded, MembersJoined, MembersDeparted, ArrivedAtLocation, PigSlayed and BossSlayed.

For our reading side, we have two aggregates, QuestParty, with general information like the current members of the quest, and MonstersSlayed, containing the pigs and bosses we have killed so far.