Here’s a high quality video of last weekend with slides included.

How do we collaborate through software?

• The CAP Theorem – Eric Brewer
• Eventual Consistency – Werner Vogels
• Event Sourcing – Greg Young and Udi Dahan
• Partial Order

Download the slides

It is official. I have been invited to speak at Ignite Dallas.

In my prior recording of the talk, I neglected to mention the names of some folks who have been influential in collaborative application design. This new recording remedies that oversight.

I also modified the second half of the talk to tell the partial ordering story a little better. It still needs the slides in order to really get it, but fortunately those are coming along.

Give it a listen. Hope to see you next Wednesday!

I am preparing for the upcoming Ignite Dallas event. 5 minutes, 20 slides that auto advance every 15 seconds. The final speaker list has not yet been determined, but I’m hopeful that I’ll have a chance to present.

I will be talking about the current thinking in collaborative applications.

• The CAP Theorem
• Eventual Consistency
• Event Sourcing
• Partial Ordering

The challenge is to condense all of this learning into 5 minutes, make it accessible, and make it entertaining. You can help.

Please take five minutes (actually 4:45) to listen to my rough draft. Leave comments on what I can improve, what I didn’t make clear, and anything I could leave out.

Then order your tickets to the event and see some really great performances. Or submit one of your own.

I now have three sites for topic-specific content.

• Historic Modeling
• Update Controls
• Q.E.D. Code

I have seeded these sites with articles from Adventures in Software. But from now on, articles specific to those topics will go on those sites. Please subscribe to those RSS feeds as well as this one.

I wrote up an example historic model that illustrates very succinctly some of the major points of the theory. It is a simple work item tracker.

So far, the example model can assign developers to projects. The next step is to allow developers to move work items between folders to track progress.

Currently, the model is only in text form. Soon, I will render it in source code using Correspondence. Eventually, it will integrate with TFS and HP Quality Center to act as a front end and perhaps a conduit between those systems.

If you would like to see a simple and reliable way of writing distributed smart clients, please follow along.

People are starting to talk about the concept of append-only models or domain events:

• Ayende on append-only models
• Udi Dahan on domain events
• Greg Young on domain events

It's not a new idea, but it has seen increased awareness recently. The idea is that changes (a.k.a. events, or what I call "facts") become a part of the domain model. Instead of performing CRUD operations on records, a system keeps a history of these changes. The changes themselves are modeled as first-class entities, not as side-effects.

The advantages of this approach are many.

• Auditability - The transaction history is the primary source of record.
• Synchronization - Merge conflicts are easily detected.
• Isolation - Publishers and subscribers know nothing about each other.

But there are some details that I haven't heard others mention, yet. These details are going to determine whether people adopt the tools and practices that will emerge from this idea.

Partial order
Many of the implementations of this idea impose a full order on the changes. Each transaction knows the time at which it is committed. The transactions then line up temporally to flow through the system. Before you can process one transaction, you must first have processed the transactions that came before.

This is a pattern I call the "Transaction Pipeline". It is highly restrictive. The only way to know the true result of a transaction is to know all transactions that have come before. This restriction takes what could be a decentralized model and gives it a central authority. The consequence is a loss of local authority.

In Greg Young's video, he draws a system in which changes flow from a client through a server and back again. The client captures the user's intent, then queues up those changes to be sent to the server in serial. The server examines each change, resolves merge conflicts, and updates a central database. The client runs all of its queries against that central database. The local client has no authority. Even though it knows that the user has requested a change, it cannot promise that change until it sees the results in the central database.

If the transactions were not so strictly ordered, the local client could make better decisions about the results of a change. Rather than ordering changes strictly by time, it could impose a partial order on facts. Facts that are related to one another must occur in the proper order. But unrelated facts are allowed to occur in any order. This allows a local client to make decisions based on a subset of the facts. It knows that it has enough information to make the decision, even if it doesn't yet have all of the information.

Parallel storage
Another common implementation detail is that objects are stored in parallel with changes. As the changes come in, they are stored in a queue. As they are processed, their effects are performed on relational data. The relational model can then be queried for current state.

A perfect example is the architecture of NServiceBus. The message bus is added as a mechanism for disparate components to indirectly communicate with one another. Once the component gets the message, it processes the command in a database. This is a natural extension of the way we build systems today.

Unfortunately, this kind of architecture makes it difficult to query the transaction stream itself. Before we can see the effect of a command, it must be executed against relational storage. A local client cannot easily combine the relational snapshot on the server with the outgoing queue of changes that it wishes to impose.

An alternative is to forego relational storage for a mechanism that can query the changes themselves. To find the current state of an entity, it is only necessary to locate all known changes to that entity. If those changes are organized in the right way, that query is as simple and performant as going to the relational database. But the advantage is that we can query through the message queue, not around it.

Disconnected operation
A smart client is often defined as one that can detect whether it is connected or disconnected, and change modes appropriately. These kinds of systems keep a local database as an off-line insurance policy against the eventuality of being disconnected. When the connection is reestablished, the client sends to the server all of the changes that have occurred since the last synchronization.

The problem with this model is that the client needs to switch modes. In "connected" mode, it acts like a dumb terminal. In "disconnected" mode, it acts like a rich client. This yields twice the code to write, test, and maintain. The developer has to decide which features will be available in disconnected mode, and degrade the experience graciously. On top of that is the complexity of switching modes, and the errors that might arise from inconsistencies between the two implementations.

It would be better for a smart client to operate in only one mode. The changes that the user requests are simultaneously committed to a local database and placed in a queue (preferably as one data structure instead of two). The only difference between connected and disconnected operation is whether that queue is currently being serviced. Either way, the application is blissfully unaware. The users changes are captured nonetheless. The developer only has to write the code once, so there is never a decision whether a feature will be supported while off-line. All features are available at all times.

The devil is in the details
Append-only or domain-event models are a powerful idea. But the fact that you only insert and never update or delete is only the beginning. It's great that events are modeled as part of the domain, but if they are just in addition to the domain then it hasn't been taken far enough. For this idea to gain the wide adoption that it deserves, the patterns and tools must impose partial order, forego parallel storage, and support disconnected operation.

For more information on a pattern and a tool that supports these concepts, please see Historic Modeling and Correspondence.

Greg Young gave a talk at QCon last year entitled Unshackle Your Domain. In it, he offers a solution to CRAPpy -- sorry I mean CRUDdy -- services.

CRUD services expose a set of objects. A client gets an object (Read), makes changes to it, and puts it back (Update, or Alter). One big problem with CRUD is that it is not auditable. There is no record of the change. Sure, you can keep a parallel audit history, but if your system does not rely upon that history, how can you trust it?

Greg points out that accountants don't make changes to objects. They keep histories. They never Update (Alter) records, and they never Delete (Purge). Their work is completely auditable. Their legers have a running total, but that current state is always based on history. Changes, not objects, are the foundation of their domain.

Record changes, not state
Greg proposes that a service can be modeled as a series of changes, rather than a collection of objects. In many domains, the set of changes is just as important as the set of objects. This is the only way to truly trust your audit log.

State transitions are an important part of our problem space and should be modeled within our domain.
- Greg Young

In addition to making a system auditable, this solves a whole host of validation and consistency problems. For example, in a CRUD service, we would model a customer as:

class Customer
{
  identity CustomerId;
  string FirstName;
  string LastName;
  string City;
  string State;
}

As a domain object, it would have validation. For example, the City must be within the State. But what if the customer moves from Dallas, Texas to Tulsa, Oklahoma? There is no method to get us there in one step. We either have to transition through Tulsa, Texas or Dallas, Oklahoma. We have to allow our object to be in an invalid state.

More subtle is the FirstName/LastName pair. There is no validation here, but it is just as incorrect to change one without the other.

Now consider the CRUD operations:

CreateCustomer(Customer c);
UpdateCustomer(Customer c);
DeleteCustomer(Customer c);
List<Customer> GetCustomersByCity(string city, string state);

Suppose one client gets a customer and changes their name. At the same time, another client gets the same customer and changes their address. How does the server reconcile these two changes? There should be no conflict: the two clients performed two independent operations. But because we funnel all of the changes through UpdateCustomer, we cannot easily separate them.

A change-oriented service model
Greg proposes that the service should model the changes as first-class citizens. For example:

change AddCustomer
{
  identity CustomerId;
  string FirstName;
  string LastName;
  string City;
  string State;
}

change RenameCustomer
{
  identity CustomerId;
  string FirstName;
  string LastName;
}

change MoveCustomer
{
  identity CustomerId;
  string City;
  string State;
}

change RemoveCustomer
{
  identity CustomerId;
}

Each of these changes is a command object that is routed through the system. They are stored in a sequential log on the server. The current state of a customer is driven exclusively from this set of command objects. Since there is no other way to change a customer, the audit log can be trusted.

Command/query separation
A command changes the state of the system, and a query examines it.

Queries tend to be synchronous, since the server returns results. Some patterns such as AJAX perform queries asynchronously by providing a callback method to capture the results. Even so, the client is in a waiting state until the results are delivered, so the query is synchronous in essence if not in actual fact.

Commands tend to be asynchronous, since the server returns no results. The client only needs a guarantee that the server has received the command. Message queues are often used for commands.

A query can be retried with no ill effects. But this is not always true with a command. if a command is not idempotent, then we must ensure that it is executed once and only once.

Queries and commands are two very different things. Because they have different requirements, they should be handled with separate mechanisms. RPC-based systems, unfortunately, provide onlyl one mechanism for all messaging. There is no enforcement that queries have no side effects, or that commands return no results. As a consequence, queries and commands tend to be intermingled within a single RPC.

Greg proposes a different architecture. When commands are treated as first-class citizens, the separation between commands and queries is easy. Commands are sent to an audit log, and queries are processed out of a reporting database. Don't let the term reporting database confuse you; to Greg any query is a report.

Eventual consistency
The client workstation, the audit log, and the reporting database are three separate bounded contexts. Each has its own storage schema, its own transaction boundary, and its own interface. The three environments are isolated from one another. The only way for them to reach an agreement is for them to communicate.

Communication takes time. We must allow our distributed systems to be out-of-sync for that time. We cannot guarantee consistency across these bounded contexts at all times. The best we can do is to guarantee that they will become consistent eventually.

Most bounded contexts can interact with relaxed consistency.
- Greg Young

It will take time for data to flow through the system. Sometimes that time is not important. But sometimes decisions must be made with the most up-to-date data possible. Only a domain expert can tell us the difference.

When we acknowledge the fact that communication takes time, we open up a whole new set of discussions with the domain experts. Now we can ask how long is too long. We can start defining service level agreements at the command level. And we can start measuring those times to see when the SLA is violated.

Don't take consistency as a given. Don't automatically define a CRUD service. Take the time to model the changes in your domain. Make those changes part of your discussion with the domain experts. Make them part of your solution.

You are aware of the CRUD operations:

• Create
• Read
• Update
• Delete

Two of these operations are perfectly safe. The other two are destructive.

When you update state, you destroy what used to be there. The old information is lost, overwritten with the new. An audit cannot see beyond the update into the past.

When you delete state, you destroy it outright. I'm referring to a hard delete. A soft delete does not destroy information. But a soft delete is not really a delete.

Make things more explicit
I propose new names for these last two operations, in order to emphasize their destructive nature. Instead of Update, let's call it Alter. An update sounds like your adding fresh data. What you are really doing is altering what was there.

Instead of Delete, let's call it Purge. When you purge something, it no longer exists. It's as if it never was. While you might confuse Delete with a soft delete, you cannot mistake the meaning of Purge.

Auditability
Armed with our new language, let's think about how to audit our systems. Can explain to your legal department how you are Altering records? Can you convince your IT staff that it's OK to Purge data? Don't use terms that sugar-coat the facts. Tell it like it is.

Synchronization
Discrete machines in a distributed system need to exchange information in order to keep each other up to date. If they have each Altered the same data, how do they know which is the correct version? How do they even know that there has been a conflict. The prior version upon which they both agreed is lost.

If one Purges data, should the other Purge it as well? Or should the first Create it again? How can they tell the difference?

Post an article in an RSS feed, then change it. See what happens to the feed aggregators. Create a contact in Outlook and then delete it from your iPhone. See if it goes away.

The data destroyed by Altering and Purging records is precisely the data required to synchronize discrete systems. Retain it.

• Create
• Read
• Alter
• Purge

I think that's more explicit.

Historic Modeling may be a new concept, but you are probably already familiar with a popular example.

A person is a fact. The person exists. Time may pass, and perhaps she'll die, but the fact that she existed will always be true.

The same is true of an animal.

A person might consume an animal. Usually the reason is sustenance, but we can't be sure. We can document the fact that a person swallowed an animal for a reason.

Any of a number of reasons can be given. Maybe we just don't know. Or maybe the person swallowed the animal to catch an animal that they had previously swallowed. The fact that they swallowed an animal in the past is a prerequisite for the reason to swallow another. The prior swallow must come before the reason.

From model to instance
To demonstrate the model, let's create some instances of these facts.

• There was an old lady.
• There was a fly.
• There was an old lady who swallowed a fly. I don't know why she swallowed the fly.
• There was an old lady who swallowed a spider. She swallowed the spider to catch the fly.

Continuing in this manner, we construct a graph of fact instances.

This recitation of facts could go on. Until she dies, of course.