The promotional model
Yesterday I modified a Bugzilla entry to indicate that the change was ready for QA testing. QA responded that not only was the bug fixed, but it was also working in production. That's when I realized that the build that went to production the night before actually included my fix, before QA had had a chance to validate it. Our promotional tracking failed.
If you manage a live production system, you need a good set of procedures to define how changes are rolled out. It's a common problem. How do you control what changes to into production? How do you notify QA of all the bug fixes that they should be testing? How do you know what version of the software currently in QA? How do you pinpoint an issue to a particular build? Although the problem is a common one, it seems that every place I've worked has had a different solution.
The ideal solution is known as the promotional model. This is the world view whereby builds are promoted from one platform to the next. At a minimum, you need five:
The purpose of these five platforms is to test different parts of the system with reasonable isolation. Programmers can test their individual features in development, and then make sure they don't break each other in integration. Meanwhile, QA is running regression tests on a stable build. The staging area is used to test the deployment process itself before it finally sees the users in production.
The promotional model forms a pipeline. A build of the software moves through this pipeline, stopping at each of the five platforms along the way. When it fails to pass the tests at a particular platform, it falls out of the pipeline and another build comes up behind it. No build proceeds to the next platform until it passes all tests at the previous platform.
The promotional model has no forks. So while it is likely that you will have different builds installed on different platforms at any given time, a build cannot pass another one in the pipeline. Unfortunately, this makes emergency releases extremely disruptive. If QA is testing the new features when you find a problem in production, you have to take the feature build out of the pipeline to allow the fix build to flow through. That means that QA has to stop testing new features in order to perform a regression test on the fix. I've been on teams that have tried to short-circuit this process to their own detriment. Just take it as a given that you will disrupt QA testing to push an emergency fix.
The promotional model is not the same as source control, but they do touch each other. The source control system is allowed to branch. This is necessary when a change needs to be applied to a build that occurred in the past. So while it is not possible for a build to pass another one within the pipeline, it is possible to push through a build from an older branch.
The promotional model is not the same as bug tracking, but they do touch each other. A bug is opened against the particular build currently installed on a particular platform of the pipeline. It should be possible to identify the exact branch and revision that were used to create the build in question. So it should be standard practice for a developer to pull the code in question before looking for the bug. It should be, but in my experience it rarely is.
The industry has known about the promotional model for many years. In fact, some source control systems claim to support the promotional model. They usually support this claim with branching and tagging features. It's not the same thing. To do promotion right, you should implement both source control and bug tracking, and then integrate them with a database that tracks builds through the pipeline.