To: scottp, martin, dan, aris, karen, ajh Cc: Bcc: Subject: software process Reply-To: As you know, we have a serious problem with stabilising bugfixes. We have all wasted a lot of time dealing with this problem. To fix an evil bug (X), a developer (Q) needs to get two packages (A and B) deployed in a hurry. But heaps of other guff (G) has been committed to the packages since the last stabilisation!! Does this scenario sound familiar? Q has three choices: 1. (painful) remove G, make the stabilisation, and commit G again 2. (risky) convice everyone that G is stable, and include it in the ECP 3. (forgetful) get distracted, and leave X at large for another month I've tried all three of these alternatives; no one of them is pleasant. It's a nightmare situation. This letter is my contribution to solving our change management problem - it is my point of view. It took me quite a few hours to write; PLEASE take the time to read it and give me your feedback. ------------------------------ 1: HOW WE CURRENTLY USE CVS CVS supports a development tree, with branches. Currently, we have a branch for each release of our code (e.g. sina_3_10), and the main trunk is the development version. This means that changes we make in the development version are not automatically included in older releases. This is a good thing! There are two types of changes that developers make and commit to CVS - "bugfixes" and "new features". 1.1: BUGFIXES Fixing bugs is problematic. Most of the bugs in our product occur in more than one release. Once a developer (Q) has found the bug, Q must make repairs in the main trunk (e.g. sina_3_13), and test that they work. Once the bugfix has been tested in the development version, Q must make the same changes in each release branch. There are two ways to do this - manually, or by using the merge facility of CVS. Duplicating the changes manually is a waste of time, and it's risky, because Q might make a mistake in one of the branches. The second option is much safer: CVS is able to automatically merge a change from one branch into another branch. Fortunately Q, unlike many of our developers, knows how to use this feature of CVS. There is a catch - many of our files contain embedded CVS "keywords" and "logs", which are different for different branches, and cause conflicts during the merge process; Q wastes time resolving these conflicts. We could avoid this problem by removing the embedded "keywords" and "logs" from all our files - they are redundant. We would want to modify the betaise and stabilise programs; at the moment they only display log entries if the file has an embedded log. All this trouble with applying bugfixes to multiple releases is an annoyance and a waste of time - that's why many bugfixes don't get applied across all releases as they should. 1.2: NEW FEATURES Adding new features can be even more problematic than fixing bugs. It usually takes a number of days to complete a new feature, sometimes it might take weeks. Suppose that two developers (Q and R) are adding features in the same package (this is quite common). Q hasn't finished his feature, his code doesn't even compile yet, but R wants to build an alpha package to see if her new feature is working. R has to wait until Q fixes his code (what if he's on holiday?), or she can try to fix it herself, or roll back CVS to remove Q's broken feature, or else she can forget about doing any testing. It's a nightmare situation. Suppose that developers add ten new features in a product, but one of these features doesn't turn out right and we decide to exclude it from the release. How do we remove that one particular feature? We would have to go through each file that was changed, look through its CVS log, try to guess when work commenced on the unwanted feature, then roll it back. If other changes had been made in that file since that time, it would be very difficult to remove just the one feature - it might have to be done manually, which is error prone. Another nightmare situation. 1.3: CVS CHECKIN COMMENTS Every time a developer checks something into CVS, CVS requires that her to write some sort of comment describing what she has done. Currently some developers try to write meaningful comments most of the time, and some don't. We have a policy that all checkins relating to bugfixes should contain a line like: BUG: 123432 fixed foo where 123432 is the job number of the bug in the SINA management jobtracker. There is a similar convention for new features. Very few if any developers write these special comments correctly and consistently, and we certainly cannot rely on them being present. Also, it is very easy to accidentally commit a file with the wrong comment, I have done that quite a few times. ------------------------------ 2: HOW WE SHOULD BE USING CVS Change management is the name of the game. We need to be able to select which changes to include in a release. We need to be able to automatically apply changes to multiple revisions. We need to be able to test each change independantly of all others, even when many changes are being worked on for the same package. We need to be able to remove changes easily when they don't work as planned. In order to acheive this, it is necessary to somehow encapsulate each change that is made, be it a bugfix or a new feature. 2.1: CVS CHECKIN COMMENTS ARE NOT RELIABLE One possibility would be to identify changes by their CVS checkin comments. We could require that EVERY CVS checkin comment must contain a management job number, and identify changes in this way. As I've already mentioned, CVS comments are not a reliable source of information, and never will be. We cannot rely on correctly formed comments being present, and we certainly can't rely on incorrectly formed comments NOT being present! For something as important as change management, this is inappropriate. 2.2: ENCAPSULATING CHANGES The only practical way to encapsulate changes is to create a separate CVS branch for each change, and never to check code directly into the release branches. We need only branch files that are actually involved in the change, not the whole package. Changes encapsulated in this way can be added to and removed from releases automatically and independantly, using CVS's merge facility. On rare occassions, conflicts may occur during the merge, but it is easy to resolve these conflicts by hand - much easier than applying a change manually. We can get a list of all changes by listing the CVS branches. Changes that span multiple packages can be managed easily. Another advantage of this technique is that we can use CVS checkin comments however we like in these little branches (twigs?), but program the build system to automatically insert well-formed, meaningful comments in the release branch when we merge changes into it. This means we will have a concise log, with no noise, showing exactly what changes have been added into a particular release; and we will also have separate logs for each change, showing the progress of work in detail. I have experimented with this technique, and I am absolutely convinced that we can resolve all of the "nightmare" problems I have mentioned, without creating any new ones, by managing changes in this way. 2.3: DOCUMENTING CHANGES We should not attempt to store documentation in CVS comments, because there is too much noise to filter out, it is too easy to make mistakes, and we can only add comments when something changes in the package (not, for example, during testing). The appropriate place to document changes is in the SINA management jobtracker, which was designed for that purpose. We can link each encapsulated change to the jobtracker by putting a job number in its branch name. This is a much better idea than putting job numbers in every CVS comment associated with a change, because it only has to be done once. So long as using the jobtracker is optional, many developers (including myself) will not bother to use it consistently - but in fact it is not difficult to use. I think that EVERY change should be associated with a job in the jobtracker, because every change requires documentation. It is essential to docuement testing. We should keep track of the status of a change in the jobtracker, including what testing has taken place. When the change has been comprehensively tested and its quality is assured, a "tick" can be entered in the jobtracker. A change should never be stabilised until testing is complete and documented. 2.4: ALPHA, BETA AND STABLE In order to support this kind of development, it will be necessary to enhance our build system and stabilisation process. We need to support multiple concurrent alpha builds for a package, so that developers can test their changes independently. Only after a change has been alpha tested, and approved for inclusion in a release, should it be committed to the release branch. We will need to modify "buildalpha" so that it merges selected changes into a checked-out, stable base package before building. There may be some issues installing these packages on alpha boxes, becuase it will not be possible to give them meaningful version numbers. We need to add a build facility to merge and commit changes to the release branches, once they have been approved for release and are ready for beta / integration testing, and another facility to remove changes that are no longer wanted from a release branch. We will need to account for the possibility of merge conflicts, which may occur when applying a change derived from one base to a different base, or when applying two changes that touch the same fragment of code. Stabilisation would take place as it does now, by tagging the release branch. It might possibly be useful to have another stage "gamma" between beta and stable, because the "beta" that I have described would be the first opportunity to test integration of changes, and in some cases there may be manual conflict resolutions committed at the betaise stage. ------------------------------ 3. SUMMARY I hope all this makes it clear that it is not just my personal preference, to use CVS branches to encapsulate changes - it's an objectively better way to work, it would lead to greater productivity, and it would save us from a lot of pain and suffering in the dev and systems teams. The points that I have suggested are: 1. Encapsulate changes - Place each bugfix or new feature into a small branch of its own, based on a stable package. 2. Identify changes - Include a management job number in the tag of each change branch. 3. No embedded logs - Remove the embedded CVS logs and tags from code, so that CVS merges will work better. 4. Automate processes - Enhance the build system to do custom alpha builds, and to merge changes in and out of release branches. The problems that these suggestions would solve are: 1. Stabilisation panic - Unstable or unwanted changes have been committed to a package, so the package can't be stabilised. 2. Applying bugfixes - It is annoying having to apply bugfixes manually to multiple releases, so some bugs are neglected. 3. Premature alpha - One developer's unfinished but committed changes get built and break the package for everyone else. 4. No way back - Currently it can be very difficult to undo changes. Please let me know what you think about this. Sam