We make use of continuous integration on the project I'm on at the moment. A summary of Continuous Integration (for purposes of this post) is that there is a dedicated computer set up that is constantly checking out the latest version of the source, compiling it, and running a comprehensive set of unit tests and regression tests against it to make sure that we're not breaking existing functionality.
In addition to this, we also deploy a nightly build to a webserver (with this being a web application) that can be tested manually. If testing on this goes well, that release will make it into a testing environment including having exposure to some end users for constant feedback on the progress of our development. These releases (which I'll incorrectly label "UAT Releases" for the purpose of this post) will sometimes contain bugs that can be fixed quickly with a patch. To do this we create a branch from the release code, merge the change into this branch, build, test and then deploy that release... or so we thought!
Even though we've got Team Foundation Server installed, we're using Cruise Control .NET for our build server. Because of this, the process isn't quite as integrated as it could be. We check out the latest version of the source, build it, run the unit tests, if all those are successful we update the assembly version numbers in Assembly.cs, check that back in and then apply a label with the current build number. Seems like a nice straightfoward thing to do, and it works perfectly for a nightly build. Now some of you out there will have spotted the problem with this... those of you that haven't, it's to do with how long this process takes, and the fact that on a nightly build no one is checking stuff in at that time of day.
Yes, as soon as we started doing these during the day, we found that what we were labelling was in fact slightly different than what we were building (or not so slightly, depending on the size of the checkin). Our major problem here was that our action wasn't atomic. If someone did a checkin between our initial checkout and our labelling, their changes were included in the label. As building and testing the project can take between 30 and 60 mins, this is quite a big window during the day.
Our resolution for this was to label the branch *before* doing the checkout, checking out the label, and deleting the label if the build failed (in essence rolling back our changes). Due to the way TFS handles labels, we could check in Assembly.cs with the updated build number and get it included in the label for if we decide to branch from it to patch it or reproduce issues.