Google's Innovation Factory

Here's an interesting paper and the accompanying slides from a Keynote presentation by Patrick Copeland of Google about some of the build/test/scm/etc infrastructure that Google's software development efforts depend on.

This, as they say, is the "money quote":

Code is expected to have high reliability as it is written and we adhere to a socially reinforced code review and check-in practices. Development teams write good tests because they care about the products, but also because they want more time to spend writing features and less on debugging. Teams with good testing hygiene upstream have more time to innovate, and are thus more adaptable and competitive. In addition, there is one source tree and poorly written code is quickly identified because it breaks other people's tests and projects. Aggressive rolling back is employed to keep the tree building "green".

Oh, what the heck, how about two money quotes:

Within milliseconds of a code check-in, our build process will automatically select the appropriate tests to run based on dependency analysis, run those tests and report the results. By reducing the window of opportunity for bad code to go unnoticed, overall debugging and bug isolation time is radically reduced. The net result is that the engineering teams no longer sink hours into debugging build problems and test failures.

Everybody "knows" that investment in development tools, infrastructure, and productivity support is important. But Google not only talk the talk, they back it up with real systems and processes. Very impressive!