Which change caused this test failure?

A test failed this weekend.

We hadn't been running our tests as often as we should, so it wasn't immediately clear what had caused the test failure. The best that the Build Farm could tell us was that it had happened in the past few days, and that it was one of about 15 changes to the code that we had made during that time.

So, I used a feature of the Build Farm, and submitted 2 builds:

• One build against Perforce changelist 122438, which was about 5 changelists back from the head of the code,
• One build against Perforce changelist 122401, which was about 10 changelists back from the head of the code.

After a couple hours, the Build Farm came back with the results: the test passed in both builds.

This told me that the change was in the last 5 changelists. One more build and we were able to identify the precise changelist which caused the test to fail.

Some methodologies perform a complete build after every single change, but for us that's very hard because complete test runs take hours or days.

So we run our build and test runs less frequently, which leaves us exposed to the problem of trying to figure out which change caused a particular test failure.

Sometimes it's obvious, but when it's not, as long as we are careful about submitting small, focused changes in each changelist, Perforce can help us figure out which change caused the test failure.