Fuzzer farms

Dave Aitel made a great posting on his DailyDave mailing list the other day. You do read DailyDave regularly, don't you? Dave was in Shanghai for SyScan, which is a security conference with a somewhat unusual format, being held in 4 different locations serially over a period of several months.

At any rate, Dave was talking about some work done by Ben Nagy of COSEINC in the area of fuzzing, and specifically regarding fuzzing of Microsoft Word.

Fuzzing is a technique for searching for bugs in software, specifically searching for security bugs, but it can be used for finding various types of bugs. I first heard about fuzzing in the network protocol field, but I see that its use has expanded to include API fuzzing, data fuzzing, and more. Fuzzers, which are the tools that perform fuzzing, use randomization or exhaustive enumeration techniques to generate a succession of arbitrary semi-random input to a program, and then watch the results. Since the generated input is random, the fuzzers generally don't really care what the program does, so long as it doesn't crash or hang.

Fuzzers have been around for a long time (20+ years), but really took off when the security field began using them to find security vulnerabilities.

So, back to Dave's report on Ben's talk. The talk, which I think covers work Ben has been doing for years, involves ways to throw hardware at the fuzzing problem. The problem with fuzzing is that it can be slow, because you may have to try a lot of random data input before you find some input with a problem.

So Ben has apparently built a fuzzing farm, which allows for automated processing that can, in fact, process a lot of data. The reported results are that they can run 1.2 million fuzz cases per day against Microsoft Word. I'm guessing, but I think that a fuzz case is probably something like:

1. generate a random document that looks enough like a Microsoft Word document to be interesting.
2. Invoke Microsoft Word to open the document.
   
3. See if it came up properly, or if it hung or crashed. If it hung or crashed, record that fact somewhere, saving the random document which triggered the hang or crash.

They apparently have 72 machines in their fuzzer farm, with each machine able to execute 20 fuzz test cases per second, and have discovered that, amazingly, 10% of these fuzz test cases result in crashes or hangs.

That means that they are able to generate about 150 Microsoft Word crashes or hangs per second in their farm.

Now, each crash or hang then needs further investigation. Many of them may be completely boring and unusable as security vulnerabilities. But some of them represent new bugs that could be used by some virus author somewhere to build the next virus. So finding them faster than the bad guys can find them is a Good Thing.

However, as Dave notes, the problem then shifts from finding crashes to evaluating them.

Presumably, the next step will be that somebody will figure out how to automate the assessment of a Microsoft Word crash, to determine whether the crash represents a possibly exploitable security vulnerability.

In fact, such a thing apparently exists!

My friend Gil loves to write tools. I'm very much a tool writer myself, but Gil is a tool writer in the extreme. When I worked with him, I saw that he would always look for an opportunity to write/extend/enhance a tool.

When you get in that mindset, and stay there, you start building some great tools.