Michael Howard on Fuzzing

Check out this article from Michael Howard from Microsoft:One Tool Does not Rule them All

You would of course want me to comment on (rather counter-argument) this, and I will not fail you.

Michael starts with a comment:

If you read much about security, you’ll see that fuzzing is a very effective security and reliability testing technique, but it is far from perfect.

Well, fuzzing is not one technique but at least three. Depending what technique you use, you either catch only simple flaws (or no flaws at all) or you will catch most remote flaws. But nobody has ever claimed fuzzing is perfect. The study from Charlie Miller (check out chapter 8 of the Fuzzing book by Takanen, DeMott and Miller) indicated that model-based fuzzing finds about 80-90% of flaws, whereas random (dumb) fuzzing only finds between 0-30% of the flaws. Evolutionary fuzzers are difficult to map in this comparison as sometimes they are able to find almost as many flaws as model-based fuzzers (especially with simple text-based protocols like Michael had) but sometimes fail to find any (with more complex protocols).

Not catching flaws such as shown in the example just means you are using the wrong tool, or you need to educate the people building the tools in how systematic fuzzing is done. Just doing fuzzing means nothing. The important question is: What type of fuzzing you are using?

Fuzzing is never the only tool you should depend on when building a product security practice. But it is the most effective one, and the fastest to deploy. And did I remember to say that there are no false positives? A crash is a crash (if you do not attempt to hide it with exception handling, of course). That is why most people will start out with fuzzing, and then move to other techniques that cost much more to use, such as code auditing.