Coders at Work: Brad Fitzpatrick

Chapter Two of the mesmerizing Coders at Work is the interview with Brad Fitzpatrick.

I hadn't heard about Brad Fitzpatrick before I read the chapter, but I was quite familiar with some of his work. I have visited LiveJournal many times, and I have read a fair amount about memcached, and I've looked at a lot of the work that Six Apart have done.

Early in the interview, Fitzpatrick has this to say about testing:

I now maintain so much code, and there's other people working with it, if there's anything halfway clever at all, I just assume that somebody else is going to not understand some invariants I have. So basically anytime I do something clever, I make sure I have a test in there to break really loudly and to tell them that they messed up. I had to force a lot of people to write tests, mostly people who were working for me. I would write tests to guard against my own code breaking, and then once they wrote code, I was like, "Are you even sure that works? Write a test. Prove it to me." At a certain point, people realize, "Holy crap, it does pay off," especially maintenance costs later.

There's a lot of hard-earned experience in that quote.

But what really resonates with me is Fitzpatrick's instinct to trust nothing until it's been proven by fire, by writing and running tests and proving that it works. That sort of question-every-assumption attitude is crucial to the construction of really solid code.

I was also struck by the way that Fitzpatrick didn't get caught in the trap of over-specializing in any one area, but rather was fascinated by all sorts of different software in the complex systems that now exist:

When I was doing stuff on LiveJournal, I was thinking about things from JavaScript to how things were interacting in the kernel. I was reading Linux kernel code about epoll and I was like, "Well, what if we have all these long TCP connections and JavaScript is polling with these open TCP connections that are going to this load balancer?"

I'm rather a generalist myself, and find myself interested in all sorts of different software, from front-ends and UIs to middleware to networking protocols to file systems and database servers. I think there's lots to learn at all these levels, and lots to explore.

I liked Fitzpatrick's very practical suggestions about how to approach a new library of code:

First step, take a virgin tarball or check out from svn, and try to get the damn thing to build. Get over that hurdle.
...
Anyway, once you have one clean working build, kill it, and just make one damn change. Change the title bar to say, "Brad says, 'Hello world.'" Change something. Even if everything's ugly, just start making changes.
...
Then send out patches along the way. I find that's the best way to start a conversation.
...
When I fix a bug in their product the first thing I do is send them a patch in the mail and just say, "What do you guys think of this?"

I think this is a great approach, to get your hands dirty with real code, and then to start discussing actual code, and actual changes, with the community. I've seen this technique be very successful in the Derby community. Dan Debrunner, in particular, is a big supporter of the "let's talk about actual code" approach. It makes things concrete and specific, and it's remarkable how effective it can be to discuss a patch.

I was quite impressed with Fitzpatrick's approach to solving problems in code:

Thinking like a scientist; changing one thing at a time. Patience and trying to understand the root cause of things. Especially when you're debugging something or designing something that's not quite working. I've seen young programmers say, "Oh shit, it doesn't work," and then rewrite it all. Stop. Try to figure out what's going on. Learn how to write things incrementally so that at each stage you could verify it.

The curse of the voodoo change has a strong hold on people. I've seen many an otherwise great engineer say: "I don't understand what's wrong, but I know that if I just make this small change here, the problem goes away. And I'm tired, and I have other things I need to do, and I'm just going to do this and move on."

It's terribly seductive and Fitzpatrick is completely right: you have to be ever vigilant against this false solution, and force yourself to really do the job right.