Thomas Ptacek on how to recruit and select programmers

Thomas Ptacek, formerly of Matasano Security, and now at Starfigher, has written a long and detailed article about how he goes about trying to recruit and select programming talent: The Hiring Post.

Ptacek's article isn't the typical four paragraph overview that you'll read in your weekly business magazine; it's detailed and born of years of pain trying to hire security software experts.

But, his essay is so good that I think it will end up on the short list of articles about the software engineering hiring process that are worth going back to over and over, like Joel Spolsky's The Guerrilla Guide to Interviewing, Michael Lopp's A Glimpse and a Hook and The Sanity Check, or Laurie Voss's This is why you never end up hiring good developers.

Let's use Ptacek's own table of contents to break it down:

• The telling success story that motivated me to write this.

  Ptacek starts out by noting how pleased he is with a recent Matasano hire, and how he realized that they had only managed to connect because he'd changed the way their hiring process works:

  A few years ago, Matasano couldn’t have hired Alex, because we relied on interviews and resumes to hire. Then we made some changes, and became a machine that spotted and recruited people like Alex: line of business .NET developers at insurance companies who pulled Rails core CVEs out of their first hour looking at the code. Sysadmins who hardware-reversed assembly firmware for phone chipsets. Epiphany: the talent is out there, but you can’t find it on a resume.

• The part where I join the chorus of people telling you why interviews are terrible.

  Ptacek bemoans the lack of consistency and objectivity of the interview process

  Driven in part by an oral tradition of how the last 20 years of technical job interviews has resulted in terrible hires, interviewers try to assess for “smart and gets things done”. In other words: “subjective and even more subjective x-factor”.

• But at least read this bad thing about interviews, please.

  Ptacek makes an observation I haven't seen widely highlighted, though it should be: the hostility and stress of the technical interview is directly damaging to the potential value of the interview:

  I walked into the conference room to meet him. He appeared to be physically shaking. You didn’t need to be a psychologist to detect the nervous energy; it radiated from him, visibly, like in an R. Crumb cartoon.

  Engineering teams are not infantry squads. They aren’t selected for their ability to perform under unnatural stress. But that’s what most interview processes demand, often —– as is the case with every interview that assesses “confidence” —– explicitly so.

  As Ptacek observes, you don't want to hire people who are good at interviewing; you want to hire people who are good at writing software. So, how do you do that? Well, says Ptacek, the first thing you must do is realize you have a problem:

  For every genuinely competent and effective developer who can ace a tough dev interview, there are many more genuinely competent and effective developers who can’t. While we’re selecting for the ability to control a conversation, we’re missing ability to fix broken 2-phase commits.

• The warm-up trick.

  Ptacek suggests that the first thing you can do is to be open about the process, and about your expectations:

  Not knowing what to expect makes candidates nervous. That’s a pointless handicap.

  At my last firm, we had the “first-call” system. Every serious applicant got, on average, 30-45 minutes of director-level time on the phone before any screening began.

• The work-sample trick.

  Ptacek makes an ever-more-common suggestion. Instead of trying to ask people questions about programming, ask them to do some programming. But as Ptacek notes, this doesn't mean tinker-toy "reverse a string at the whiteboard" gimmicks, but rather a realistic project which you have to carefully design and provide:

  You can’t do this with a trial period where a candidate gets paid to fix whatever random bugs are in the issue-tracker. Your goal is to collect data you can use for apples-apples comparisons, which means every candidate has to be working on the same problems. You have to design a test, create a scoring rubric, and iterate.

• One simple trick that will make your team hate you.

  In this section, Ptacek continues to hammer on the "objective, reproducible, and reliable" aspect of the hiring process:

  Interviewers hate it. Thomas’s First Law Of Interviewing: If interviewers enjoy it, you’re doing something wrong. But we kept doing it, because we found that we were comfortable with “why” we were making hire/no-hire decisions when we had facts to look at. We could look at the results of an interview and compare them to the same data generated by successful team members.

• Grade your hiring process.

  Lastly, Ptacek reminds you not to sit back and relax, but to work to constantly improve your hiring process, just like you'd work to constantly improve any other process at your organization.

  Ask yourself some questions about your hiring process.

  Are you learning both from your successes and failures?

Ptacek really cares about this, so much so that he left Matasano to start a new company to develop his ideas into a realistic hiring tool: Announcing Starfighter

I'm not sure how Starfighter will do; for one thing, it's clearly aimed directly at the "penetration tester" sub-specialization, which is certainly an important job category, but it's just one of thousands of software specialties. I am skeptical that the CTF style of programming will be applicable to other software specialties, but these are some very smart people, and they really believe in their idea, so I'm interested to see how it all goes.