Puzzle-solving interview questions: How you do it can’t save you

In my vitriolic diatribe on puzzle-solving interviews, I argue that making engineers solve puzzles as an interview strategy does nothing but confirm what their transcript already tells you, and over time yields an engineering staff incapable of strategic thinking. To my credit, I did not use the term “boneheaded,” which speaks to my inimitable sense of self-restraint.

Rebuttal from a reader: “Perhaps some real value can be found out of the puzzle question response by looking for out-of-the-box, creative thinking (both in problem analysis and solution exploration). Even if they get the thing wrong (which is not necessarily a bad thing since most puzzle questions are designed to be devious and tricky), they may prove the ability to look at the problem from a unique perspective and think critically as they problem solve.”

Agreed. There is some value in watching a person solve a puzzle in order to see not just whether they can solve the puzzle, but how they approach the solution of said puzzle. But this insight does not address the profound problem with the entire notion of a puzzle-solving interview in the first place: Puzzle solving is not even remotely the most significant skill of a superb professional software engineer!

Software is built by humans within the context of social structures. Software construction is dramatically affected by a personal productivity variable of at least an order of magnitude. That means that my good day is at least ten times more effective than my bad day.  Equally important, one engineer may, on an average day, perform at least an order of magnitude more productively than another engineer on that same average day. Engineering productivity is massively impacted by the collective cognitive and emotional state of a given team, not to mention the dynamics of that team as it interacts with the rest of its company (and even further, with the company’s partners and customers).

If you want to really know whether a candidate is going to save your bacon six months from now, stop asking the puzzle questions. Now.

If you want smart and clever engineers, then limit your recruiting to schools you trust and require a minimum GPA of the students you interview. If that’s not good enough for you, then actually read their transcript and look at how they did in specific courses.

Now that you’ve determined the candidate is smart (see above), try to figure out these three things:

1) Can this candidate think intelligently about a world outside an engineering cubicle?

2) What is the ratio of bad to good days in the engineering life of this candidate?

3) Is this candidate an effective written and verbal communicator?

You then have to develop great interview questions that evoke meaningful answers to each of these three issues. I call them, “The only interview questions you’ll ever need.” All will be revealed in the next few posts.

4 thoughts on “Puzzle-solving interview questions: How you do it can’t save you

  1. I don’t know if I completely agree yet. (I reserve judgement until the promised detail is posted.) I do agree that puzzle questions are mostly a waste. But I do think some questions to get a sense of how somebody thinks are important. How someone approaches a problem is a “soft skill” that is important in terms of culture and creativity.

    I eagerly await your future posts on what sorts of interview questions you think are valuable.

  2. Touché.

    You bring up some great points, Dr. K. I think we agree that some value can be distilled from a puzzle question, and that was the point of my comment you quoted. But I am in agreement with your follow up comments; that fact doesn’t automatically qualify puzzles as “great” interview questions. Why settle for them when you could be asking better, more specific questions that sift the excellent software engineers to the top?

    I suppose the intention of my first comment was to justify that puzzle questions properly utilized (emphasis on properly) are not as harmful to your company’s hiring process as your first post so vehemently asserted. But again, why stop there when there is a better way to interview and find top talent?

    I also look forward to the criteria you will propose in your upcoming posts. Thanks for the great discussion.

  3. I enjoyed these last 2 posts, but it’s left me with some questions. I’ve tried to articulate my questions well and ask them in the context in which I am thinking about them. I’m hoping you will address them in upcoming posts.

    1) Different skills are needed at different levels. If you hire an entry level programmer, you don’t need them to think strategically. You want them to code the product and not waste their time gathering market data. I suspect puzzle questions work better for lower level positions. How should interview questions vary with the level?

    2) I took an organizational behavior class that talked about interviewing. Iit was in favor of a standard set of questions so you can better compare candidates. This makes sense, if you change both the questions and the interviewee, you’ve got 2 variables and can no longer objectively compare candidates. If you have 2 candidates and 1 opening, how can you objectively choose the better one?

    3) Graduates from the same university greatly vary. I was on an interview panel with an intern of ours. The intern and the candidate both spent a significant amount of time at the same university. Our intern was finishing up his bachelors and the interviewee was finishing up his masters. Our intern was eating the masters student for lunch. If you want the top 10%, you need to do your own evaluation, GPA and transcripts won’t cut it. How can you filter out the lower 90% of graduates at a university?

    4) I try to ask every interviewee to write a function that calculates the Nth Fibonacci number. I don’t expect anything fancy, just one that will work–no matter how slow or inefficient. I expect every CS graduate to be able to crank this out in under a minute or 2 because this is a classic example of recursion. I have found a surprising number of people cannot solve this in under 15 minutes and a few couldn’t solve it at all. I find asking several easy programming questions an easy way to filter out those who are missing some skills. My idea is simple, if they can answer all of these questions they are good, if they can’t do one, we don’t want them. It is surprising to see how many people don’t know the basics. If you don’t ask these questions where they “regurgitate code”, how can you test their fundamentals? Unfortunately, I’ve found this to be an issue.

    I’d also like any comments you have on this article: http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html

    I’m looking forward to any answers you have.

  4. Fantastic comments!

    I shall indeed address these issues in future posts, especially these four very insightful and interesting comments/questions from Rick.

    I will also respond in a blog post to Joel’s post on Guerrilla Interviewing.

Leave a Comment