Puzzle-solving: Not the driving function of software construction

Yeah, I know I promised three follow-on posts after the last one on puzzle-solving interview questions. I’m getting to it. Really. But I wanted to take a minute first and specifically address some of the comments left on the last two posts.Here’s one:

In my opinion, there’s nothing wrong with hiring engineers primarily for their puzzle-solving skills; that’s their primary function.

I’m going to respectfully disagree. A software engineer’s primary job function is to write software, which involves a number of things, one of which looks a bit like puzzle-solving. But the driving function of the complexity and cost of software construction is neither writing software, nor its puzzle-solving component. The driving function of software construction is the collective emotional state of a software engineering organization.

Individual engineers have to be decently smart. Spectacularly smart is even better. But uber smart with a whiny attitude is a no hire. The smartest new college graduate at some university who can’t stay focused for more than three weeks at a time is a no hire. The kid who solves the Rubik’s cube in less than a minute blindfolded but brings other team members down emotionally is a no hire. The brilliant prima dona is a no hire. The defeatist genius who minored in learned victimhood is a no hire. Every single one of these guys can pass the puzzle questions with flying colors. Every single one is a no hire.

Failure to interview for emotional attitude, team play, communication, and can-do attitude ABOUT THINGS OTHER THAN PUZZLES is failure to interview for a successful engineering corps.

But what about division of labor? Why does an engineer have to think big? Can’t the high mucky mucks think the big thoughts and leave the puzzle-solving engineering crew to hunker down?

To wit, this comment:

When you expect or allow those same engineers to decide the direction of the company is when you’re headed for disaster. Large companies like Microsoft and Google can get away with this practice if they balance out their spiffy puzzle-solvers with globally-thinking big-picture people. Smaller companies can’t afford to specialize and compartmentalize their workforce that way and should follow different hiring practices.

Again I’ll respectfully disagree. The notion that there is some clear separation between the engineers in the trenches and everyone else in the company up to and including the CEO is a myth. The collective attitude of an organization, for better or worse, is manifest in everything the company produces, including the products it ships, the ad campaigns it runs, the personality of its sales force, the attitude of the secretary at the front desk, and the code that engineers produce, not to mention the rate at which they produce it.

Personal case study: In 1988 I was a software engineer in the Personal Computer Group at Hewlett-Packard in Sunnyvale, California. I showed up just before HP’s OEM release of OS/2 1.0 for the Vectra PC, and then stuck around to write device drivers for OS/2 1.1. I had access to all the source code for OS/2, which had been jointly developed by Microsoft and IBM. As I cruised around the code for OS/2, it was always immediately obvious which code had been written by IBM engineers and which had been written by Microsoft engineers. In all cases, the coding style was consistent with the collective corporate attitude that you could see on a billboard from a clogged expressway. You could drop code in front of me and I could tell you which company it came from.

Engineering work, even among the most junior engineering staff members, involves creativity and decision making that has nothing to do with puzzle solving. Individual engineers make stylistic decisions about design issues that were not fully clarified by someone outside their team. Attitudinal decisions that affect code maintainability happen every single day. A renegade codeslinger who refuses to follow the corporate style guide has a negative effect that no amount of puzzle-solving speed can repair.

The idea that there is a wall of separation between the puzzle-solving work of the engineers hunkered down in their cubes and the collective vision of the corporation is fantasy. Engineers who can see outside the cube (both theirs and Rubik’s) write code that blends with corporate purpose. They induce a positive ripple effect that transcends their keyboard and compiler.

Conversely, engineers who primarily solve puzzles are more likely to write unmaintainable code, because they don’t see the world from outside their own perspective. They struggle to figure out why all this coding style garbage is being shoved down everyone’s throat. They view sales and marketing as pure overhead, management as the dark side, and software engineers as the only ones around here who do any heavy lifting.

I agree that big picture thinking is absolutely essential for a small company where everyone ultimately does everything. But big picture thinking turns out to also be critically essential for a large company as well.

Puzzle-solving interview questions: How you do it can’t save you

In my vitriolic diatribe on puzzle-solving interviews, I argue that making engineers solve puzzles as an interview strategy does nothing but confirm what their transcript already tells you, and over time yields an engineering staff incapable of strategic thinking. To my credit, I did not use the term “boneheaded,” which speaks to my inimitable sense of self-restraint.

Rebuttal from a reader: “Perhaps some real value can be found out of the puzzle question response by looking for out-of-the-box, creative thinking (both in problem analysis and solution exploration). Even if they get the thing wrong (which is not necessarily a bad thing since most puzzle questions are designed to be devious and tricky), they may prove the ability to look at the problem from a unique perspective and think critically as they problem solve.”

Agreed. There is some value in watching a person solve a puzzle in order to see not just whether they can solve the puzzle, but how they approach the solution of said puzzle. But this insight does not address the profound problem with the entire notion of a puzzle-solving interview in the first place: Puzzle solving is not even remotely the most significant skill of a superb professional software engineer!

Software is built by humans within the context of social structures. Software construction is dramatically affected by a personal productivity variable of at least an order of magnitude. That means that my good day is at least ten times more effective than my bad day.  Equally important, one engineer may, on an average day, perform at least an order of magnitude more productively than another engineer on that same average day. Engineering productivity is massively impacted by the collective cognitive and emotional state of a given team, not to mention the dynamics of that team as it interacts with the rest of its company (and even further, with the company’s partners and customers).

If you want to really know whether a candidate is going to save your bacon six months from now, stop asking the puzzle questions. Now.

If you want smart and clever engineers, then limit your recruiting to schools you trust and require a minimum GPA of the students you interview. If that’s not good enough for you, then actually read their transcript and look at how they did in specific courses.

Now that you’ve determined the candidate is smart (see above), try to figure out these three things:

1) Can this candidate think intelligently about a world outside an engineering cubicle?

2) What is the ratio of bad to good days in the engineering life of this candidate?

3) Is this candidate an effective written and verbal communicator?

You then have to develop great interview questions that evoke meaningful answers to each of these three issues. I call them, “The only interview questions you’ll ever need.” All will be revealed in the next few posts.

Puzzle-solving interview questions: You get what you ask for!

Interviewing: The process by which an interviewer determines via a face-to-face interaction whether an interviewee possesses the chops necessary to assist the hiring company in its quest for profitability and market domination. (My definition. Feel free to quote me.)

About a decade ago, Microsoft pioneered an approach to engineering interviews based primarily on puzzle solving. The approach became so popular that companies like Google adopted it, books were written on it, and websites dedicated to it. Candidates were even encouraged to consult up to a dozen specific tomes in order to successfully crack the puzzle-solving puzzle that the interviews evolved into. (See the Wikipedia article “Microsoft interview.”)

It bugged me when I first heard about it a decade ago, and it bugs me even more now that other companies have jumped on the bandwagon. I think it’s a pathologically bad practice for reasons I will now reveal.

When you ask a senior in Computer Science to write a function for string comparison in some arbitrary language (for example), you’re testing his or her capacity to either a) regurgitate code they successfully wrote three years earlier, or b) cram puzzle-solving code snippets into their brains in order to impress an interviewer on demand. You might as well ask a candidate to solve a Rubik’s cube blindfolded in less than two minutes for all the value you derive from a question like this.

Seriously, what do you learn from this question?

  1. That the candidate is smart. Congratulations. You just verified their cumulative GPA and spot checked specific courses on their transcript.
  2. That the candidate is a puzzle solver. Congratulations. Next time just break out a Rubik’s cube and be done with it. For the bonus round, break out a blindfold.
  3. That the candidate knows how to prep for your highly arbitrary and ultimately predictable IQ test. Congratulations. You just learned that the candidate is motivated to game your goofy interview process in order to secure a position with you.

My most significant concern is this: Fill your company with a few thousand smart kids who know how to solve puzzles and you’ll populate an engineering staff who can’t think strategically or globally about where the company needs to go, or how products need to evolve. You may find a few who have broader abilities, but your odds of finding them are about the same as discovering new hires that can hit a sweet jump shot or play the cello. You get some by chance, but not because you interviewed for it.

My assertion/prophecy ten years ago was that companies who hired spiffy puzzle solvers via spiffy puzzle solving interviews would ultimately suffer from a vast and pervasive lack of vision and global thinking among its engineers.

Memo to the industry: Your company’s biggest problem isn’t the puzzle solving ability of your engineers. It’s your collective ability to figure out what the market will cough up for the products you never thought about building.

Interview by testing for small thinking? Don’t be shocked when you find yourself working for a fading company devoid of a critical mass of engineers capable of thinking big.

Programming, gender, and masculine metaphors

Dr. Margaret Burnett, one of my professors from my grad school days at Oregon State University, gave a wonderful keynote address at the PPIG (Psychology of Programming Interest Group) workshop in Madrid, Spain on September 20, 2010, entitled “Gender HCI and Programming.” In her keynote, Dr. Burnett outlined seven years of research that she and her colleagues have conducted into issues of gender and computer programming. (For more detail about their research, see the Gender HCI Project page.)

One of the most insightful moments for me was when Dr. Burnett pointed out that  computer programming and systems often employ violent or crude metaphors that are potentially offensive or abrasive to many women. For example, we abort processes, kill programs, spawn zombies and daemons, etc. Who invented these terms? For the most part, guys. And not just normal, run of the mill, let’s shoot some hoop and watch the game sort of guys. But nerdy, hard-core, dungeons and dragons, caffeinate and code all night sort of guys.

I hereby add my own small data point. Some of my kids had been pestering me for a few months to teach them the material from BYU’s CS 100 course, “Introduction to Computing.” The Saturday after Margaret’s keynote address, I relented, and my lecture in the family room was attended by two sons (ages 17 and 14) and one daughter (age 15). At some point, near the end of my lecture, I was talking about programs and how you launch them so that they run. I said something like, “So when you click on the icon, it executes the program.” My sons nodded blankly. But my daughter cried out, “Ew! Execute the program?!” See, that’s the gender gap. My boys didn’t even blink and my daughter is having flashbacks to the French Revolution.

So there, Dr. Burnett. You can add this to the list of seemingly innocuous technical terms that in fact evoke a negative reaction from a teenage girl while her brothers, raised in the same home, don’t even notice. And we wonder why we have a gender gap in Computer Science enrollment.

CBC Radio — Ideas: How to Think About Science

Commuter recommendation: Turn off the classic rock. Now! You can only listen to “La Grange” by ZZ Top so many times before you realize that you’re wasting your drive time every single morning and every afternoon. If you’re commuting 20 minutes like me, it’s a modest waste. If you’re living somewhere more metropolitan, the cost is far worse.

What, pray tell, do you do with the time if not drum on the steering wheel or air guitar on the emergency brake? You equip yourself with some MP3 capable sound transmission device (iPod, iPhone, Zune, doesn’t matter) and start looking for meaningful podcasts to energize your mind and soul during an otherwise monotonous daily commute.

First recommendation: CBC Radio has a regular broadcast program called “Ideas” in which they explore a stunningly broad area of topics. Regular broadcast reception is limited to Canada and the northern U.S. But courtesy of podcasting, you can enjoy all these programs at your leisure.

A year or so ago CBC ran a 24-part series entitled, “Ideas: How to Think About Science.” Great material, and very thought-provoking. From Episode 1 (“Leviathan and the Air Pump”) to Episode 24 (“From Knowledge to Wisdom”) this series presents a fresh perspective on science, research, and the nature of what we consider truth and knowledge. Whether you agree with every point made or every interviewed guest, the program is bound to cause you to examine the way you think about the world.

My preferred access is via iTunes subscription with content synchronized to my iPhone. Total running time is about 24 hours, which took me approximately two months to work through during my modest commute and occasional pedestrian meandering. (For those of you in the Bay Area, you should be able to bang this out in about 3 commuter days… ;)

Enjoy!

Programming: Nature or Nurture, Part II

My most recent post to the PPIG mailing list…

On Jul 4, 2007, at 10:55 AM, Frank Wales wrote:

Charles Knutson wrote:
I believe there is a taxonomy of four types of people, relative to
professional software construction:
1) Those born to code, who need almost no coaching;
2) Those born capable but in need of training in order to be successful;
3) Those not really born to it, but who can be trained sufficiently to make a living;
4) Those whose brains are really not wired to build software at all.


As a matter of interest, do you have a sense of how the general population
is distributed across these suggested types?

Rather than just categories or types, I’d actually propose a spectrum
of capabilities, with people who just ‘get’ computer technology at one
end, people who would sooner eat lint that use it to check their C
programs at the other, and the substantial mass of people bulging
somewhere along the middle. (Of the spectrum.)

I’d speculate, completely without anything beyond years of experience
and arrogant presumption, that the “just get computer technology” end
of the spectrum glows with about 5% of the population, the “couldn’t
write ‘hello, world!’ despite years of practice” end is dimmed by maybe
15% of the population, while the remaining 80% of the population huddles
around various quantumly-distributed embers in between.

I think the end groups are inherent, while the huddles are plastic.


Of course I have no objective evidence for what the actual distribution might be. If anyone has seen any empirical take on this distribution, it would be very interesting. I could easily buy the idea that the top is 5% and the bottom 15%. I also concur completely with the idea of a spectrum. My taxonomy was more of a highly granular way of describing such a spectrum. My apologies to anyone who was expecting a fist fight ;)

Two last thoughts on inherent aptitude:

1) I’ve recently run into students who took the first semester programming class at BYU, did well, got their ‘A’, and then washed their hands of the field and went off to study business or something else. Clearly they are capable of programming, at least at the entry level, but the biggest dissuading factor to them was the inherent motivation (or lack thereof). One of them told me that after spending 15 hours on a programming project, and finally getting it working and passed off, they felt like they had been robbed of 15 hours of their life that they would never get back! That was such a news flash to me! In *my* experience, every program or project I ever finished has carried with it some sense of euphoria and satisfaction that seemed to make all the pain worthwhile. And much of my programming time has been somewhat rhapsodic, like a runner’s high. So while I don’t consider myself to be the most gifted programmer (or even necessarily a member of the first group) I have always been quite satisfied and happy with the process of constructing software.

2) I just started reading Stephen King’s “On Writing.” Interesting to find this quote on page 4 (substitute “computer programmers” for “writers”):

“This is not an autobiography. It is, rather, a kind of curriculum vitae — my attempt to show how one writer was formed. Not how one writer was *made*; I don’t believe writers *can* be made, either by circumstances or self-will (although I did believe those things once). The equipment comes with the original package. Yet it is by no means unusual equipment; I believe large numbers of people have at least some talent as writers and storytellers, and that those talents can be strengthened and sharpened. If I didn’t believe that, writing a book like this would be a waste of time.”

Just a bit more grist for the mill.
Chuck

Professional software developers: Nature or nurture?

The following is an email I just posted to the PPIG mailing list, in response to the following comment:

For the record, I believe anyone can learn to program at a professional level. The question is, are they willing to put in the time to acquire all the chunks needed to be an expert? Unfortunately, we can’t force our students to put in the time.

I’m not convinced that absolutely *anyone* can learn to be a professional X (whatever X is). I think there are some who are really just wired to do other things. But I am confident that there are varying degrees to which inherent aptitude plays a role, and similarly varying degrees to which effective learning experiences contribute to facilitate those individuals who can, in fact, be successful at profession X.

As evidence, I offer the following non-empirical anecdote. I started programming in 1973, when I was 13 years old. Our high school had a timesharing account on a mainframe at the University of Northern Iowa, and a DecWriter with a suction cup modem and a rotary phone with a dedicated line to the university. About a half dozen of us math geeks gathered daily in a room to play with the computer (which largely consisted of playing Dungeons and Dragons and Star Trek, with intermittent fits of attempted software design and code construction). Several of my friends just seemed to have the knack right out of the chute. We’d dumpster dive at the university for discarded manuals, and that was all Brian and Doug needed to build software. I tried desperately but couldn’t get it beyond a fundamental level. The other guys were more or less like me, in love with the technology, but not fluent with the incantations.

Years later, in my second semester at the University of Iowa, I had a really well constructed and well presented Computer Science class that focused primarily on design. During that semester, the light came on, and I got it! From that semester it was simply a matter of learning new skills and piling them onto the foundation I had now acquired. I had a very successful professional career building software (HP, Novell, various small companies and consulting gigs), picked a few graduate degrees along the way, and then retired to the university to stop producing and begin pontificating. :)

As an epilogue, of the group of math geeks that gathered together daily in high school to play with the DecWriter, all but one of us acquired degrees in Computer Science, with the other one (Brian) doing Electrical Engineering.

My personal experience is that I was always fascinated, I was obviously capable, but I needed someone to throw the switch for me to understand how to become self-sustaining after that.

I believe there is a taxonomy of four types of people, relative to professional software construction: 1) Those born to code, who need almost no coaching; 2) Those born capable but in need of training in order to be successful; 3) Those not really born to it, but who can be trained sufficiently to make a living; 4) Those whose brains are really not wired to build software at all.

Just my two cents. Your mileage may vary.
Chuck

Namin’ the Lab

One of my all-time favorite a cappella groups, The Bobs, did a brilliant song in 1989 called “Naming the Band” which includes the following lyrics:

“We’re lookin’ for a drummer
Or someone with a van
Our hair is getting longer
But the most important thing is namin’ the band
Namin’ the band.”

It totally captures the dilemma of naming inanimate objects like bands and labs. With my recent research transition from wirelessness to software engineering, we’ve been going through the pain. The old entity was the “Mobile Computing Lab.” Pretty clean, reasonably catchy, only 862 Google hits — and the first hit is us at BYU!

But who wants to be the “Software Engineering Lab”?! Apart from being polysyllabic and terribly boring and generic, it generates 48,500 Google hits. Who can throw their support behind something that vanilla?! Besides, the TLA (“three-letter acronym”) is SEL. It begs the question, “At what price?!” You could refine it to “Software Engineering Research Lab,” which adds two syllables, generates a slightly silly ETLA (“extended three-letter acronym,” a.k.a., “four-letter acronym”), SERL, and more than 2,000 Google hits. So far no good.

We toyed with “Software Engineering Research Group,” which is also polysyllabic, generates 45,900 Google hits, and sports an acronym (SERG) that suggests a sycophantic relationship with one of the Google co-founders. No good.

“We should be writing tunes
and learning where to stand
Instead we’re spending all our time
Doing nothing but … naming the band”

er… lab…

Refusing to accept long-winded mediocrity, we struggle tremendously with naming the lab. We went through “Software Quality Research Lab” (SQRL, pronounced “squirrel”), and the extended version, “Software Quality Research Lab Big Basic Questions” (SQRL BBQ — draw your own conclusions).

“We were gonna call ourselves Elvis Hitler
But someone beat us to the punch”

We had an inspired idea to call it LASER (“Laboratory for Advanced Software Engineering Research”) until we realized that Lori Clarke and Lee Osterweil at UMass Amherst had already stolen our idea. We then toyed with settling for “Laboratory for *Ordinary* Software Engineering Research” (LOSER). Despite its draw, we rejected it for obvious reasons.

“We’ve got our own equipment
and a great rehearsal space
All we need’s a heavy name
to throw in your face”

Like Archimedes, I had my “Eureka!” moment in the tub while struggling in desperation, scribbling ideas on a partially soaked notepad. Unlike Archimedes, I did not consequently run through the streets of Syracuse (or Salem for that matter) naked. Best for everyone involved really.

Okay. Here we go… (Cue the drummer…)

The Sequoia Lab. SEQUOIA — Software Engineering Quality: Observation, Insight, Analysis.

Everyone in the lab immediately jumped on board. Unanimous consent. One explanation is that the idea was brilliant. Another is that the lab members were sick of namin’ the lab and would have agreed to just about anything I threw myself behind. Another is that a rumor had begun to circulate that I was seriously considering going back to SQRL BBQ.

For the record, “Sequoia Lab” generates only 282 Google hits. Also for the record, all of those labs involve forestry (go figure). Not a single software hit. Looks like it’s ours for better or worse. If we begin to be pestered by the spotted owl people, we can always talk to Mr. Brin for potential lab sponsorship and a convenient name change.