Interviewing candidates

Posted on by

A quick note: I’m going to be traveling for much of the rest of June and I haven’t got articles queued up, so the blog will go dark for a bit; see you in July!

In the last two episodes I did reruns of earlier articles on the technical interview process. I thought I might go into a little more detail about how I structure interviews and what I’m looking to get out of them.

I have some primary goals:

  • Prevent bad hires
  • Make good hires
  • Leave the candidate with a positive impression of the company

Of course preventing bad hires is of far higher priority than getting good hires. If we fail to get a good hire, that’s too bad, but if we make a bad hire, that can drag down the productivity of a team for years.

That last point is also key; if we want to hire the candidate then obviously they need to have a positive impression. But I want all the no-hire candidates to have a good impression as well. They have friends who might want to interview. They may be in a position to make purchasing decisions or product recommendations now or someday. An interview is a very expensive “high touch” business process; if we don’t get a hire out of it, at least maybe we can get a customer, or if not that, at least some good will.

The actual interview goes like this.

First I’ll make sure that the candidate is comfortable, has a drink, doesn’t need to use the restroom, and so on.

Next, if I am not the first interview of the day then I ask “how’s it going so far?”

The function of this question is to put the candidate at ease, get the conversation going, and see if they have any concerns about the process so far. In some cases I have information about how the candidate is doing, but it is not intended to be a “gotcha” question. (At Microsoft we were encouraged to give feedback ASAP to the other interviewers down the line so that they could dig into potential problem areas; at Coverity we are more inclined to have every interviewer start unbiased. Both approaches have their pros and cons.)

Then I pick something that looks vaguely interesting and fairly recent on the candidate’s resume, and ask them to give me first the “elevator pitch” — a few sentences describing the project and why it was important or interesting — and then to describe the problem they had to solve in more detail.

This is, again, to put the candidate at ease because they are talking about something they chose to put on their resume and are presumably an expert in. I want to find out first, can they communicate well? You’d be surprised at the number of people who put things on their resume that they cannot describe the purpose of in a sentence or two. Second, what actually did they do on this project? This may come as a shock to you, but people inflate their resumes all the time. By digging into the details of what exactly they did on the project, I get a sense of whether they really are “smart and gets stuff done” or not.

Then we come to the really interesting part: the technical question. The purpose here is to figure out whether or not the candidate will be successful writing the kind of code that they’re going to have to write every day. Now, I know all the criticisms of whiteboard coding questions, and I agree with them. It’s an artificial environment, it’s not how people actually code, it is high stress, the problems are unrealistic, and so on. I try to carefully design coding questions that minimize these difficulties while still telling me whether or not the candidate can actually do the job. (And if a candidate would prefer to not write code on a whiteboard, I can set up a computer as well. But the questions I ask seldom require writing more than, say, six lines of code; most successful candidates should be able to write a half-dozen short lines of code without help from their editor of choice.)

And it really is important. The jobs that I interview candidates for have an unusual set of requirements. Candidates for language analysis tools teams have to be strong in both theoretical computer science fundamentals and able to work on a real-world, imperfect codebases. I’ve had plenty of interviews with academics who understood theory deeply but could not write practical code, and with experienced industry coders who could not tell me what a binary tree was.

A good technical question has characteristics such as:

  • can be stated and solved in a short amount of time with a short amount of code
  • does not require an “aha!” insight that is difficult to come up with under stress
  • is not a “stock” problem or “brain teaser”
  • can be made arbitrarily harder or easier, to better gauge what level the candidate is really at
  • is similar to the sorts of problems the candidate will have to solve on the job
  • tests skills other than straight up coding, like ability to read existing code, or deal with an ambiguous specification

A problem that I used for a long time at Microsoft and works even better at Coverity has this basic outline:

First, here are two public API methods written in C++, each consisting of three lines of very straightforward code. The APIs handle “jobs” on a “server” and are called by a “client”. The scenario is deliberately very vague, but the code is clear. I talk the candidate through each line and make sure they understand it. I’m looking for just a basic understanding of straightforward syntax here. (The vast majority of candidates list familiarity with C++ on their resume as that is a job requirement.)

Next, I tell the candidate that they are tasked with code reviewing this code for any problems whatsoever. I see what sorts of general problem areas they look for when reviewing code they did not write. The code I give has design, robustness, security, testability and portability problems. I see which of those areas they dig into naturally, and if they miss some, I suggest “what if the client is deliberately attempting to damage the server; what can they do?” or “what if I recompiled this code to run on a 64 bit machine?” and see if they take the hint.

This is a real-world skill that I have to use every day — first, in reviewing my own code and that of my coworkers. And second, because I’m in the business of writing programs that look at buggy code and figure out that it’s buggy. If the candidate can’t review other people’s code then they’re unlikely to be successful anywhere. If they cannot find any of a dozen bugs in a program that has only six statements, they’re unlikely to be successful writing analyzers that find bugs.

Here I’m looking for comprehension, critical thinking and basic domain knowledge. Red flags often surface at this point. I’ve had candidates with PhDs in computer science, for instance, who did not know that on a 64 bit architecture, pointers are 64 bits wide. And why should they? As is often pointed out, astronomy is not the science of building telescopes. Knowing a lot about theoretical computer science does not necessarily entail knowing how wide the bus is between the processor and main memory. But a candidate who does not understand how pointers are implemented on typical machines is unlikely to be successful working on a team that designs analyzers that look for errors involving pointer size.

I also see a lot of candidates who know that code is wrong but cannot say what the consequences of it are. Undefined behavior is by definition undefined, but describing what might happen if a hostile client, say, forges a pointer to arbitrary memory and causes the server to execute a virtual method with that pointer as “this” demonstrates that the candidate understands how the compiler actually generates the code.

That’s the first phase of the technical question. The next phase involves design, and maybe some small amount of coding. “You can change the implementation of these APIs but not change the method signatures; how would you fix each of the bugs you found?”

Again, this is a basic job skill that we have to exercise every day: fixing a bug in a callee without being able to change the behavior of the caller. Perhaps the calling code is owned by the customer, or another team, or has already shipped, or whatever.

At this point I am looking for things like:

  • Are there any more red flags? Good heavens, the number of candidates I’ve seen who tried to fix the portability problem by compressing a 64 bit pointer into a 32 bit integer is enormous. There’s no clever trick here that I’m asking them to figure out. You can’t fit twenty pounds of flour into a ten pound sack, that should be obvious, so you need to find another way to solve the problem.
  • What tools are in the candidate’s toolbox? All candidates eventually figure out that they will need to add an auxiliary data structure that maintains a map from a 32 bit integer to a 64 bit pointer, because of the aforementioned ten pound sack. Do they know that there are off-the-shelf map classes? If not, do they have confidence that they could write one?
  • Does the candidate identify the actually hard part of the problem? The hard part of the problem is not realizing that you need a map, but rather ensuring that the keys are unique when a new key-value pair is added to the map. (As I alluded to in the previous episode.) Even when I repeatedly ask “in your sketch of the new code you never say how you compute the value of the key; how is that value generated?” some candidates just don’t follow up on it and keep on trying to tell me how the map works. Or they go back to the previous point and start trying to compress the 64 bit value into a unique 32 bit key, which is impossible, and is the reason why we have a map.
  • How does the candidate deal with an ambiguous situation? The problem is deliberately vague, and the hard part involves generating unique values. If there are two clients each generating two jobs and then shutting down, generating unique 32 bit integer keys is not hard. If there are millions of clients generating millions of jobs then generating unique keys may be very difficult indeed. Many candidate never ask me how many clients and jobs there are in this system, and therefore some generate highly over-engineered solutions, and some generate solutions that do not scale at all.
  • Can the candidate make the connection to an existing solved problem? This is hard to do under stress, but it is still interesting to see. Some candidates very quickly figure out that the problem “generate a unique 32 bit number and don’t re-use it until the client is done with it” must have been previously solved by whoever wrote the 32 bit version of their favorite memory allocator. After all, that’s what a memory manager is: a device which gives you a unique pointer when you ask for one, and does not give it out again until after you say you’re done with it. Therefore the problem I’m posing can be reduced to a previously solved problem. Some candidates exhibit magical thinking about memory allocators and the like, as I alluded to previously. I have had many candidates realize that the problem is analogous to a heap; I’ve only had one candidate say “I’ll solve the problem by having Windows make me a four billion byte heap, and then allocate single bytes out of it to generate unique 32 bit numbers”. That is really reducing the problem to a previously-solved problem!

If the candidate is doing very well then I can easily make the problem harder. Suppose the clients are calling on many threads; what locking strategy keeps throughput high? Suppose the jobs need to survive the computer being rebooted. Suppose the server is on a cluster and can offload work onto other servers. I want to find out just how deep their knowledge goes. Or I can make it easier; suppose there is never more than ten clients. Suppose that jobs starting and finishing are in some reasonable order, like, a job started later than another always finishes later as well. And so on. I want them to walk away feeling like they solved at least some version of the problem.

Finally, I try to save some time at the end for the candidate to ask me questions about the team, the position, the company, and so on. (No candidate has ever said “could you write some code to reverse this linked list for me?” and I would be highly amused if they did. What is sauce for the goose is good for the gander, I suppose.) This is another chance for me to “sell” the company to the candidate, and also I get some insight into what is important to them from their questions.

36 thoughts on “Interviewing candidates”

  1. “If the candidate is doing very well then I can easily make the problem harder.”
    It would be very funny to see an interview with a candidate which reaches the following questions:
    – Suppose the jobs needs to survive the destruction of Earth
    Or the opposite:
    – Suppose the unique number has to be “1”

    Reply ↓

    • It’s very important to have a question that allows you to dig deeper or dig wider (or, just dig a shallow grave). Back when I was at Microsoft, I had a handful of simple questions that allowed me to start a conversation with the candidate. If he/she was doing well, my response would be “great, that works, but what if XXX” and we’d go off on that tangent for a while to see how well the candidate did in that area. If the initial question didn’t let the candidate shine, I’d back off and try in another direction.
      Of course, every now and then you get a candidate whose knowledge and experience are very different from the impression their resume makes. I can remember one candidate where we decided the candidate’s spouse must have written the resume – the candidate had no idea what we I was talking about, and I was only picking skills directly from what was listed (and we found out from conversations that the spouse was in the S/W dev business as well).

      Reply ↓

  2. Good developers eventually understand that making software is about people, not languages or frameworks. Nowadays you can find plenty of technical folks that are able to build great software, but how do you hire developers that build software that people actually want to use? That is a skill that is generally not taught on universities and I rarely see being looked for on interviews.

    Reply ↓

    • A developer who understands people but doesn’t understand languages or frameworks can’t do the job they were hired to do. A developer who understands languages and frameworks but doesn’t understand people can at least be paired up with a good program manager who can figure out what to build. It’s great when you find someone who is good at both areas, but it’s hard enough even to find good coders. Most of them are already being treated well at their current job.

      Reply ↓

      • I still haven’t met a person that calls himself a “developer” that understands people and is not technically great. If a developer really understands people you can be sure she/he has built enough software in his career to not focus so much on languages/frameworks. Like in music, a great composer is the one that focuses on the overall emotion of the song and not the individual notes. You really think you could pair an artist with a PM and get a masterpiece?

        Reply ↓

    • I take your point but I somewhat deny the premise of the question. You figure out what software people want to use by getting the marketing team to do marketing research, by conducting focus groups, by making relationships with thought leaders in industry, by analyzing the data that comes back on the usage patterns for existing software, and by many other things that have very little to do with actually sitting down to write software. And even so, many software projects fail to gain traction in the marketplace.

      Writing software that is correct, robust, secure, performant, and so on, is a precondition of writing successful software, and I know how to weed out developers who write wrong, brittle, unsafe and inefficient code. But I don’t know of any test that can be done to determine if a developer is only going to choose projects that will be a success in the marketplace. Past performance is no guarantee of future results.

      Reply ↓

  3. Nice article. Now I’m waiting in the future for clueless hiring manager that follow this article, measuring the candidate by how correct they are in the syntax / answering the question (not by how they analyze, etc). Later then complain that this article does not work 😀

    Reply ↓

  4. Pingback: The Morning Brew - Chris Alcock » The Morning Brew #1877

  6. Pingback: Dew Drop – June 9, 2015 (#2031) | Morning Dew

  7. I do love the idea that if I’ve got a first in Mathematics, have 25 years experience, have worked one all kinds of systems, languages and architectures, and a history of delivering, someone guy has to put me into some artificial pressure situation to test if I can solve problems or know some bit of knowledge that he/she happens to. Many people don’t perform well in those situations, for a start, and the ones who do are not necessarily the ones you want to hire. I used to use techniques like this many years ago, but gave up on them. I find it more instructive to try to get a friendly dialogue out of people about what they have done and their experiences. You quickly get an idea of their level of understanding,. You aren’t pressurising them, and you are looking at what they know all about, not what you happen to know about. Ultimately you need people who are smart and care about their work. The rest you can develop.

    Reply ↓

    • I have to agree with this comment. Since you are the compose of the question you have some context, or a priori information, which the candidate does not have. There is the additional time constraint and pressure, which is also not representative of real life scenarios.

      At my current place, we had such a quiz and decided to get rid of it since is was not an accurate representative of people’s abilities, unless this is an exercise that spans more than 3 hours. Instead, we bring to the table, relevant architectural challenges we have faced in the field, which candidates typically will have had to solve in their careers.

      Reply ↓

  8. I consider myself a good developer, but I would have failed your interview with flying colors. I think I may have ultimately felt judged and mistreated. I might be successfully weeded out, but I would end up with a bad vibe.

    Each candidate has unique skills that are often not a subset of the interviewer’s. Among those skills are, hopefully, the ability to adapt and grow into the needs of the organization. Diversity in skills, abilities, and perspectives are valuable. Knowledge is not as valuable now that we all have and use the same internet.

    Reply ↓

    • I am somewhat confused by your comment; do you have a development job that does not involve reading code, looking for bugs, identifying security and robustness issues, proposing solutions, knowing what off-the-shelf parts are available to you, understanding the needs of the users of the code, and designing a correct, testable solution that meets customer needs?

      None of those things are things you can look up on the internet.

      Reply ↓

    • Diversity in skills and abilities is only useful if some of the skills and abilities actually apply to the job in question. Beyond understanding the language or syntax, what he’s trying to get at is whether they have problem-solving skills and how they handle ambiguity and change.

      Doing it in a stand-up, pressure situation gives insight into how the person handles those situations. If you only work completely alone, you might never have to share your solutions to a problem with a team, but then, of what use would you be to a team if that’s your style?

      Reply ↓

  9. I was reminded of Joel Spolsky of JoelOnSoftware.com when I read “smart and gets stuff done”. Did you both pick that up at Microsoft or did it just come out of similar experiences?

    Reply ↓

  10. To what extent should interviewees be expected to squawk at code that won’t be robust on platforms that define “int” as 64 bits? If hash1 and hash2 are type “int32_t”, would you prefer to see a hashing function written as “hash1 *= hash2;” or “hash1 *= 1u*hash2;”? In the latter case, would you consider as useful of superfluous a comment that the “1u*” was necessary because the rules of C or C++ would allow the compiler to render the code meaningless on a platform with 64-bit “int”?

    My personal belief is that massive amounts of programmer time would be saved if the Standard were to eliminate possible Undefined Behavior from constructs like “uint32_t hash1,hash2; … hash1 *= hash2;” where values are coerced into unsigned types smaller than “int” [I have no problem with the idea that MAX_INT+1 should not be required to wrap around to MIN_INT, but in cases where the upper bits of a value can have no defined effect on the result I see no benefit to allowing them to cause Undefined Behavior]. With the standard as it is, the construct without the type coercion won’t be safe if “int” is 64 bits, but at the same time I think the type coercion is ugly. What would you consider the right approach?

    Reply ↓

  11. You obviously require a very specific set of knowledge and experience and experience and are holding up a high standard and that’s fair enough. I’m wondering if you are stringent enough in filtering out C.V in the first place i.e. only considering candidates with clear knowledge and experience, rather than one or the other.

    An alternative might be to design a practical task for an open day where any candidates that fit a minimum profile can attend and only say the top 5 or 3 can would proceed to interview? In that circumstance the onus would be on you to design a problem and environment that would emulate their working conditions and to establish some objective measures on their output.

    Reply ↓

  12. Having sat on a Codility test and failed miserably, I would say that this style of interviewing is somewhat flawed, especially if the question is not concise and the requirements very clear. Practically, when a dev is assigned a task with a set of requirements, they may have to make multiple round trips with the PM to clarify ambiguity, which typically exists, before committing to a solution. And if this is a solution that requires more than say 6 LOC, experienced devs typically like to come up with some quick design capturing one or more alternate solutions, before committing to coding. If the design is right, typically the task of coding is relatively simple and a lot of these quizes are testing your ability to code, since there is insufficient time to come up with a proper design. If this is a bug fix, one still has to understand the code and have some context. You may have to talk to the original author or reverse engineer the code. If the problem is being posed as an essay, it better be very clear, concise and succinct, otherwise, not many devs will get it.

    I get the whole notion that someone with 15+ years of experience should know what a binary tree is. But outside of university and probably in specific types of software, does it not suffice to only know what type of C# collection I should use: Dictionary vs LinkedList vs Queue vs Stack etc? Do I need to fully understand the underlying implementation when there is all this other stuff I have to know to compose a total solution?

    In my current place of work, we have gotten rid of such quizzes for senior dev positions, simply because they emulate an artificial environment which is not representative of the day-and-life of what we do. We reserve such for entry level simply because this is the only skill we need to judge given their lack of practical experience. Also, they are fresh out of school and should still be able to relate to some of these theoretical questions.

    I do understand that you are looking for a particular skill set and that is all fair. Hence your questions may be tailiored towards software optimization, security and performance. However, a more practical setting may be to give senior candidates some sort of practical challenge based on tough problems you currently are dealing with or have dealt with in the field, of course without divulging much, and give them about 24-48 hours to research into possible solutions and present their result. The candidate’s result can then be discussed in the interview, much like is done when defending a thesis. I suspect this will be practical and representative of today.

    Reply ↓

    • Many kinds of algorithms and data structures can fall victim to some weird corner cases. Most of the corner cases won’t be relevant in most applications, but an understanding of what corner cases one should be concerned about and which ones will be taken care of “automatically” can be essential when trying to distinguish software that is “correct” from software that “seems to work”.

      Reply ↓

    • Regarding the tests that you mentioned, well I’m not sure what kind of tests you used and honestly I don’t have much experience with those testing platforms but I’m familiar with TestDome (http://www.testdome.com/).
      My current company uses them and essentially they have the “good technical question characteristics” that are mentioned in the article.

      “they may have to make multiple round trips with the PM to clarify ambiguity”
      I believe these tests should not have any ambiguity about its requirements, the task needs to be straightforward.

      “If this is a bug fix, one still has to understand the code and have some context.”
      Yes I agree, to fix a bug you need to understand the concept that is presented to you, but if you are making a bug fix test then you’ll definitely provide a self explanatory and very clear code.
      Also as mentioned in the article: “If the candidate can’t review other people’s code then they’re unlikely to be successful anywhere.”

      Regarding the data structure implementation tests, in my previous company they got rid of those tests even for entry level, well… actually sort of…
      You see I remember a colleague that conducted a rather interesting and quite successful tests for interns and juniors.
      He would start of with a simple (and by know probably a well known) FizzBuzz test, after that he would just pore out more and more requirements for which the candidate needed to adjust his previous solution.
      To me the impressive thing was that he literally had hundreds of ideas for those additional requirements, variating from the simplest ones (like just a parameter changes) to complex ones (like consuming some data structure as an input data source).

      Reply ↓

  13. ‘No candidate has ever said “could you write some code to reverse this linked list for me?”’
    Well, thanks for the suggestion: next time I’m looking for a job, I swear to God I’ll do that. See if they are worth my time, after all! 🙂

    Reply ↓

  14. Pingback: FractalizeR’s WebSite » Is X a good technical question for an interview?

  15. Pingback: News-y programistyczne 11-07-2015 | Blog Programisty.NET

  16. I recently attended interview in Microsoft where I was asked to connect the siblings of the binary search tree. When I did that the interviewer did not agree with my code. Why ? Because she mugged up some other way of implementing it. I am serious. No hard feelings.

    Reply ↓

  17. ‘No candidate has ever said “could you write some code to reverse this linked list for me?”’

    I ask interviewers similar questions in the end. This is usually the moment when the person just asked me “how to write a program which should outlive The Earth” struggles with something which is way simpler than reversing the linked list.

    Reply ↓

  18. Pingback: Where do you see yourself in 3-5 years? | Seyoung Jeong

  19. Pingback: Top Marijuana Addiction Choices — Форум риелторов

  20. Pingback: Facts, Fiction and non 12 step – gmgformation

Leave a comment