Effective Interviewing

In Thinking, Fast and Slow, Daniel Kahneman gives us a detailed overview of our mind, how we think, our biases, and clever experiments for figuring this stuff out.

BLUF

Overview of Kahneman’s advice for hiring effectively:

  1. First, select a few traits that are prerequisites for success in the position (technical proficiency, engaging personality, reliability, etc.). Six (6) dimensions is a good upper limit.
  2. The traits should be as independent from each other as possible.
  3. You should be able to assess the traits reliably by asking a few factual questions.
  4. List the questions for each trait and think about how you’ll score it, e.g. on a 1-5 scale a. have an idea of what you’d call “very weak” or “very strong”.
  5. Score each trait before going to the next, and don’t skip around.
  6. To evaluate the candidate, add up the 6 scores and firmly resolve to hire the candidate who scored highest, even if there’s another one whom you like better.

Context

Thinking, Fast and Slow, by Daniel Kahneman, explains the way we think:

  • System 1 is fast, intuitive, and emotional
  • System 2 is slower, more deliberative, and more logical.

The book covers how the judgement and decisions of those two systems impact our overconfidence, difficulty in making predictions, and cognitive biases.

Side note: this site has a good overview of common cognitive biases.

On Hiring

The normal state of your mind is that you have intuitive feelings/opinions about almost everything that you encounter (e.g., you like/dislike people before knowing much about them). You often have answers to questions that you don’t completely understand, relying on evidence that you can neither explain nor defend.

One explanation to why this happens (i.e., generating intuitive opinions on complex matters) is substitution: if a satisfactory answer to a hard question is not found quickly, System 1 will find a related, easier question and will answer it instead.

This comes into play when assessing candidates: prediction of the future (how the candidate will perform) is not distinguished from an evaluation of current evidence. I.e., people are asked for a prediction, but they substitute an evaluation of the evidence. This is guaranteed to generate biased predictions; e.g., completely ignoring regression to the mean.

Officer Selection

In his early life, Kahneman spent some time helping a military unit select candidates for officer training on the bases of interviews and field tests. The criterion for successful prediction was the cadet’s final grade in officer school.

The original interview process involved psychometric tests followed by a 15-20 minute interview. The interviewers covered a range of topics and were told to form a general impression of how well the recruit would do in the army. Follow-up evaluations showed that this interview procedure was almost useless for predicting the future success of recruits.

Kahneman was convinced (based on a classic book by Paul Meehl) that simple, statistical rules are superior to intuitive “clinical” judgements. Furthermore, part of the failure of the current process was caused by allowing interviewers to do what they found more interesting: learn about the dynamics of the interviewee’s mental life. Instead, he suggested, they should

  1. use the limited time to obtain specific information about the interviewee’s life in their normal environment
  2. exclude the interviewer’s global evaluation from the final decision-making process a. (concluding from Meehl’s book that global evaluations should not be trusted; statistical summaries of separately evaluated attributes achieve higher validity.)

Kahneman’s initial proposal: (1) interviewers evaluate several relevant personality traits and score each separately. (2) The final score is computed with a standard formula, without further input from interviewers. He

  1. wrote a list of traits that appeared relevant to performance in a combat unit
  2. wrote, for each trait, standardized, factual (i.e., objective) questions about the individual’s life before enlistment
  3. instructed interviewers to go through the traits in a fixed order
  4. instructed interviewers to rate each trait (5-point scale) before going to next trait (to avoid the halo effect)
  5. instructed them: “your function is to provide reliable measurements”

Note: he still collected the “global evaluation” for later analysis.

The new interview process was a substantial improvement (from useless to “moderately useful”).

Surprisingly, he found that the global evaluation was now a good predictor, i.e., it did as well as the sum of the ratings of the traits.

Why?

Because intuition adds value, but only after a disciplined collection of objective information and disciplined scoring of traits.