Most hiring loops are five conversations that happen to involve the same candidate. Each interviewer asks what they usually ask, forms an impression, and arrives at the debrief with a verdict and a story to support it. The loudest or most senior voice wins, and the decision gets recorded as "gut feel, mostly positive." Three months later you're not sure why you hired this person, and you can't tell whether your process works.
Once you're hiring for several teams, or asking other managers to run loops for you, that stops being a quirk and becomes a risk. This post shows how to design a loop in which every interviewer has a job, every answer is scored against the same scale, and the debrief is a short meeting instead of a negotiation.
Why unstructured loops fail
Two well-documented problems sit underneath most bad loops.
Interviewers don't agree, and don't know it. Kahneman, Sibony and Sunstein's Noise describes how different professionals judging the same case reach noticeably different conclusions, and how interviews are a prime example: the interviewer's mood, the candidate's first minute, and whichever question happened to go well all leak into the verdict. Their remedy is to break the decision into separate assessments, make each one independently, and combine them afterwards rather than discussing first.
Structure predicts better. The industrial-psychology research on hiring, from Schmidt and Hunter's 1998 meta-analysis through later updates, consistently finds structured interviews (same job-related questions for every candidate, scored against a rubric) to be among the better predictors of job performance, and unstructured conversations to be weaker. Laszlo Bock's Work Rules! describes Google's move in the same direction. You don't need to treat the exact numbers as settled to act on the direction.
The PTMA study guide covers both under recruiting and hiring. The rest of this post is the mechanics.
Start with the scorecard, not the questions
Before you open the role, write down what a successful hire in the first year looks like. Then turn it into four to six criteria that interviews can actually test. A scorecard for a senior backend engineer might look like this (hypothetical):
| Criterion | What good looks like | Tested in |
|---|---|---|
| Designs for failure | Names failure modes unprompted; makes trade-offs explicit | System design |
| Writes maintainable code | Readable, tested, sensible structure under time pressure | Coding exercise |
| Debugging approach | Forms hypotheses, narrows methodically, asks for data | Debugging session |
| Works across teams | Has changed another team's mind with evidence, not escalation | Behavioral |
| Learns and adapts | Specific example of changing their approach after feedback | Behavioral |
Three rules keep the scorecard honest:
- Every criterion must be job-related. If you can't say how it connects to the work, drop it.
- Each criterion is tested once, by one interviewer, or at most twice if it's critical. Five people all asking about "teamwork" means four other things go untested.
- No criterion called "culture fit." It's the most common place for similarity bias to hide. If someone wants to raise it, ask which scorecard criterion their concern maps to. If none, it doesn't count. If one, it goes under that criterion with evidence.
Give each interviewer one job
A loop of four to five interviews is usually enough. Assign each person a focus area from the scorecard and the questions to use. They may add follow-ups; they may not swap in their own favorite question.
| Slot | Interviewer | Focus | Length |
|---|---|---|---|
| 1 | Hiring manager | Motivation, scope of past work, expectations | 45 min |
| 2 | Senior engineer | Coding | 60 min |
| 3 | Staff engineer | System design | 60 min |
| 4 | Peer from a partner team | Cross-team work, debugging story | 45 min |
| 5 | Another manager | Learning, feedback, how they handle disagreement | 45 min |
Write the question, a short description of the situation it probes, and what weak, good and strong answers sound like. A two-line rubric per question is enough:
Question: "Tell me about a time you disagreed with another team about a technical decision. What did you do?"
Weak: Describes the other team as the problem; no action by the candidate beyond escalating.
Good: Specific example; the candidate gathered evidence or proposed a trial; the outcome is clear.
Strong: All of the above, plus the candidate understood the other team's constraints and the relationship was better afterwards.
If a candidate answers in generalities ("I usually try to keep everyone aligned"), the standard follow-up is: "Give me one real example. What did you do?" That's STAR-style probing: situation, task, action, result.
Include a short, scripted introduction every interviewer uses, covering what the session will involve and that they'll have time for questions at the end. Candidates who know the format perform closer to their real level, and you reduce the variation in how each person is treated.
Score before you talk
This is the step that changes the debrief most. Every interviewer submits a written assessment before anyone discusses the candidate. No hallway chat, no "what did you think?" in the chat channel.
Use a form with the same structure each time:
- Score for each criterion you were assigned, on a fixed scale with defined anchors (for example 1 = clear concern, 2 = below the bar, 3 = meets the bar, 4 = strong, with no midpoint to hide behind).
- Evidence: two or three specific things the candidate said or did that justify each score. "Seemed smart" isn't evidence. "Identified the race condition in the queue consumer without a hint and proposed an idempotency key" is.
- Concerns and open questions that someone else could probe.
- Overall recommendation: hire, no hire, or need more information.
Require it within a few hours, while memory is fresh. As the hiring manager, submit yours too, and don't read the others' first.
Run the debrief in 30 minutes
With the written feedback in hand, the debrief has a clear shape. Share the scores before the meeting, or on screen at the start.
- Read out the pattern, not the opinions. "Four of five scored design 3 or above; coding has a 2 and a 4." Disagreements show up immediately, and they're about specific criteria.
- Resolve the disagreements with evidence. Start with the interviewer who scored the criterion, not the most senior person. Ask: "What did you see that led to the 2?" Then: "Does anyone have evidence that points the other way?"
- Decide whether the gap can be closed. If it's a real uncertainty, say whether one more targeted session would resolve it. If not, treat it as a no.
- Decide against the bar. Everyone states hire or no hire. The hiring manager decides; the debrief informs that decision and doesn't replace it. Set the bar before the loop starts: "We hire on no scores of 1 and at most one score of 2, offset by a 4 elsewhere." Whatever your rule, write it down in advance so you aren't negotiating it about a candidate you've fallen for.
- Record the reasoning. Two or three sentences on why, linked to the scorecard.
For borderline cases, resist the "let's give them a chance" compromise. Poor hiring decisions are expensive and slow to correct, and an uncertain yes costs far more than a clear no.
Pitfalls to watch for
- Anchoring on the first opinion. If the most senior person speaks first, others align. The written-first rule exists for this reason.
- Over-weighting the last interview. The final session is freshest in everyone's mind. Scores should reflect the whole loop.
- Interviewer drift. After a few loops, people start improvising. Review a sample of feedback forms each quarter: are the scores backed by evidence, and does the distribution look sane? An interviewer who rates everyone a 4 isn't giving you information.
- Pay surprises at the offer stage. Settle compensation ranges before the loop, and share them early. In several jurisdictions, including under the EU Pay Transparency Directive, you're required to tell candidates the range before interviews. Even where you aren't, it saves everyone's time.
- Treating the loop as finished. The point of a scorecard is that you can test it. Six months after a hire, compare how they're doing with how they scored, and adjust the criteria that didn't predict anything.
Your first step
Take the next role you're hiring for and write the scorecard before the job description goes out: four to six criteria, each with what good looks like and which interview tests it. Then ask each interviewer to send their written scores to you before the debrief, and notice how differently the meeting goes.
The question to ask yourself after the next hire: if someone challenged this decision, could I point to evidence for each criterion, or only to a feeling?