Humans are unreliable narrators.
Even the most honest interlocutors will confidently say untrue things. We recount false memories, accidentally imply speculation is fact, succumb to cognitive biases, misinterpret causal relationships, struggle with cognitive dissonance, engage in lossy paraphrasing, and kind of just say nonsense sometimes. It’s part of being human! (And, it seems, part of being a language model too).
As professionals, we work to reign in these errors. A lot of the classic advice about effective communication revolves around exactly this error correction: active listening, repeating what you think you’re hearing, responding to surprising information with curiosity – all the greatest hits.
Where things kick up a notch is when you need to conduct an interview. Whether you’re a hiring manager, a detective, or an investigative journalist, you’ll need certain techniques and approaches for getting true and meaningful info from your interviewees. You won’t get far with “How many times in the last year did you do crimes?”
While this is intuitive in the adversarial case, things get less intuitive when your interviewee is ostensibly cooperative. You might think, “This is a friendly low-stakes interview, so I can just ask what I want to know.” Unfortunately not! Even in the friendliest research interview, many question types tend to trigger wild confabulation and misrepresentation – unbeknownst to you or your interviewee.
There are a lot of techniques for conducting effective research interviews, but today I want to highlight one technique that recently leveled up how I conduct interviews, and especially product interviews: story orientation.
Tell me about a recent time
When you’re doing product discovery research, your goal typically is to learn about pains and opportunities in some population. The approach is to regularly conduct interviews with customers or potential customers, continually improving your understanding of how they think and what their needs are. This helps you build much better products than you could from principles alone. It’s pretty sweet.
Now, there are some pitfalls here that are well known. For example, you can’t just ask customers if they want the cool thing you’ve envisioned. They probably don’t even know, and they’ll lean heavily toward saying “Uh sure yeah, I agree that your idea would be cool.” Instead of asking about potential solutions, you’re meant to ask open questions about the problem space. Given that, you might ask questions like:
- “How often do you struggle with X?”
- “What do you take into account when choosing Y?”
- “Why don’t you use approach Z?”
These questions are certainly better than “Wouldn’t it be great if Q?”, but will still tend to produce untrue answers. In her book Continuous Discovery Habits, Teresa Torres explains why:
During a workshop, I asked a woman what factors she considered when buying a new pair of jeans. She didn’t hesitate to answer. She said, “Fit is my number-one factor.” I then asked her to tell me about the last time she bought a pair of jeans. She said, “I bought them on Amazon.” I asked, “How did you know they would fit?” She replied, “I didn’t, but they were a brand I liked, and they were on sale.”
What’s the difference between her two responses? Her first response tells me how she thinks she buys a pair of jeans. Her second response tells me how she actually bought a pair of jeans. This is a crucial difference. She thinks she buys a pair of jeans based on fit, but brand loyalty, the convenience of online shopping, and price (or getting a good deal) were more important when it came time to make a purchase.
I love this example so much. It crystallizes something I’d often seen in interviews firsthand: people feel much more self-aware than they are. If you ask for generalizations, you’ll get confabulations. If you ask for recent stories, you’ll have a shot at building a true model of what’s going on.
This isn’t just true in product discovery interviewing. Asking about past behaviour is a core skill when interviewing for journalism, social science, hiring, or any other field where you’re more interested in how people act than in how they think they act. There’s a good reason job interviews no longer ask “How do you approach conflict in the workplace?” and now instead ask “Tell me about a time you had conflict in the workplace.”
Of course, memory errors still abound when somebody recounts a story. The thing is, when we ask somebody to generalize about themselves, those errors compound. Asking “How much do you struggle with X?” requires somebody to take all their error-prone memories around X and summarize them in a lossy way. They’ll filter through the lens of self-perception. Am I the kind of person who should struggle with X?
Meanwhile, “Tell me about a recent time you struggled with X” makes things more concrete. It plants seeds for “why” follow-up questions, and leads to a deeper understanding.
Levelling up the interview
I saw this starkly in a round of product research I was doing recently. I’ve been asking leaders how they manage their time and attention (a fascinating topic I’ll share more about soon.)
Most leaders strive to spend their time on important things, and they reasonably see themselves as competent. Thus, in my early interviews, questions like “How often do you spend time on things that are urgent but not important?” got me answers about how interviewees keep focused on important work, how they avoid getting bogged down in minor firefighting, and how overall they’re gosh darn awesome leaders. I wasn’t finding many pain points and opportunities.
After reflecting on these initial interviews, I reworked my questions to be more focused on specific stories – behavioural interviewing, if you will. For example:
- “Can you tell me about a time recently when you dropped a ball?”
- “You described planning every week the Friday before. Tell me about how you planned out this week, and how that went.”
- About how many Slack notifications did you defer from yesterday to today?
Many of my interviewees have confidently explained how rigorously they manage their time and tasks, then backpedal and rethink their answers when they’re invited to share specific stories. Stories reveal the delightful mess that is their actual workweek. The interviews are now more fruitful and more fun.
Of course, from a product marketing perspective, it’s still worth understanding folks’ self-deception around their problems. If you solve a problem people don’t know they have, it’s harder to sell them a solution. Critically though, your principal goal is to build something that solves substantial problems people really have. You know, to help save people money or make people money. To do that, you gotta understand what users really do, not just what they think they do.
Even for products running at scale, festooned with telemetry and A/B testing, interviewing is an important part of product development. But when you’re building something new, interviewing is absolutely critical. You need to understand your customers.
So next time you’re interviewing, check your notes. Are you soliciting stories, or just generalizations? Are you mapping out the truth, or just prompting people to speculate about themselves?
Challenge yourself to dig in and find the real story. You’ll make better products – and have more fun doing it.