Content notes:

• Clarity on what success looks like makes it easier to test rough concepts instead of running code

  • Define the customer benefit very concretely

    • Yes to “The budget agreement will resolve to agreement without me needing to chase people down and mediate conflict”
    • lots of good examples
      • Milena - if there’s a functioning compensation system, then people will be able to do things that motivate them, and the company will feel like things are fair, easy to negotiate price
    • No to “I can spend my time at the beach”

  • Think about how behavior would change, and how you would see (measure) that

    • Metrics suck, and Goodheart’s law reigns. So be humble in this work and always change your metrics

    

• Concept tests (word prototypes) step into the “asking opinions” space, so you have to be careful to ask something concrete enough that they don’t lie to you

  • Ask “Do you think this product would work for you and your team?”
  • Listen to the “I don’t think it would work” more than “sure, that’s great!”
    • Ask why ALL THE TIME. “What part would break down?” “Would it be hard for you, or for other people?” “What are the situations it might work in? What are the situations where it definitely wouldn’t work?”

• Testing assumptions rather than testing solutions

  • Use that list of riskiest assumptions to decide what to build first

• Sepehr’s need:

  • it’s a huge space, what small part can I build now?
  • How do I know that a small initial product will solve a real need?
  • How do I know something is a whole product (rather than a set of random features)?

Doing the work

1. Pick the most important problem in the space that you can solve
   1. Have your list of problems
   2. pick the ones you know how you’d solve. No more than 10
   3. Rank them by how painful they are (even if it’s to a small number of people today)
      1. If you can’t rank, consider running a survey. Divide the Dollar or Pairwise comparisons are good survey methods
      2. Save the contact information - these are people you need to test with
2. Define the customer benefit very clearly
   1. You know the goal and the problem, maybe the root cause. What will they be able to do if this pain point is solved? You should know this from your earlier research, if you don’t reach back out to people and ask
3. Brainstorm and narrow on your solution
   1. You might already have this, so only do this if your team has multiple approaches
   2. Describe your solution in words as clearly as possible. This should be 3 sentences not 10. Rubberduck if you need to. Maybe ask ChatGPT if your description makes sense.
4. Sketch a storyboard
   1. Storyboards are like cartoons - they tell a story in a standard narrative arc
      1. Once upon a time, there was a …. - The customer, their context, and their goal
      2. And every day and every day … - The thing they can’t do because of the cause
      3. Until suddenly … They found a solution that did this
      4. Like a magic spell … and worked liked this
      5. Which got rid of the monster … and how it looked
      6. And then they lived happily ever after … the customer benefit realized
   2. SKETCH - you can draw stick figures and sketchy screens.
      1. Drawing will help you think about the interaction in the real world.
      2. It also helps your participant pay attention understand what you are saying
      3. I do it on post-its I can move around on the template until I know what I’m happy with the result.
      4. Sketching tells people it’s OK for them to criticize deeply
5. Test with 3 people at a time
   1. Go to people who had the problem you described
6. Ask concrete questions and probe into their answers
   1. “Do you think this product would work for you and your team?” “In what situations would it definitely work? Would it definitely not work?”
   2. Pay more attention to the people who say it won’t work
      1. Probe into why it wouldn’t work. Ask why. Ask them to tell a story of how it would break down.
      2. But also ask the people who say yes why it would work
   3. Capture your notes and debrief as usual
      1. Don’t just count up how many said yes or no
      2. Find the insights about the necessary behavior of the system. Can you build that?
      3. What’s the smallest part of it that’s needed?
7. Decide if this is your first product to build
   1. If yes, code up only what you need and start running it, even with humans in the background
   2. If maybe, what’s the riskiest assumption? “This will only work if…” How would you test that with no coding?
   3. If no, go to the next solution (or next problem) on your list

Material

Interview guide:

Thanks for giving us time

We want to make sure we are solving a real problem that you have..