Rescreen participants to contextualize results

Why does this matter?

When kicking off usability sessions we tend to start with a couple of "get to know you" questions such as "Where are you from?", "How did you find this study?"

Such questions are good to get the conversation going but completely irrelevant to the usability test. Why not gather the information that does matter for the test? Let's make every question count.

There is a lot of information we could be learning about participants. Most of that is useless for the study. Only those participant characteristics matter that influence the participants' performance, that influence how they understand the product, and how they approach the tasks.

How to do it?

The participant characteristics that typically influence their performance are:

• experience and domain knowledge: experience in the field that we are investigating. We would expect different performance from a power user than from a noob. We would expect different performance from a hands-on user than from a high-level manager who only supervises the actual users of our product.
• familiarity with tools: it's good to know how familiar participants are with our product. Or if they are not familiar with it, what similar tools do they know and use? Users who are new to our product will expect it to work in particular ways because that's what they were used to elsewhere. We will understand their struggles easier if we know where they are carrying their mental models over from.
• Use cases: what has the participant been doing with our product (or a similar product)? What do they use it for? Maybe they are familiar with the field, but not with the exact workflows we are testing.

Having these pieces of information will give us the context to interpret our participant's test behavior correctly.

In theory, we could get this information from the screener survey that precedes the sessions. In practice, however, surveys can only scrape the surface of these topics. Better discuss them in the session.

Extra benefits of rescreening

Push back on excuses: Product Teams may try to repel usability findings by claiming that the participant was e.g. too inexperienced in the domain; not familiar with the target workflows and use cases and so on. If we establish the participant's experience level and familiarity at the beginning of the session in front of everyone, they can't deny the results.

Filter fakers: fakers will not remember exactly how they filled out our screener questionnaire. If we ask them some of the same questions, chances are they will give a different set of lies, so we can spot them.