a single source, nothing corroborates it

Run the session like a study, not a chat

Recruit through a short screener that keeps the company name out of it, invite from a personal address so nobody studies the site beforehand, and send about three times as many invitations as you want calls. Half an hour each with screen sharing, one person asking and one taking notes, working from a script of open questions plus tasks. Tell the participant you did not build it so they say what they think.

The method

1. 01

   Invite roughly three times as many screened candidates as you want calls with, and send the invitations quickly after the form goes out.

2. 02

   Rank your questions by importance before the calls so you can drop the low-value ones when a participant arrives late, and interrupt politely when the conversation drifts.

3. 03

   Recruit through a short screener form and keep your company's name out of it, emailing invitations from a personal address so participants cannot study the site before the call.

4. 04

   Run the sessions as scheduled 30-minute one-to-one video calls with screen sharing, one person asking questions and a second taking notes, paying each participant a gift card afterwards.

5. 05

   Script the call with an intro, instructions and open-ended questions that also ask the participant to complete tasks, so you can watch whether key information is findable.

What it returned

• 32

  32 invitations produced 10 scheduled calls, a 30% response rate, with a week's delay between the form and the invitations.

• $35

  For Scalyr: $35 Amazon gift cards, $315 and about 6 hours of effort total, producing more than 20 pages of notes, using Google Hangouts and Google Forms.

• Questions he used include what would worry them about the service, how they would describe it to a colleague, and which alternatives they would mention; his sample script is published as a GitHub gist.

• One participant out of ten did not show up.

Sources

• Usability Testing is Easier Than You ThinkGreg Kogan · gkogan.co · Methodology: How We Ran Successful Usability Tests, and You Can, Too · 2014-05-07