Moderated usability testing
Testing our applications on real users is a great way to get direct and specific feedback in order to understand where any problems lie..
Usability testing is a way of identifying parts of the user interface that we can focus on developing, which can lead to a massive improvement in the user experience.
Why to use the method
Get in-depth feedback about a specific screen of the user flow
Test out the screen or user flow on 5 people which results in something that provides a lot of insights in a short space of time
Testing on bitcoin products
Doing usability testing on bitcoin products is unique in the sense that we are asking people to possibly share sensitive and personal information with us.
There are 3 main considerations when testing:
Conducting tests
Testing applications requires some sensitivity since a person's personal wallet will contain too much personal information. We usually do tests on prototypes or on a test account. Below you’ll find some information about each approach:
Prototype: Testing a design concept on a prototype is also a good solution, clickable prototypes can be created with Figma or with other prototyping tools
Test account: Creating a login username and password for a bitcoin wallet which could then be provided to people when testing
Rewarding participants
It’s always nice to reward participants, a quick and easy way would be to send them over some sats. Fiat is also an option but if they have a bitcoin wallet then it’s a nice gesture to send over some stats after the test. Paying with bitcoin is somewhat easier than fiat as it's open and borderless protocol. We can test users around the world and do not pay high fees or wait a few days for wire transfers.
We can also do a draw to reward one participant, so for example let all participants know that there will be a draw after the sessions and one person will win 50 dollars (sats equivalent)
Know beforehand how the payment will be made: lightning/bitcoin address, standard lightning invoice, on-chain or some other way?
If you're paying with bitcoin, but compensation is priced in fiat, take into account exchange rate volatility (especially when sending on-chain)
Take into account fees, and make sure participant receives the full promised amount
Make sure participants know how to receive payment
Privacy & sensitive data
Protecting the sensitive information about people we test with is very important along every step of the testing process. Some things to keep in mind are:
Keeping the information of user private when sharing the UX research insights with the team
Many bitcoiners care about privacy a lot, they may not be willing to show their faces/name/voice or any sensitive data (age, country of origin, etc). They can probably be treated as a separate persona and have very special needs. That means we may get less data about the users, if we want to respect their anonymity.
Ask users for permission prior to the call if they are ok with recording the session
Letting users know prior to a testing session that you will not be sharing any of the recordings with the public and that they will always remain anonymous.
A good practice is sending out to participants a research participation agreement and getting their consent is some written form.
Tools
Below are a list of tools you’ll need to start conducting usability tests:
Google Forms, Formbricks - to create pre-screening questions
Zoom, OBS Studio - to record the whole session
Calendly - schedule test sessions with participants
Moderated usability testing sprint
An example of usability testing sprint schedule. It may not perfectly fit your research situation — be flexible and adjust it according to your needs and possibilities.
Step 1: Scope, Define & Prototype
Define research goals, identified problems, and assumptions we have.
Define target participants and create a screening survey.
Decide on tasks users will be doing during the test.
Start prototyping.
Step 2: Write a Script & Recruit
a) Start recruiting participants with the screening survey, you can also choose to ask 1-3 questions directly via dm.
We are aiming at testing between 4-6 users. That number usually is sufficient to identify about 80% of problems.
b) Prepare a usability test script
Conducting each moderated usability test session should be done following a beforehand prepared script. Questions should be asked in the same order but can be followed with deepening questions if the interviewer feels it will be beneficial for understanding their experience or the problem.
Each session should consist of an introduction, opening questions, design-focused questions, closing questions, and outro. Here’s a script template we can use as a basis for each new script.
c) Prepare an outreach message
Hi, XX here from (name of project), we’ve just finished building V1 of the XXX wallet and we really wanted to get some feedback from the community. I was wondering if you would be open to helping us test it out?
Where can we find participants?
Social media: Nostr or Twitter
Bitcoin/lightning Discord channels
While recruiting, it’s important to pay attention to the audience we are connecting with and reach out to them in an appropriate way. It’s a good idea to get a feel of the community first and see which channel in the community would be best to post in.
Options when establishing contact:
You can ask people to reach out to you first and then after they reach out send over a link to a screener survey.
You can ask the screener questions just via messaging
Once the person is screened send over a Calendly link or suggest dates/times based on their time zone.
Step 3: Pilot Test
Pilot test the whole procedure with 1 or 2 persons from your company or project group..
Make changes to the prototype, Maze, or script based on discovered issues.
Step 4-6: Conduct the Studies
Remind participants about pre-arranged sessions (eg. 1 day before).
Conduct sessions.
Whole session shouldn’t take more than 1 hour.
Each session should be recorded in audio and video format, which can be later used to analyze and make notes.
It is desirable to record participants' faces as well, but this is not critical. We always need to ask for their permission to record and inform participants what we will do with the collected data.
Step 7: Interpret & Recommend
Create transcripts out of each recording
Highlight sections in the transcript that are insightful
Organize observations into affinity map or whiteboarding to identify common themes. Affinity mapping is a method to sort through user data.
Use insights to prepare design recommendations
Step 8: Post Sprint
Redesign according to recommendations
Compensate participants/winner of between 20 - 50 USD (or sats equivalent) to addresses they provided.
Thank participants and share research findings with them
Optional:
Add the research findings to a repository. (link out to repository)
After the redesign has been done the team can then conduct another round of usability testing to see how the new design performs in comparison.