Testing on bitcoin products
Doing usability testing on bitcoin products is unique in the sense that we are asking people to possibly share sensitive and personal information with us.
There are 3 main considerations when testing:
Testing applications requires some sensitivity since a person's personal wallet will contain too much personal information. We usually do tests on prototypes or on a test account. Below you’ll find some information about each approach:
Prototype: Testing a design concept on a prototype is also a good solution, clickable prototypes can be created with Figma or with other prototyping tools
Test account: Creating a login username and password for a bitcoin wallet which could then be provided to people when testing
It’s always nice to reward participants, a quick and easy way would be to send them over some sats. Fiat is also an option but if they have a bitcoin wallet then it’s a nice gesture to send over some stats after the test. Paying with bitcoin is somewhat easier than fiat as it's open and borderless protocol. We can test users around the world and do not pay high fees or wait a few days for wire transfers.
We can also do a draw to reward one participant, so for example let all participants know that there will be a draw after the sessions and one person will win 50 dollars (sats equivalent)
Know beforehand how the payment will be made: lightning/bitcoin address, standard lightning invoice, on-chain or some other way?
If you're paying with bitcoin, but compensation is priced in fiat, take into account exchange rate volatility (especially when sending on-chain)
Take into account fees, and make sure participant receives the full promised amount
Make sure participants know how to receive payment
Privacy & sensitive data
Protecting the sensitive information about people we test with is very important along every step of the testing process. Some things to keep in mind are:
Keeping the information of user private when sharing the UX research insights with the team
Many bitcoiners care about privacy a lot, they may not be willing to show their faces/name/voice or any sensitive data (age, country of origin, etc). They can probably be treated as a separate persona and have very special needs. That means we may get less data about the users, if we want to respect their anonymity.
Ask users for permission prior to the call if they are ok with recording the session
Letting users know prior to a testing session that you will not be sharing any of the recordings with the public and that they will always remain anonymous.
A good practice is sending out to participants a research participation agreement and getting their consent is some written form.
Below are a list of tools you’ll need to start conducting usability tests:
Figma, Penpot - to design and create prototypes
Google Forms, Formbricks - to create pre-screening questions
Zoom, OBS Studio - to record the whole session
Calendly - schedule test sessions with participants
Moderated usability testing sprint
An example of usability testing sprint schedule. It may not perfectly fit your research situation — be flexible and adjust it according to your needs and possibilities.
Step 1: Scope, Define & Prototype
Define research goals, identified problems, and assumptions we have.
Define target participants and create a screening survey.
Decide on tasks users will be doing during the test.
Step 2: Write a Script & Recruit
a) Start recruiting participants with the screening survey, you can also choose to ask 1-3 questions directly via dm.
We are aiming at testing between 4-6 users. That number usually is sufficient to identify about 80% of problems.
b) Prepare a usability test script
Conducting each moderated usability test session should be done following a beforehand prepared script. Questions should be asked in the same order but can be followed with deepening questions if the interviewer feels it will be beneficial for understanding their experience or the problem.
Each session should consist of an introduction, opening questions, design-focused questions, closing questions, and outro. Here’s a script template we can use as a basis for each new script.
Usability Test Script example
c) Prepare an outreach message
Hi, XX here from (name of project), we’ve just finished building V1 of the XXX wallet and we really wanted to get some feedback from the community. I was wondering if you would be open to helping us test it out?
Where can we find participants?
Post a bounty on stacker news
Social media: Nostr or Twitter
Bitcoin/lightning Discord channels
While recruiting, it’s important to pay attention to the audience we are connecting with and reach out to them in an appropriate way. It’s a good idea to get a feel of the community first and see which channel in the community would be best to post in.
Options when establishing contact:
You can ask people to reach out to you first and then after they reach out send over a link to a screener survey.
You can ask the screener questions just via messaging
Once the person is screened send over a Calendly link or suggest dates/times based on their time zone.
Step 3: Pilot Test
Pilot test the whole procedure with 1 or 2 persons from your company or project group..
Make changes to the prototype, Maze, or script based on discovered issues.
Step 4-6: Conduct the Studies
Remind participants about pre-arranged sessions (eg. 1 day before).
Whole session shouldn’t take more than 1 hour.
Each session should be recorded in audio and video format, which can be later used to analyze and make notes.
It is desirable to record participants' faces as well, but this is not critical. We always need to ask for their permission to record and inform participants what we will do with the collected data.
Step 7: Interpret & Recommend
Create transcripts out of each recording
Highlight sections in the transcript that are insightful
Organize observations into affinity map or whiteboarding to identify common themes. Affinity mapping is a method to sort through user data.
Use insights to prepare design recommendations
Step 8: Post Sprint
Redesign according to recommendations
Compensate participants/winner of between 20 - 50 USD (or sats equivalent) to addresses they provided.
Thank participants and share research findings with them
Add the research findings to a repository. (link out to repository)
After the redesign has been done the team can then conduct another round of usability testing to see how the new design performs in comparison.