Santa Barbara User Experience Testing12:32 PM
Local Loans User Experience Testing
When designing an onboarding flow for Local Loan’s borrower app, our goal was to increase the number of users who input their invite code and connected with a loan officer. Our hypothesis was that once the borrower has input their invite code both the borrower & loan officer would receive more value, and thus have a higher likelihood of converting the loan officer into a paid customer. Here’s what the borrower home screen looked like at the time:
The borrower would have to tap on "Invite Code" & then input their invite code they received from the loan officer. The concept was sound, but the user experience still needed a bit of polish.
The user onboarding flow was simple with no tutorial, but required the user to know to tap on "Invite Code" even though the green "Get Pre-Qualified" button was attracting all of their attention.
Recruiting the right people
In order to get the right results from the user test we needed to recruit the right users. For our Local Loans user onboarding designs, we wanted to get feedback from 10 people. To recruit them for the study, we sent emails like this one to unengaged trial users.
My name is Derren and I am on the research team at Local Loans. I see you had signed up for a trial of Local Loans recently.
How’s everything going?
My team is interested in improving the Local Loans experience. Would you be open to touching base on a quick call to discuss why you signed up for Local Loans and your experience with invite codes thus far? If you’re available this Thursday (4/21), Friday (4/22), Monday (4/25), or Tuesday (4/26), I’d love to schedule some time to talk.
I just want to clarify that this is not a sales call. You will be speaking with myself and a Local Loans product manager.
The goal of the call is to learn more about your experience so far with Local Loans, why you started a trial, and to show you a new feature we’re working on to hear your feedback on it. This is not a sales call – it is purely for research, but we’re happy to answer any questions you have about Local Loans along the way. For taking my call, I will be happy to offer you a $25 Amazon gift card.
The call will take around 30 minutes and will be at a time of your choosing within the next week. I look forward to hearing from you.
Usability Testing & Research
I focus on a few key points during these e-mails:
- I make it very clear that it is not a sales call. This will improve the odds of people committing to the call.
- Include an incentive to improve the odds of people committing to the call.
- Select users that have signed up, but did not complete the task you wanted them to. This will improve quality of results from potential users. In this case we were looking for loan officers that have not sent out invite codes to borrowers or realtors yet.
Generally response rates to these types of emails are pretty high (around 50%), so I will send out no more than 10 at a time until I have the desired number of tests scheduled. If people do not respond to the first email I make sure to follow up a second time before dropping them.
At Local Loans, we schedule these calls virtually and use Appear.in to run them. When possible we will schedule meetings in person & videotape sessions.
Creating the User Testing Process
I follow the same process for each user test I perform. This helps ensure that I can compare my observations apples to apples, and draw logical conclusions without bias from my findings. There are three primary parts to my process.
- Background questions
- Task-based walkthrough
- Wrap up
First I ask the basic questions to get to know the user, put them at ease, and discover relevant information about their background with the product thus far. Here are some background questions we used in the Local Loans user test:
How did you find out about Local Loans?
What do you know about us?
How would you explain Local Loans to a friend?
How do you keep track of referrals & leads?
How do you find potential leads or clients?
Why did you decide to start a trial?
What expectations did you have about what Local Loans is? Did the trial match that?
What have you done with Local Loans so far?
What value have you found in Local Loans?
Why haven’t you used the invite codes to lock in potential leads?
I tend to stray away from yes or no questions to get the person to open up about their experience.
Next, we move to the task-based portion of the interview. Before the user takes any action, we set the context. In our Local Loans test, we told one user to “Imagine that you’ve been doing research on the Local Loans app and decided to start a free trial. You know that you’ve been looking for a way to lock in new business. Start the trial and walk me through what you are doing.”
We knew this user was looking for a way to lock in leads through background questions and our recruiting processes, and that’s what we use to make the tasks contextual. We could have framed the task for another user by saying, “Your boss said you need to sign up for Local Loans and decided whether it’s the best lender solution for your company. Start the trial and explain what you are looking for. How do you decide that this is the best solution?”
During the task based portion of the interview, I ensure that we:
- Record the session so I can share the observation with my colleagues later on.
- Encourage the user to think out loud. I want to hear everything that goes through the user’s head when completing the tasks.
- Tell the user that I am testing my designs & not them. This helps put them at ease by letting them know they cannot do anything wrong.
- Get every user to go through the same tasks without my help. This allows me to aggregate my findings and agree on design changes with my team.
At Local Loans for this test we gave users a beta product to test with. However, I have used staging versions of software, hand drawn mockups, or clickable prototypes built in Invision as well.
With each new screen or step in the task, I allow the user to think out loud and make a note of his or her initial impressions. If I feel that there is more to learn then I will ask the user follow up questions to gain further insight.
Once the user has completed all the tasks I’ll wrap up the interview. Letting them know that they have reached the end of the tasks, their feedback was helpful and insightful, before finally thanking them for their time and saying goodbye. Shortly thereafter, I follow up with a thank you email and the promised compensation.
After each interview, I’ll take 10 minutes to go over everything I just heard with my team. This ensures that everyone who was watching the interviews is on the same page and leaves a little time for discussion and synthesis. This 10 minutes potentially saves hours of debating and going over video later. It is vitally important to digest the most important takeaways, document them, and decide on the next step together to keep the process moving forward smoothly.
Turning Observations into Action
After all the interviews have been completed its time to aggregate and see how many users completed the tasks. This could be as simple as:
3/5 users skipped reading user onboarding marketing materials
4/5 users filled out the account information correctly
3/5 users clicked on the "get started" button first
All users completed the entire flow
Sharing questions or comments that came up during testing tend to be helpful as well. For example, I took note of things like:
- “What if I want to service multiple areas?” - This user wanted the ability to represent multiple localities to be able to receive more leads. This indicated an additional potential revenue
- "The underwriting guidelines allows me to completely do my job on the go. This is great!” – This indicated that the feature set was correctly chosen for the product.
Conducting these customer interviews, there were three themes that emerged:
- Loan Officers were sending invite codes out, but Borrowers were confused about how to input the invite codes properly.
- Loan Officers wanted to share the invite code with realtors as that is their primary means of referrals.
- Add tooltips to help explain what each field is during the sign up process to reduce confusion.
From these observations, we made the following changes to the product:
- We added 5 marketing slides to the user onboarding flow at the start of the app.
- We added the ability for users to share their invite code with realtors & track the amount of referrals they receive from realtors.
- We extended the onboarding flow to include tooltips and additional step by step walkthrough once the user was inside of the dashboard to walk the user through the most valued features of the app.
After making these changes, we shipped a variation of the below screens and saw a 400% lift in the number of invite code use after one week. All because we solicited feedback on a design before building it out.