UXR4-Usability

Moderate Usability Testing

Collect User Feedback and Identify Usability Issues and Challenges Before Alpha Release

📆 5 weeks | 🧩 Usability Testing | 📊 Sample Size: 9 | 🧰 Figma, Miro

The in-store product team plans to migrate a legacy merchant and customer-facing product (touchscreen device) to an updated web codebase. This update will unlock new features that can generate more business opportunities.

Challenge 

The migration will significantly alter current features and user flows. The team is concerned that these changes could result in dissatisfied users and potential churn.

How might we execute a seamless migration with minimal disruption and deliver an improved user experience that caters to the users’ needs?

 

Solution

We outlined three research projects to achieve this goal. This page focuses on the 2nd project, we leverage both unmoderated and moderated usability testing to gather user insights on the new prototype to identify potential issues before the MVP launch.

Methodology

Previously, we conducted initial discovery research with internal users/merchants to identify challenges and features priority for the product migration. Our designer then created Figma prototypes for usability testing based on the research.

  • Before finalizing the prototypes, I guided the designer through unmoderated tests on a couple of prototypes. Working closely with the designer, we incorporated feedback from 2 unmoderated tests (n=10) conducted on usertesting.com into the final prototypes.

  • We then conducted moderated usability testing to gather live user feedback and ask follow-up questions—an aspect that unmoderated usability testing cannot facilitate.


Objective

  • Collect input from existing user regarding the new prototype to evaluate their responsiveness and uncover any potential obstacles to adoption.

  • Identify and address any concerns or issues before finalizing the product for release.

     

Methodology

  • unmoderated usability testing via usertesting.com

  • 30-minutes moderated usability testing via GoogleMeet

Prototype Testing

Simulating Real-World Interaction

In order to simulate real-world experiences in a remote testing environment, I created scenarios for participants to role-play as coffee shop owners.

Dual Screens

We included user flow screens for both customers and merchants, enabling participating merchants to see what is happening on the customer’s screen during interactions. The comprehensive context allowed participants to provide holistic feedback on the prototype.

p3_2

Participant Instruction

Remote observation of user interactions during usability testing can present challenges. To address this, I provided participants with detailed instructions to maximize the effectiveness of the tests. Participants were

  • instructed to share both their prototype screen and webcam, allowing us to observe their on-screen actions and facial expressions

  • prompted to verbalize their thoughts, expectations, and feedback as part of a “think-aloud practice,” which helped us gain insight into their decision-making process

Participants

Through collaboration with our internal product marketing team, we have recruited a diverse group of 9 internal users/merchants from small businesses to participate in our remote usability testing. The participants included:

  • Coffee & Tea (n=2)

  • Quick Casual Restaurants (n=1)

  • Retail (n=5)

  • Fitness Studio (n=1)

Setups

We divided participants into two groups for a comprehensive evaluation of the prototype.

 

Group A : Two Prototype Links, Thorough Guidance (n=5)

For this group, I wanted to gain a detailed understanding of the usability for each flow, including any minor issues or confusions.

  • Participants received 2 separate prototype links and more detailed instructions to complete each task.

  • Follow-up questions were then asked after completing each flow

Group B: Combined Prototype Link, Minimal guidance (n=4)

For this group, I wanted to gather feedback on the overall experience and added complexity to evaluate how well participants could complete the tasks with minimal guidance.

  • Participants were given 1 prototype link that combined two user flows and were encouraged to explore the prototype at their own pace.

  • Follow-up questions were then asked after the entire user testing.

Prototypes​

Prototype A

  • Evaluate the merchants’ responses to the updated/new customer sign-in flow.

  • Assess the usability of the revamped reward redemption flow.

  • Explore solutions employed by merchants when faced with challenges such as unavailable rewards.

Prototype B

  • Assess the usability of checking in a new customer and setting up new customer profile

  • Explore participants’ thought process of adding notes and saving customer preferences on the new prototype.

Click ➕ to view examples of scripts and questions

“The customer orders a latte with oat milk. How would you save this information in the customer profile for future use? Describe what you see on the screen and before you take any actions, please think out loud when you click anywhere. “

[open expanded profile or tap on note field, brings up keypad, auto-populate entry, tap ‘done’, confirmation notification]

“The customer wants a free muffin with their purchase, but unfortunately you ran out of muffins earlier today. However, your customer does have another reward available, a ‘Bagel Special’. Walk me through how you would complete this transaction and remember to  share your thought process out loud.”

 

[confirm reward selection override – do they know how to reject and select a different reward? Do they give a verbal answer?]

[if they don’t know what to do next, we want to prompt them] Take the actions you think will help you satisfy your customer.]

When you think about the overall flow from start to end,  how easy was it for you to use the upgraded product? Please rate it on a scale of 1 to 5 (with 5 being very easy).

Can you explain why you gave that score? If there were areas for improvement, please mention them.

Did you encounter any moments where you felt stuck or confused during the process?

Analysis

I introduced collaborative analysis sessions with stakeholders to expedite the analysis process and engage them in discussions. 

  • Following each usability testing interview, a 15-minute debrief session was scheduled with participating stakeholders. 

  • During these sessions, I provided prompts and guiding questions on the Miro board to facilitate discussions and help them recall the initial impressions, key insights, and memorable quotes from the interview. 

This approach saved my research time on the interview transcript analysis and promoted alignment with stakeholders.

Insight

Overall, we received positive feedback from participating merchants regarding both prototypes, with minor usability concerns.

  • Merchants find the new prototypes similar to their existing flow, and even more user-friendly. They anticipate a smooth transition when the prototype replaces their current user experience after product migration.

  • Although they identified some areas for improvement in the user flow, it did not affect their ability to complete tasks during testing.

Impact

Based on the insights from multiple rounds of usability testing and prior discovery research, our product manager prioritized features and finalized the user flow design for building an MVP for the alpha launch.

  • The feedback received from the prototype boosted our confidence in replacing ​​the current user flow when migrating the product.

  • Furthermore, our designer utilized the feedback and minor issues identified in usability tests to explore and improve ​the user experience in ​​the next iteration.

Reflection

Throughout this project, I learned to balance agility and research quality while engaging stakeholders effectively. This accelerated the research timeline and taught me to rely on the expertise of product managers and designers for quick understanding.​

 

🤝 Collaborative Working Model Enhances Research Effectiveness

Through previous collaboration with the product manager 👨‍💻 and designer 🎨, we quickly established trust. We also applied the same collaborative working model to improve the speed ⏩ and effectiveness of the research. This experience has prepared both me and the stakeholders for the upcoming research project 🚀 on this product line.

 
👥 Building Research Capacity & Engaging Stakeholders 

I enabled the product manager and designer to take on more responsibility in the research process, while I took a step back to provide guidance and best practices. Throughout the study, I engaged stakeholders to ensure that the findings 📊 were applicable and relevant for end users. I conducted debriefing sessions 📝 to facilitate reflection 🤔 and sense-making of data, allowing the research team to discuss and analyze 📈 emerging themes and patterns.

 
🎯 More Targeted Merchant Recruitment

An important UX flow we wanted to test was the new customer profile for note-taking 📝. However, the majority of our participants 👥 were not accustomed to taking notes during the checkout process.  It would have been more valuable to involve merchants who habitually take notes on our device, to explore how to optimize the UX design for note-taking.