research goals

To improve our designs, we evaluated our prototypes for usability

Our primary goal was to improve our designs based on experiences real applicants may experience with our tools.

Another one of our goals was to uncover user pain points and expectations based on their mental models with voice and assistant experiences, like Alexa and Messenger bot.

Evaluating usability of our two designs - for Alexa and Messenger - was key to our testing.

methods and tools

We navigated virtual recruitement and testing to get our designs infront of potential applicants

In short, our study focused on 1x1 interviews designed for remote usability testing.


In our toolbox, we used Zoom to facilitate our research sessions and had an Alexa Echo device 'on call' for participants to interact with. We also tested our Messenger bot which participants accessed and screenshared from their own devices.

Our recruitment through Facebook Ads reflected our target audience and was focused on students, single parents,and seniors based in Pennsylvania who are currently receiving SNAP benefits, never applied for SNAP previsouly, or are awaiting SNAP decisions.

Metrics

To measure the success of our designs, we incorporated metrics which focused on qualitative and quantitative aspects of usability

Task-level: time on task, success rate, error rate

Test-level: UMUX-LITE, NPS, Satisfaction

Bugs: areas for immediate fixes

Aggregating Feedback

Insights

Linear

Input should feel natural, regardless of channel

Literal

Provide clear questions and options

Contextual

Reword, elaborate, and clarify jargon

Digestible

Present information in chunks to prevent information overload

Clear

Questions are straightforward, offer instructions like "for example, say 1983"

Transparent

Provide details on how estimate is calculated.