top of page
TL;DR
Improved accesSOS' UX Operations
a. Secured an annual budget of $1000+ for UX Research.
b. Created their first standard operating procedure for planning tests with Communities with Disabilities (CWDs).
c. Improved the organization's research culture.
Project Background
About The Company
-
accesSOS is a 501(c)(3) non-profit based in San Francisco and they made a mobile app that makes contacting 911 more accessible especially for the Deaf and Hard of Hearing.
The Beginning
You know, I think it'd be okay if for the time you're looking at this I tell you about how I supported the complete redesign of the app, reduced the clicks needed to complete the main user flow, lowered the mean user time on task from 78 seconds to 53 seconds, and caught two critical bugs affecting Android v12 and v14 (all true btw).
But what I think is important is what I did to improve the culture of research at this organization that gave rise to improvements to the cool OKRs UXers love to talk about.
So what's the scenario?
The accesSOS native app has received a new update, and the Founder of accesSOS did not want to put additional marketing money and effort towards this new version of the app until she felt confident that the app was usable and all critical bugs were caught and fixed.
Below is how I designed a study to meet the needs of the business, the users, and chose the right method.
Should find the output detailed enough to respond to an emergency.
Accessibility Considerations
1. An ASL interpreter is expensive so the interview guide needs to be an economic length.
2. Participants who are Deaf cannot think out loud while using the app so the interview guide must reflect this.
3. Testing with communities with disabilities need 25% more time allotted for testing sessions for set up and tear down.
Constraints
Research Design
Research Method
A moderated, task-based usability test will guide users through the app's four major flows, helping us identify pain points within each flow and giving us the opportunity to explore the "why" behind them.
Research Question
How does the order of the information presented and the expression of that information (i.e., copy and iconography) impact how quickly and efficiently a user can report an emergency in the app.
Operationalizing Key Variables
Usability will be measured by time on task because the time it takes to contact emergency services is crucial for saving users' lives when reporting emergencies in the app.
1. accesSOS has never done UX Research with communities with disabilities so there are no internal processes to guide the study.
2. No incentives are being provided to participants which reduces our ability to quickly get a random representative sample.
3. There is no recorded time on task data from previous prototypes or to compare our results against.
Problem Statement
accesSOS has a beta app for iOS and Android that has yet to be tested with their key audiences and if this is not done the app may be ineffective for contacting emergency services, putting the userbase in danger.
Key Testing Audiences
Primary
Secondary
Tertiary
Deaf & Hard of Hearing
First Responders
Non Visible Disabilites
Should find the app easy to use and efficient.
Should view this as an easy alternative to call 911.
PAUSE!!! DID YOU CATCH THAT!??
Despite being their primary audience, accesSOS has never done UX Research with the Deaf & Hard of Hearing!
I discovered this through an analysis of their research repo spanning back 4 years and after having chats with the Founder, COO, and PM.
Generative Research Opportunity
Keep in mind that this wasn't my first rodeo (in fact it was 3rd with this company).
In all my studies here, something kept coming up... even if users felt the app was usable, they commonly said they wouldn't use it compared to other alternatives.
This tips me off that there may be a value risk with the app. So in the beginning and end of my usability test, I asked users...
1. What technologies do you use to keep yourself safe on a day-to-day basis?
2. After using this app, how do you see this fitting into your daily safety strategy?
This gave me the chance to gain generative user insights that may impact accesSOS' strategy.
I was curious to know if the app was in any way useful to them and I wanted to know what other kinds of technology did they consider useful.
This will help me understand where accesSOS could provide more value.
Getting Stakeholder Buy-In
"Kal, Why should we do this? We've been doing fine as it is?"
Now this isn't a direct quote from anyone from this lovely organization but it's the source of many of the questions I received leading up to the execution of this study and I think it's a fair one. Here's how I fielded stakeholder questions to get buy-in to test with the Deaf & Hard of Hearing...
The COO
'An ASL Interpreter is expensive, can't we use a free online translation tool to reduce the cost of this study?'
Kal
While there may be free tools that exist to help with the language barrier between people with nonvisible disabilities and the Deaf and Hard of Hearing, the community consensus is that they are still too far behind in closing that gap and this will significantly impact the data we capture. We also want to make this experience human for the participants and show them we care about the community and are willing to pay professionals working in the Deaf & Hard of Hearing space for their services. I have also made sure I piloted our scripts to ensure we are only asking the questions we need to ask.
The PM
What if this doesn't go well? What if it's a disaster and we spent all this money for nothing?
Kal
I hear you and that's a very real and stressful thought to have in mind as we inch closer to executing this study. I think that this can ONLY be a disaster if we don't learn from this. This is going to be challenging no matter what because we've never done this, but I promise you, I will have an after-action report that details what we can improve next time and this is data we will never have if we don't at least attempt the study. I'll also have several witnesses present from Design and Eng to attest that the account of what happened is truthful and accurate.
Kal
I understand that including CWDs in our research is challenging and time-intensive, especially with the urgency to market this redesign. However, it’s crucial to ensure we’re meeting the needs of our primary audience before marketing widely. This effort will not only help us get it right but also build institutional knowledge, reducing costs and time for similar studies in the future. Additionally, it’s an opportunity to foster empathy across teams by allowing management, design, and engineering to truly understand who they are building for.
The Founder
'What's your reason for doing this now and not at a later date? Is this something at has to be done right now?'
Executing The Test
Funny Story
The community who agreed to be a part of our study asked if I'd like to see a film that they had put together for the Deaf featuring actors from their community airing the day before our tests.
Who am I to say no? My partner and I went to the movie showing to build rapport, and without an interpreter, we got a feeling for what it's like to be in their world.
I was so personally moved by the experience I immediately went home and revised my script to be even more specific based on the experiences I had communicating with them.
Analysis & Findings
Prioritizing Pain Points
While there were several pain points discovered, we had to prioritize which one deserved the most attention. To do this I made a user journey map that showed what parts of the user's journey took the most time and caused the most pain.
CRITICAL FINDING: The Deaf and Hard of Hearing Reject The App
Mine and the PM's analysis revealed a handful of things that affected our user's time on task I'd love to talk about if you invite me for an interview (please!), however the most SHOCKING finding was as the subtitle suggests, the Deaf and Hard of Hearing see very little value in this app.
Why?
Well going back to my questions about the technologies that people use to keep themselves safe, it turns out this community has many free technologies paid for by the government that make contacting emergency services free, and in some participant's accounts, easier to use.
We found that contacting emergency services was not the most challenging part of having an emergency while Deaf, it's actually interacting with the police once they arrive on scene. So much so that participants shared they've called a family member to contact emergency services for them to avoid misunderstandings with the police altogether.
This rightfully came as a shock to the organization.
accesSOS primarily gathers insights from university student cohorts partnered through their courses.
Due to students' time constraints and the lack of recruitment support or participant incentives, these cohorts often recruit peers from their inner circles—primarily other college students.
However, the Deaf and Hard of Hearing community typically skews older, and college students without visible disabilities differ significantly from accesSOS's key demographic. Years of testing with this group have resulted in an interface that disproportionately reflects their needs rather than those of the target audience.
How did we end up here?
I constructed an affinity diagram with the Product Manager to discover emerging pain points and themes.
Method Of Analysis
Solution
Having A Standard Operating Procedure for Testing with CWDs
I made a standard operating procedure on the steps and resources that should be used to help accesSOS stay on top of research with CWDs.
1. $etting Aside Resources
Set aside a budget greater than or equal to 5 hours of ASL interpretation to cover a hypothetical set of 5, 1 hour long, usability sessions with 5 participants who are Deaf or Hard of Hearing that would occur once a year. This is around $1,000+. This recommendation recognizes that accesSOS is a resource constrained organization.
2. Break Glass in Case of Major Update
The Deaf & Hard of Hearing should be included at most every time there is a major update to the app that may affect their ability to contact emergency services.
3. Piloting & Quotes
Ensure scripts are piloted at least once within the company, preferably twice to gauge how long the test should take and then add 25% more time to the estimate (US Government recommendation) to know how much to time quote for in the event you need a certified ASL interpreter.
Lasting Impact
A Matured Research Culture
In a follow up with the PM it was confirmed that all of my recommendations remained in place and have incorporated testing with CWDs into their product roadmap!
bottom of page