The Benefits Of Cognitive Inclusion In UX Research

The Benefits Of Cognitive Inclusion In UX Research In the summer of 2024, I became co-chair of a working group of expert researchers who came together to determine how best to perform accessibility testing with people with cognitive disabilities. This was work I did for Fable, where I am currently VP of Innovation. Cognitive disability is an umbrella term for several disabilities that impact how people process information, and it usually affects memory, focus, and/or learning. It is the most prevalent disability in the U.S. (13.9% via CDC), and cognitive disability is increasing rapidly (Yale study). We set four goals for ourselves to learn how to work with this audience: - How should we recruit and screen participants? - What are best practices for research with cognitive participants? - Do these methods work in a real study? - Documenting what we learned so that we could share it. We created a screener to recruit people who self-identified as having challenges with memory, focus, and learning. We also reviewed published studies that involved cognitive testers to learn best practices for working with them. Next, we tested these best practices with an initial group of 25 testers in a pilot study. We fine-tuned our approach iteratively and created a guide to running user interviews with cognitive testers and a survey that could quantify their experiences using digital products. Finally, we documented what we learned. After our pilot study with this new group of testers finished, I felt that they would uncover more usability insights than the general population (gen pop) user research participants I’d worked with in the past. I set out to validate this hunch. The Cognitive Usability Study I decided to run a joint study with Fable’s partners at the University of California, Irvine, in collaboration with Syed Fatiul Huq and with help from Fable researchers Pranav Pidathala, Ali Brown, and Michael Fagan to see if my hypothesis about finding more insights with cognitive testers proved true or not. I generated three websites for the study using an AI prototyping tool. I wanted three different types of sites with different user goals and content so I could test a variety of tasks in the study. Table 1: Websites And Tasks Tested | Website | Strong Snacks | Turning Pages | Crown & Comb | |---|---|---|---| | Description | This is a website for three-ingredient high-protein recipes. Recipes can be browsed by category (vegan, muscle building, etc.). The site also features blog posts about protein and contact information. | This website is for a bookstore with a catalog of curated reads. It features extensive filtering by book genre, a book swiping feature to build a profile of likes and dislikes, custom book lists, a shopping cart, and checkout. | A website for a hair salon that allows you to book appointments and consultations online. It has a VIP program and a variety of special packages visitors can buy. | | Design | Simple, brutalist, bright, lots of pictures. | Moody, classic, dark, lots of pictures of book covers. | Bold, clean, black and white with bursts of color. | | Content | Recipes, blog posts. | Books and book lists. | Services, experience guide, membership information. | | Key functionality | Filter by category, newsletter subscription. | Shopping cart, book matching, book lists, recommendations. | Appointment booking. | | Tasks | | | | We used a single screener with questions about memory, focus, and learning, and screened participants into two groups based on whether they self-identified as having cognitive challenges or not. Cognitive disability includes neurodiversity. Neurodivergent is an umbrella term used to describe people whose brains process information and learn differently. It is most commonly used for people who have learning disabilities (e.g., Dyslexia), ADHD, and Autism. We ran 30 user interviews, 10 per website, with an even 5⁄5 split between cognitive and gen pop participants for each website. In each session, a participant completed all the tasks for one website during an online user interview facilitated by one of the researchers involved in the study. All participants completed an Accessible Usability Scale (AUS) survey at the end of their session. This is a free, Creative Commons-licensed 10-question survey to evaluate the usability of websites and mobile apps. Data Analysis Approach I reviewed all the study recordings and transcripts and made note of every time a participant raised a concern, question, difficulty, or asked a question about how something worked. I counted all of these as issues. I also noted where a participant missed something that was part of a task, even if they didn’t notice it themselves. I also noted every suggestion for improvement made by participants. Examples of issues found included: - Photo is too tall and requires a lot of scrolling to get to content (noted by participant). - I get no feedback when I like or dislike a book (noted by participant). - Participant missed the required P.O. Box checkbox the first time (observed by me). Examples of suggestions included: - I would like to see a protein comparison in a table. - The “More information” tab should be moved up higher. - I would like more information on how the recommendation list is created. Issues and suggestions were counted once per participant, even if they mentioned the same thing twice, but there are, of course, repeat issues and suggestions across the different participants. It is expected in UX research with multiple participants that you’ll find similar issues with each participant, and that is a signal that an issue is a universal challenge. Findings Of The Cognitive Usability Study Across the three websites tested: - Cognitive participants identified 197 issues. - Gen pop participants identified 113 issues. - Cognitive participants made 93 suggestions. - Gen pop participants made 54 suggestions. - Cognitive participants surfaced more issues related to content, buttons, icons, visual elements, and media than gen pop participants. The results aligned with my instincts: participants with cognitive disabilities identified 1.8 times more issues and made 1.8 times more suggestions than gen pop participants. Let’s dive deeper into the data for each website. Note that an AUS score ranges from 0 to 100, with higher numbers representing better usability than lower numbers. Table 2: Strong Snacks This site had the simplest design and content of all websites tested in the study and accordingly had the lowest overall issues and the highest median AUS scores. The data aligns with what you’d expect from an easy-to-use and simple website. On this website, cognitive participants found 3.4 more issues and made 2.2 more suggestions on average. Their average score of the overall experience was 13.7 points lower than that of the gen pop participants. | Total issues | Average issues | Median issues | Total suggestions | Average suggestions | Median suggestions | Average AUS | Median AUS | | |---|---|---|---|---|---|---|---|---| | Gen pop | 32 | 6.4 | 6 | 13 | 2.6 | 2 | 90.5 | 97.5 | | Cognitive | 49 | 9.8 | 9 | 24 | 4.8 | 4 | 76.8 | 73.0 | Table 3: Turning Pages This was the website with the most varied functionality and the most tasks to complete (4), so it’s not surprising that participants found the most issues. Here, cognitive participants found 6 more issues and made 3.2 more suggestions on average. They also scored the overall experience 17.2 points lower than gen pop participants on average. | Total issues | Average issues | Median issues | Total suggestions | Average suggestions | Median suggestions | Average AUS | Median AUS | | |---|---|---|---|---|---|---|---|---| | Gen pop | 55 | 11 | 10 | 26 | 5.2 | 4 | 78.0 | 80.0 | | Cognitive | 86 | 17 | 15 | 42 | 8.4 | 6 | 60.8 | 58.0 | Table 4: Crown & Comb This website was intentionally designed to be complex, and task 3, finding the bridal package, was meant to be extremely difficult to complete. On this last website, cognitive participants on average found 7 more issues and made 2.4 more suggestions. Their average score for the overall experience was 14.3 points higher than the gen pop participants. | Total issues | Average issues | Median issues | Total suggestions | Average suggestions | Median suggestions | Average AUS | Median AUS | | |---|---|---|---|---|---|---|---|---| | Gen pop | 26 | 5 | 4 | 15 | 3 | 3 | 49.5 | 35.0 | | Cognitive | 62 | 12 | 11 | 27 | 5.4 | 2 | 63.8 | 68.0 | Something interesting happened with the AUS scores for cognitive and gen pop participants in Tables 3 and 4. Cognitive participants scored Crown & Comb higher than Turning Pages, but gen pop scored the opposite — higher for Turning Pages and lower for Crown & Comb. If I had to guess why, I suspect finding more issues on Turning Pages impacted the cognitive participants’ perceptions of usability more than the gen pop participants’. The other major difference between the sites, outlined in Table 5 below, was that cognitive participants found many more issues with buttons and links on Turning Pages and more issues with icons and visual elements on Crown & Comb. This suggests to me that the interactions being challenging on Turning Pages were a more significant challenge than issues with visual elements. Qualitative Findings When it comes to the more qualitative findings, I looked at trends in the types of issues found by both groups of participants. Cognitive participants: - Were more likely to flag issues with icons or visual elements. - Surfaced problems with content more frequently. - Gave richer qualitative commentary, often explaining why something was hard to find or confusing. Gen pop participants: - Were less likely to flag conceptual or comprehension barriers. - Gave shorter feedback, often stopping once the task was complete. Table 5: Number Of Issues By Category When I grouped issues by category, the following issues surfaced more often with cognitive participants: content