UX Research: Diary Study on users' interactions with AI assistants
Role: UI/UX Designer
Tools: Google docs, Google Forms
Timeframe: 3 weeks
Overview
AI-powered personal assistants such as Siri, Alexa, and Google Assistant are becoming increasingly embedded in our daily lives. While they offer convenience, their effectiveness varies based on user needs and expectations. This study explores real-world interactions with AI assistants through a diary study and follow-up interviews to uncover user experiences, frustrations, and suggestions for improvement.
Research Objective
Question: “How do users engage with AI-powered personal assistants, and how do these interactions shape their experiences?”
To answer this, participants documented their AI assistant usage over three days, responding to these prompts:
Which AI assistant(s) did you use today?
What tasks did you perform with the assistant?
How did the AI assistant’s performance align with your expectations?
Would you rely on this AI assistant for the same task in the future?
Following the diary study, we conducted semi-structured interviews to delve deeper into users’ experiences and concerns
Methods
Participant Selection & Study Design
We recruited individuals who frequently use AI assistants to participate in a three-day diary study. Each day, they submitted responses via a digital form to capture real-time experiences. The diary entries helped document recurring themes and variances in usage.
Following the diary study, we conducted one-on-one interviews with participants to:
Expand on diary responses
Identify common pain points and areas of satisfaction
Explore user concerns related to privacy and reliability
This mixed-method approach allowed for a holistic understanding of how AI assistants fit into daily routines.
Interview Responses
Key findings and insights
Most participants found AI assistants helpful for simple, routine tasks like setting reminders, playing music, and retrieving basic information. However, they frequently encountered issues with misinterpretations and inconsistent performance.
2. Privacy Concerns and Data Security Hesitations
A major theme that emerged was uncertainty about data privacy. Many participants expressed concerns over what AI assistants track and store, influencing how much personal information they were willing to share.
3. AI Assistants Are Primarily Used for Low-Stakes Tasks
Participants overwhelmingly used AI assistants for routine, non-critical tasks rather than complex decision-making or high-stakes needs. Their reluctance to rely on AI for anything urgent stemmed from inconsistent reliability.
4. User Recommendations for Improvement
Participants provided several suggestions for making AI assistants more reliable and effective:
Better Context Awareness: AI should remember previous interactions and respond accordingly rather than treating each request in isolation.
More Accurate Responses: Reducing vague or repetitive answers would improve trust and usability.
Enhanced Privacy Transparency: Users want clearer explanations of how their data is stored and used.
Improved Integration with Other Tools: AI assistants should work more seamlessly with third-party apps and smart home devices.
Study Strengths
The diary study captured real-time, unfiltered experiences with AI assistants.
Follow-up interviews provided deeper context and clarification on pain points.
The study highlighted both positive and negative aspects of AI assistant usage.
Study Limitations
Some diary responses were brief and lacked detail, requiring additional probing during interviews.
Participants occasionally forgot to log interactions in real-time, leading to potential recall bias.
The small sample size limits generalizability to broader populations.
Future Considerations
To improve research depth and credibility, future studies should:
Implement structured reminders to encourage real-time diary logging.
Expand the participant pool to capture diverse usage patterns.
Extend the study duration to analyze evolving AI interactions over time.
Conclusion
Our research reveals that while AI assistants offer undeniable convenience, users remain wary of their reliability and data privacy implications. The combination of diary studies and interviews provided a rich, nuanced understanding of user behaviors, frustrations, and needs. Future improvements in AI technology should focus on increasing contextual awareness, accuracy, and transparency to build greater user trust and adoption.
Acknowledgments: This project was a collaborative effort as part of my INFO 690 UX Research Methods class at Drexel University. I had the privilege of working with an amazing team: Marian Gasinu, Katherine Cassandra Stolaki, Katherine Wilhelm. Their insights, skills, and dedication were instrumental in making this research successful.