AI-driven B2C Voice Interaction IOS App
My Role
Bridging AI Tech with UX @ Bay Area
Harmony is a startup project at UC Berkeley, built on a voice biomarker ML model that analyzes speech patterns to detect early cognitive decline. My role on the cross-discplinary team was to translate this technical foundation into a clear, trustworthy, and emotionally supportive user experience.
Define real-time voice data flow and consent checkpoints
Map model outputs into interpretable, human-centered feedback
Design privacy-first interaction and visualization patterns
Ensure smooth integration with iOS voice UI components
Type:
B2C, IOS App Design
Timeline:
Jan 2024 - Aug 2024
Responsbilities:
Define the problem and use cases;
Wireframe;
Build prototype for testing;
Iterate interaction;
Craft visual details.
Deliverables:
Interactive prototype;
High fidelity mocks;
Design files & doc;
Presentation.
Background
The voice amplifies wellness
Voice Biomarkers are measurable characteristics in a person's voice, such as tone, pitch, or speech patterns, that can indicate health conditions or changes, like cognitive decline or stress.
We trained AI models to proactively predict dementia through voice biomarkers, reached over 80% accuracy.
Research-driven
Design Progress
We followed a structured, user-first process to make the experience feel both functional and emotionally safe.
Prototyping & Testing
Iterated fast to refine voice feedback loops, report logic, and privacy mode for long-term engagement.
Visual & Character Design
Created a lovable agent to humanize AI and make voice interactions warm and approachable.
Wireframes & Flows
Designed clear flows — onboarding, consent, daily voice check-ins, report viewing — to lower barriers and build trust.
User Research
We conducted primary research and 80+ semi-structured interviews with adults aged 45–65, focused on turning voice biomarker screening into daily rituals.
Project Goals
Harmony empowers dementia relatives with AI-powered voice biomarkers to enable early detection and personalized, accessible, and trusted care.
Agile MVP design and prototyping
Because we were productizing a new technology, our strategy was to test and iterate fast. I quickly designed a functional MVP to validate assumptions, capture real user reactions, and build a solid foundation for further iterations.
Test and Iterate Fast
Validate features design and iterate
I tested the prototype with over 10 participants at a public pitch event. This group included caregivers, middle-aged adults, and medical stakeholders. Instead of just observing usability, I wanted to understand their emotional reactions to this new kind of health interaction.
Critical findings:
Low engagement in periodic tracking: Many users expressed that while the app felt approachable, it didn’t yet create strong motivation or cues to build a consistent routine.
Anxiety peaked during results reporting: Participants felt tense when receiving health feedback, worrying about interpretation and implications.
Lack of clarity in data transparency and caregiver visibility: Users were unsure how their data was stored or shared, and caregivers wanted clearer ways to access and manage family information.
Challenge 1
Low stickiness for periodic health check
Before
Early testing showed that abstract agents left middle-aged users feeling detached and unsure how to engage — making daily check-ins easy to skip.
After
To make the experience feel more personal and inviting, I introduced a character-based conversation agent that fostered emotional connection and routine. By generating daily topics from iPhone Health data and adding expressive response animations, the experience felt more natural, engaging, and habit-forming.
Challenge 2
High anxiety when viewing medical report
Before
Purely quantitative scoring display often triggered anxiety and skepticism. Users interpreted low scores as “failing grades,” fixated on numbers, and felt uncertain about next steps, reinforcing the sense of stress rather than encouraging continued use.
After
I redesigned the report flow to focus on emotional clarity and actionability.
Instead of raw scores, descriptive language and a “Confidence Badge” build trust without panic.
Progressive disclosure softens medical terminology, and clear, actionable suggestions turn results into a feedback loop rather than a one-off test.
Journal Content
Dashboard CTA
Brief Summary
Suggestions
Shortcut
Voice Biomarker
Metrics
Journal Calendar
Suggestions
Cognitive Score
“How were the scores calculated? Is 59 a failing grade? What did I do wrong?”
“ I think it's professional and I can learn something, but gradually I feel like, ‘What does this have to do with me?’ Can I control it? ”
“When I saw the suggestions, I was happy and felt that there was hope for improvement, but the current suggestions were too general and not actionable.”
“That‘s Okay.”
Confidence crisis
Anxiety-producing
Voice Biomarker
Metrics
Journal Content
Suggestions
I hardly got out these days, often felt sleepy but just cannot slee
0:32
}
I’m Listening
}
ningful, shall we step to save today’s conversation?
}
Congratulations!
Your Conversation Saved !
“Active Days in Berkeley: A Burst of Energy”
Topic
Go to Dashboard
Apr.1 ’s Recording
Voice Biomarkers
340 Words
Happiness
Peaceful
Total Words
Tone
Emotion
Pitch: 215 HZ
Speed：180 w/min
Good Clarity
High Energy
Low Stress
9:41
Before
After
Consolatory
Educational
Acctionable
Anxiety-Inducing Quantitative Display
Eroded Trust in Results
Lack of Accessible, Actionable Solutions
I hardly got out these days, often felt sleepy but just cannot slee
0:32
}
I’m Listening
I hardly got out these days, often felt sleepy but just cannot slee
0:32
}
I’m Listening
Challenge 3
Multiple stakeholder modes and data privacy tension
Before
Unclear information presentation led to confusion about data boundaries and control. Caregivers were unsure what they could access, while users worried about oversharing.
After
To address this, I refined the interface to make data visibility and consent states explicit. Clear visual hierarchy, mode labels, and consent cues helped distinguish personal, family, and hospital access — reducing confusion and building user trust.
Final Design
Defalt Agent | Dark Mode
Final Design
Affective Agent | Light Mode
