When it comes to healthcare, the line between helpful and intrusive is razor-thin—and AI chatbots are tiptoeing right along it. A recent study from the University of Surrey sheds light on this delicate balance, revealing that while patients appreciate friendly and choice-oriented language in AI interactions, pushy reminders and blurred human-AI boundaries can be a major turnoff. Personally, I think this highlights a broader issue in the design of AI systems: the tension between creating a human-like experience and respecting users’ boundaries. What makes this particularly fascinating is how patients’ reactions to the chatbot, Asa, varied so dramatically based on its tone and approach.
One thing that immediately stands out is the importance of language in building trust. Patients described Asa’s friendly and kind tone as reassuring, especially when discussing sensitive topics. From my perspective, this underscores the power of empathy in communication—even when it’s coming from a machine. What many people don’t realize is that the gender presentation of the chatbot also played a role; some patients felt more comfortable disclosing personal information to a female-presenting AI. This raises a deeper question: Are we unconsciously projecting human biases onto AI, and if so, how should designers account for this?
However, the study also revealed where the chatbot fell short. Imperative phrasing like ‘Let’s book you in’ was perceived as aggressive, particularly by patients managing mental health challenges or neurodivergent conditions. In my opinion, this is a critical oversight. If you take a step back and think about it, healthcare interactions require a level of sensitivity that goes beyond mere functionality. A detail that I find especially interesting is how the chatbot’s attempt to blur the human-AI boundary—by encouraging users to ‘chat to me as if I am a real person’—backfired. What this really suggests is that transparency is non-negotiable in healthcare settings.
Ethical concerns were another major sticking point. Patients expressed worries about data security and the potential for impersonation, which, in my view, highlights a broader societal anxiety about AI’s role in intimate spaces like healthcare. What this study makes clear is that anthropomorphism isn’t a one-size-fits-all solution. While human-like features can build rapport, they can also undermine trust if they clash with users’ expectations for clarity and honesty.
If we zoom out, the implications of this research are significant. Cervical screening uptake in the UK has been declining, and ethnic minority groups remain underrepresented in screening programs. This isn’t just about designing better chatbots—it’s about ensuring equitable access to healthcare. Feeling seen, appreciated, and emotionally supported shouldn’t be a luxury; it should be the baseline. If patients disengage because a chatbot feels pushy or untrustworthy, the consequences could be dire.
In my opinion, the key takeaway here is that AI in healthcare isn’t just a technological challenge—it’s a human one. Designers need to prioritize respect, transparency, and fairness, treating patients not as targets to be nudged but as individuals with unique needs and boundaries. As we move forward, I’ll be watching closely to see how these lessons are applied. After all, the goal isn’t just to make AI work—it’s to make it work for us.
