In this episode of Unprompted, Rebecca Evanhoe joins Hans van Dam to explore what it really takes to design and scale conversational AI in production. Drawing on her work building voice AI for thousands of restaurants at Slang.ai, Rebecca shares a grounded, experience-driven perspective on hybrid systems, evaluation, and why strong conversation design principles matter more than ever in the age of LLMs.
The conversation moves beyond hype to focus on the realities of deploying conversational AI at scale. Rebecca explains why hybrid architectures, combining deterministic logic with carefully scoped LLM tasks, consistently outperform fully generative approaches, especially in voice-based, high-stakes environments. She and Hans unpack how conversational AI quality is shaped not just by models, but by speech recognition, latency, turn-taking, and deep contextual understanding of user intent.
A major theme of the episode is evaluation. Rebecca walks through how her team designs experiments to test prompts and AI behaviors before shipping them to production, from entity extraction and business-hours questions to multilingual translation. Rather than trusting LLMs “out of the box,” she emphasizes defining success criteria, building realistic test sets, and using human and automated evaluation together to gain confidence and reduce hallucinations.
Together, Rebecca and Hans also reflect on how the role of the conversation designer is evolving. While tools have changed, the core craft remains the same: understanding user context, shaping behavior, and designing interactions that respect human conversational norms. Rebecca challenges the idea that conversation design is becoming obsolete, arguing instead that designers who can diagnose failures across the stack, from UX to AI to infrastructure, are becoming more valuable, not less.
The episode also touches on voice-specific challenges, multilingual systems, transparency around AI disclosure, and the future of brand expression in conversational experiences. Throughout, Rebecca offers a clear message for practitioners and learners alike: mastering conversational AI today means combining timeless design principles with rigorous evaluation and a deep understanding of where AI truly adds value.
This episode also covers:
- Why hybrid conversational systems outperform fully generative approaches
- How to evaluate LLMs for real-world conversational use cases
- Designing voice AI that respects turn-taking, timing, and social norms
- Using LLMs as components rather than full conversation drivers
- The growing importance of evaluation as a core design skill
- What conversation designers should focus on as AI tooling evolves
- Balancing automation, hospitality, and brand experience in voice interfaces
Whether you’re building production-grade conversational systems or learning how AI is reshaping the role of the conversation designer, this episode offers a practical, no-nonsense look at what it takes to design conversational AI that actually works.
