ElevenLabs Launches Realtime v1 Models for Natural Voice Agent Conversations
ElevenLabs is set to redefine voice agent development with the launch of its Eleven Realtime v1 models, a new class of Speech-to-Speech models designed to eliminate the complexity of building natural-sounding conversational AI.
Traditionally, creating a sophisticated voice agent involved integrating and fine-tuning separate components such as speech recognition, voice activity detection, turn-taking logic, large language models (LLMs), and text-to-speech systems. This process is often time-consuming and requires significant expertise. Eleven Realtime models address this challenge by acting as complete, optimized voice agents for specific use cases, streamlining the development pipeline.
Introducing Eleven Realtime v1 Models
The new offering includes two distinct models: Eleven Realtime v1 Mini and Eleven Realtime v1 Max. The v1 Mini is positioned for natural conversations at a lower cost, making it an ideal choice for AI assistants, educational tools, and interactive characters, catering to most common use cases. For more demanding applications requiring complex task completion, such as customer support, lead qualification, or multi-step workflows, the Eleven Realtime v1 Max is recommended. Both models boast extensive language support, capable of handling over 90 languages, and can leverage any of the 17,000+ voices available in the ElevenLabs library.
Rationale Behind Realtime Models
ElevenLabs emphasizes that developers should not need to become configuration experts to build effective voice agents. The nuances of natural conversation—such as appropriate pause durations, interruption handling, and the synergy between voice, prompts, and models—are critical. The ElevenAgents platform provides adjustable settings for these elements. Furthermore, Eleven Realtime is designed for easy integration, with an API compatible with OpenAI's Realtime API. Developers can switch to Eleven Realtime by simply updating the endpoint and model name in their existing code.
Performance and Benchmarks
Artificial Analysis evaluated the performance of both Realtime v1 models across several public benchmarks. In Big Bench Audio, which tests speech reasoning, v1 Max achieved an impressive 98.8% accuracy, while v1 Mini scored 75.9%. Regarding speed, v1 Max responded in an average of 0.57 seconds, and v1 Mini in 0.62 seconds. These results highlight v1 Max's strong performance in both accuracy and speed, addressing the typical trade-off between the two metrics.
For conversational dynamics, tested on Full-Duplex-Bench (evaluating pauses, turn-taking, and interruptions), v1 Mini scored 95.8% and v1 Max scored 94.1%. In the Speech Agent Arena, where human participants compare models in blind conversations, v1 Max received 998 points and v1 Mini 988 points. For tool-use tasks, v1 Max demonstrated superior capability, completing 87.1% of tasks compared to v1 Mini's 60.3%, reinforcing its suitability for complex workflows involving tool invocation.
Getting Started with Eleven Realtime
Developers can begin using the Realtime models through the ElevenAgents platform by selecting a Realtime model from a dropdown menu. This provides access to production deployment features, including support for various channels (phone, web, WhatsApp, CCaaS), workflow management, knowledge bases, guardrails, testing, and monitoring tools. Comprehensive documentation is available in the ElevenAgents docs.
For those with existing integrations using OpenAI's Realtime API, migration is straightforward. By changing the WebSocket endpoint to wss://api.elevenlabs.io/v1/convai/realtime and specifying the desired model (e.g., eleven_realtime_v1_mini), developers can leverage their current code. It's important to note that the Realtime API operates exclusively over WebSocket, requiring any WebRTC integrations to be migrated first. This API operates independently of the full ElevenAgents platform, so users requiring platform features should build directly on ElevenAgents.
