calesthio/elevenlabs-agents
Build, configure, and ship production voice agents on the ElevenLabs Agents platform (branded "ElevenAgents," formerly "Conversational AI"). Use when an agent must design a spoken-conversation system prompt, pick an LLM and TTS voice/model for a real-time voice bot, wire client/server/system tools and knowledge-base RAG, connect a channel (WebRTC/WebSocket SDK, embeddable widget, or telephony via Twilio/SIP), tune turn-taking and latency, set up simulation testing and evaluation, or reason about pricing, concurrency, and the consent/disclosure obligations of a synthetic voice talking to real people. Not for plain non-conversational text-to-speech, dubbing, or music generation.
npx skills add https://github.com/calesthio/generative-media-skills --skill elevenlabs-agents
Production guidance for building voice (and text) agents on the ElevenLabs Agents platform. The product was renamed from "Conversational AI" to "ElevenLabs Agents" / "ElevenAgents" during 2026; documentation URLs use both /docs/eleven-agents/... and legacy /docs/agents-platform/... paths, and the two mostly mirror each other. All model IDs, prices, limits, and endpoints below are volatile and were verified on 2026-07-10 against elevenlabs.io/docs — re-verify before quoting them to a user, because ElevenLabs adds LLMs and revises plans frequently.
Use this skill when the work is a two-way spoken (or typed) conversation driven by the ElevenLabs Agents runtime: phone bots, website voice widgets, in-app voice assistants, WhatsApp/chat agents, outbound call campaigns, or a custom SDK integration. It also applies to configuring the agent's brain (prompt, LLM, tools, knowledge), its voice, its channel, and its evaluation.
Do not use it for one-shot text-to-speech, audiobook narration, dubbing, sound effects, or music — those are separate ElevenLabs products with their own APIs. If the user only wants a .mp3 from text, they want the TTS API, not an agent.
[Documented] An ElevenAgents conversation is an orchestrated loop over four components (elevenlabs.io/docs/eleven-agents/overview, verified 2026-07-10):
Your job is to choose and tune each stage so the whole loop feels like a person. [Heuristic] Latency and quality trade against each other at every stage; optimize the loop as a whole, not one component in isolation.
[Documented] ElevenLabs advertises first-turn latency under ~500ms for the ElevenAgents 2.0 stack (elevenlabs.io blog, 2026). Approximate per-stage contributions from ElevenLabs' own latency write-up (elevenlabs.io/blog/how-do-you-optimize-latency-for-conversational-ai, verified 2026-07-10):
| Stage | Typical contribution |
|---|---|
| ASR (ElevenLabs custom) | <100 ms |
| Turn-taking / VAD silence detection | tunable; adds the "end-of-turn" wait |
| LLM first-token | Gemini Flash <350 ms; GPT-4-class / Claude Sonnet 700–1000 ms |
| TTS (Flash v2.5) | ~75 ms model time, ~135 ms end-to-end |
| Network (same-region) | ~200 ms+ |
| Telephony (global) | ~500 ms+ |
[Independent benchmark, disclosed method] Third-party monitoring (Cekura, Deepgram, 2026) reports ElevenLabs among the fastest orchestrators — median turn ~1.7s — but with a long tail (P95 ~3.2s), driven mostly by LLM variance and telephony. Treat the sub-500ms figure as a best-case for the ElevenLabs-controlled stages, not the end-to-end phone experience.
[Heuristic] Practical latency levers, highest-impact first:
eleven_flash_v2_5 for the voice unless you have a specific quality reason not to.[Heuristic] A voice-agent prompt is not a chatbot prompt. Spoken output is heard once, in sequence, with no scrollback, so:
[Documented] The dashboard splits this into a First message (the opening line, supports dynamic variables) and a System prompt. A recent platform field, trust_context, marks an agent as low (serving untrusted external callers — restrict tool access) or high (serving the owner — full tool access appropriate) (changelog, 2026, verified 2026-07-10).
[Documented, volatile — verified 2026-07-10] ElevenAgents supports a large, frequently-updated roster including:
The exact list changes almost monthly; always read the live Models page rather than trusting a memorized roster.
[Heuristic] Choose by the job:
[Documented] LLM cascading / fallback: ElevenAgents can fall back through a backup sequence when the primary model errors or times out, protecting the conversation from a single provider outage. A documented default cascade (verified via docs search 2026-07-10) runs Gemini 2.5 Flash → Gemini 2.0 Flash → Claude 3.7 Sonnet → Claude 3.5 Sonnet v2. [Heuristic] For production, keep cascading on and make sure every model in the chain is fast enough that a fallback doesn't wreck the call.
[Documented, verified 2026-07-10] TTS model tradeoffs for agents:
| Model ID | Latency | Languages | Best for |
|---|---|---|---|
| eleven_flash_v2_5 | ~75 ms | ~32 | Default for real-time agents. Best speed/quality balance, ~50% lower cost. |
| eleven_flash_v2 | ~75 ms | English only | English-only, cost-sensitive. |
| eleven_multilingual_v2 | Higher | 29 | Highest lifelike quality/emotion; long-form; when quality > latency. |
| eleven_v3 | Higher | 70+ | Most expressive/dramatic; character work; generally not the low-latency real-time pick. |
| eleven_turbo_v2_5 | Mid | ~32 | Deprecated for agents; superseded by Flash v2.5. |
[Heuristic] Default to eleven_flash_v2_5. Escalate to eleven_multilingual_v2 only when a brand demands maximum warmth/expressiveness and can absorb the added latency. Higher-quality voice + model + LLM combined can visibly slow the turn — the dashboard itself warns of this.
[Documented] Voice settings that matter for agents: stability (lower = more emotional range and variation, higher = flatter/consistent), similarity boost (adherence to the source voice), and speaker boost (similarity at a slight latency cost). TTS output can be mp3, pcm, or ulaw — telephony typically needs ulaw 8kHz; SIP media runs G711 8kHz or G722 16kHz.
[Heuristic] Pick a voice whose native accent/language matches the caller base, keep stability moderate (very low stability can produce jarring swings across turns), and test the voice on your *actual* script — sample text flatters voices that stumble on domain jargon, phone numbers, and proper nouns.
[Documented] Agents can run in a fixed language or detect/switch languages; TTS covers 70+ languages overall, with Flash v2.5 at ~32 and Multilingual v2 at 29. [Heuristic] For a multilingual agent, confirm all three stages support the target language: ASR recognition quality, the LLM's fluency, and the chosen TTS model's coverage. A voice cloned from one language may carry an accent when speaking another — audition it.
[Documented] ElevenAgents 2.0 uses a hybrid turn-taking model: a VAD plus deep-learning signals (filler words, prosody, rhythm, micro-pauses) to judge end-of-turn and handle barge-in. Configurable controls include turn eagerness (Eager / Normal / Patient) and a turn timeout from 1–30s (verified 2026-07-10).
[Heuristic]
background_sound (ambient audio) option exists and an agent_response_complete client event now fires once when a turn is fully delivered — useful for reliable end-of-turn UI, wake-word gating, or message batching (changelog 2026).[Documented, verified 2026-07-10] Attach documents to an agent as grounded knowledge:
conversation_config.agent.prompt.knowledge_base, each with type, name, id, and an optional usage_mode. RAG can be enabled so the agent retrieves relevant chunks rather than stuffing everything into context.[Heuristic] Split large manuals into focused documents, keep them current, and mine transcripts for questions the KB failed to answer. Put *stable* reference material in the KB; put *per-user* facts (account balance, order status) in dynamic variables or a server tool — never bake a customer's data into the KB.
[Documented] Three tool types:
type: "client", a name, description, parameters, and expects_response / "Wait for response" — when enabled, the agent pauses and appends the returned data to the conversation before continuing.[Heuristic] Tool reliability depends heavily on the LLM. Give each tool a crisp description and each parameter a clear description and value_type so the model extracts arguments correctly. For flows where a mis-fired tool is costly (payments, cancellations), use a higher-intelligence LLM and add tool-call tests (below). Design tools to be idempotent where possible and mask their latency with a spoken filler.
[Documented, verified 2026-07-10]
{{user_name}}, {{account_tier}}) into the prompt, first message, and tools — the recommended way to personalize per conversation. Passed via conversation_initiation_client_data.dynamic_variables. System variables like system__caller_id, system__called_number, system__call_duration_secs, system__agent_id are auto-available but cannot be set/overridden by the client payload.[Documented] Agent Workflows are a visual editor for branching conversation graphs that route to specialized subagents, each with its own prompt, tools, and scoped knowledge base, and can hand off to a human. Two handoff mechanisms:
transfer_to_agent call is stripped from the child's visible history so it continues seamlessly. Supports an optional delay and an optional spoken transfer message (silent if blank).[Heuristic] Prefer several small, single-purpose subagents over one giant prompt: each stays short (lower latency, fewer instructions to violate) and easier to test. Reserve human transfer for genuine escalation and always give the agent an explicit fallback path so a stuck caller is never trapped.
[Documented, verified 2026-07-10]
@elevenlabs/react), JavaScript/Web (@elevenlabs/client), Swift (iOS), Kotlin (Android), React Native, plus Python/TypeScript server SDKs. A CLI (@elevenlabs/cli) manages agents as code.startSession takes signedUrl, conversationToken, or agentId and infers transport from mode.[Documented] Never put the ElevenLabs API key in client code. For authenticated agents:
[Heuristic] Always run authenticated agents in production so anyone can't rack up minutes on your agent ID, and gate the token/signed-URL endpoint behind your own auth. Public/unauthenticated agents are fine only for open demos.
[Documented, verified 2026-07-10] ElevenAgents has a built-in test framework, runnable from the dashboard Tests tab, the CLI (elevenlabs agents test <agent_id>), or the SDK (run_tests / runTests, with repeat_count for probabilistic runs):
^ORD-[0-9]{8}$), or LLM semantic check.[Documented] Post-conversation evaluation criteria (success rubrics) and data collection (structured extraction) run automatically on live conversations using the same criteria as simulation, feeding analytics.
[Heuristic] Build a suite covering the happy path, each tool, each transfer condition, and the hostile/off-topic/silent edges; run it before every prompt or model change. Prompt and model swaps routinely regress behavior that "obviously" worked.
[Documented] The platform provides conversation search, transcripts, success-evaluation results, extracted data-collection fields, A/B testing, and performance metrics for monitoring at scale. [Heuristic] Watch median *and* tail latency, interruption/barge-in rates, tool error rates, transfer-to-human rate, and success-criteria pass rate — the tail and the transfer rate reveal production pain the averages hide.
[Documented, volatile — verified 2026-07-10] ElevenAgents bills by call minutes, separate from the shared TTS credit pool; the LLM and telephony are billed on top by usage. Representative self-serve tiers (included minutes / concurrent calls):
| Plan | Included minutes | Concurrent calls |
|---|---|---|
| Free | 15 | 4 |
| Starter | 75 | 6 |
| Creator | 275 | 10 |
| Pro | 1,238 | 20 |
| Scale | 3,738 | 30 |
| Business | 12,375 | 40 |
Overage ≈ $0.08/min standard, $0.16/min burst; text messages ≈ $0.003 each. Burst pricing, when enabled, lets you exceed your concurrency cap up to ~3× at the doubled rate. There is no cap on how many agents you can create; the binding constraint is concurrent calls.
[Heuristic] Concurrency, not minutes, is what breaks at scale — a spike of simultaneous inbound calls hits the ceiling instantly and extra callers get rejected or burst-billed. For real call-center volume (hundreds of simultaneous calls) you need an enterprise concurrency arrangement; do not assume a self-serve plan absorbs a marketing blast. Model *both* levers: total minutes for the monthly bill, concurrent calls for the busy-hour peak.
This is a legal and ethical surface, not a nicety. [Documented / regulatory]
[Heuristic] Bake compliance into config: disclosure in the first message, an easy "talk to a human" transfer, a recording/consent notice where required, timezone-aware scheduling for outbound, and DNC checks before dialing. Treat "is the user OK talking to a bot, and did they consent to the call?" as a design requirement, not an afterthought.
[Heuristic, from documented behavior]
eleven_flash_v2_5, tighten the prompt, add spoken fillers for tool waits.Intent: an authenticated, low-latency support agent for a mid-size e-commerce brand, deployed as a website voice widget with a phone fallback, that can look up order status and escalate to a human.
low (external callers).eleven_flash_v2_5, moderate stability, brand voice matched to the customer base's accent.get_order_status whenever the user references an order.get_order_status(order_id) → your backend; system tool transfer_to_number for escalation. order_id parameter has a clear description and a regex-checkable format.{{customer_first_name}} and {{account_tier}} injected as dynamic variables at session start; system__caller_id used to pre-lookup the account on phone calls.get_order_status fires with a regex-valid order_id, and a partial simulation for the escalation branch — each run with repeat_count ≥ 5.get_order_status runs, clean handoff to a human on out-of-scope asks.Take calesthio/elevenlabs-agents from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.