OpenAI Realtime vs LiveKit Agents vs Pipecat: Voice AI 2026

Voice AI platforms like Vapi, Retell, and Hume handle the full stack for you. They are great for getting to production fast. But plenty of teams in 2026 need more control. Maybe you want to pick your own STT and TTS providers. Maybe you need to self-host for compliance. Maybe you are building a product where voice is the core experience and you cannot afford vendor lock-in on the most critical piece of your stack.

That is where infrastructure-level tools come in. Three options dominate the voice AI infrastructure layer in 2026: OpenAI Realtime API, LiveKit Agents, and Pipecat. Each takes a fundamentally different approach. OpenAI Realtime gives you a hosted, speech-to-speech model with the lowest possible latency. LiveKit Agents gives you an open-source, multi-model orchestration framework built on WebRTC. Pipecat gives you a pipeline-based, model-agnostic framework you can run anywhere.

Choosing the wrong one will cost you months of rework when you hit scaling or flexibility walls. We have shipped production voice products on all three at our agency, and the differences matter more than the marketing pages suggest. If you are earlier in your journey, our guide on building an AI voice agent covers the foundational decisions you should make first.

OpenAI launched the Realtime API in late 2024 and has iterated aggressively since. The core idea is simple: instead of chaining STT, LLM, and TTS as separate services, the Realtime API processes speech input and produces speech output in a single model pass. Audio goes in, audio comes out, with GPT-4o-level reasoning in between.

Architecture: You open a persistent WebSocket connection to OpenAI's servers. You stream raw audio frames (PCM16, 24kHz). The model processes speech natively, no separate transcription step. It generates a spoken response directly, with text transcripts available as a side channel. Function calling works mid-conversation. The model supports interruptions natively.

Latency: This is where Realtime shines. Because there is no STT-to-LLM-to-TTS pipeline, the model achieves 200 to 400ms voice-to-voice latency in production. That is faster than any chained pipeline can match. For comparison, a typical Deepgram STT + GPT-4o + ElevenLabs TTS chain runs 600 to 1200ms. The difference is immediately noticeable in conversation.

Model flexibility: Zero. You get GPT-4o Realtime. That is it. No Claude, no Gemini, no open-source models, no fine-tuned variants. If GPT-4o handles your use case well, this constraint does not matter. If you need specialized reasoning, multilingual capabilities beyond what GPT-4o offers, or a model you have fine-tuned on domain data, you are stuck.

Cost: OpenAI charges $0.06 per minute of audio input and $0.24 per minute of audio output as of mid-2026. A typical 5-minute conversation costs roughly $1.50. That is 3x to 5x more expensive than running your own STT + LLM + TTS pipeline with commodity providers. At 100,000 minutes per month, you are looking at $30,000 just for the Realtime API, versus $6,000 to $10,000 for a self-managed pipeline.

WebRTC: The Realtime API does not include WebRTC. You get a WebSocket. For browser-based voice apps, you still need to handle WebRTC yourself or use a service like LiveKit or Daily to bridge between browser audio capture and the WebSocket. OpenAI partnered with Agora and LiveKit for reference implementations, but the transport layer is your responsibility.

Self-hosting: Not possible. This is a fully hosted, closed-source API. Your audio goes to OpenAI's servers. For regulated industries (healthcare, finance, government), this can be a non-starter depending on your compliance requirements.

LiveKit started as an open-source WebRTC infrastructure project. Their Agents framework, released in 2024 and now mature in 2026, layers voice AI orchestration on top of that WebRTC backbone. The result is a framework where you pick your own STT, LLM, and TTS providers and LiveKit handles the real-time audio transport, pipeline orchestration, and session management.

Architecture: LiveKit Agents runs as a Python or Node.js process that connects to a LiveKit server (self-hosted or LiveKit Cloud). Audio arrives over WebRTC. The agent framework pipes it through your chosen STT provider, feeds the transcript to your LLM, and streams the LLM output to your TTS provider. The spoken response goes back over WebRTC to the client. The framework handles VAD (voice activity detection), interruption handling, turn-taking, and audio buffering.

Latency: Pipeline latency depends on your provider choices. With Deepgram STT + GPT-4o-mini + Cartesia TTS, expect 500 to 800ms voice-to-voice. With faster providers (Groq for LLM inference, Deepgram Aura for TTS), teams have achieved 350 to 600ms. That is slower than OpenAI Realtime but fast enough for natural conversation. The WebRTC transport itself adds minimal overhead, typically under 50ms.

Model flexibility: This is LiveKit's biggest advantage. You can swap any component independently. Use Deepgram or AssemblyAI or Whisper for STT. Use GPT-4o, Claude, Gemini, Llama, Mistral, or any OpenAI-compatible endpoint for the LLM. Use ElevenLabs, Cartesia, PlayHT, or XTTS for TTS. You can even use OpenAI Realtime API as a provider within the LiveKit Agents framework, getting the best of both worlds.

Cost: LiveKit Cloud charges $0.02 per participant-minute for media routing. STT, LLM, and TTS costs are whatever your chosen providers charge. A typical conversation using Deepgram ($0.0043/min STT) + GPT-4o-mini ($0.01/min estimated) + Cartesia ($0.015/min TTS) + LiveKit Cloud ($0.02/min) runs about $0.05 per minute total. That is 5x cheaper than OpenAI Realtime for equivalent conversations. Self-hosting the LiveKit server eliminates the $0.02/min media cost entirely.

WebRTC: Native. This is LiveKit's core competency. Browser clients, mobile SDKs (iOS, Android, Flutter, React Native), and SIP integration all work out of the box. You get automatic codec negotiation, bandwidth adaptation, connection recovery, and end-to-end encryption if you need it.

Self-hosting: Fully supported. LiveKit server is open source (Apache 2.0). LiveKit Agents framework is also open source. You can run the entire stack on your own infrastructure, in your own VPC, with zero data leaving your network. This is why healthcare and fintech teams gravitate toward LiveKit.

Pipecat, created by Daily.co, takes yet another approach. It is a Python framework that models voice AI as a pipeline of processors. Audio comes in, flows through a chain of processing steps (VAD, STT, LLM, TTS, output), and audio goes out. Each step is a swappable component. The framework handles async streaming, backpressure, interruptions, and pipeline lifecycle.

Architecture: A Pipecat pipeline is a directed graph of "frame processors." You compose them in code. A minimal voice agent pipeline looks like: Transport Input, VAD, STT, LLM, TTS, Transport Output. But you can insert custom processors anywhere. Want to log every transcript to a database? Add a processor after STT. Want to run sentiment analysis before the LLM sees the text? Insert a processor. Want to apply content filtering to the LLM output before TTS? Add another step. This composability is Pipecat's core strength.

Latency: Similar to LiveKit Agents since the bottleneck is the same STT + LLM + TTS chain. With optimized providers, 400 to 700ms is typical. Pipecat's framework overhead is minimal, under 10ms. The transport layer matters here. Using Daily as the WebRTC transport gives you solid performance. Using a raw WebSocket adds less overhead but requires you to handle audio encoding and client-side capture yourself.

Model flexibility: Equivalent to LiveKit Agents. Pipecat ships with integrations for Deepgram, Whisper, AssemblyAI (STT), OpenAI, Anthropic, Google, Fireworks, Together, Groq, Ollama (LLM), ElevenLabs, Cartesia, PlayHT, Azure TTS, XTTS (TTS). Adding a new provider means implementing a single Python class. The ecosystem of community-contributed processors is growing fast.

Cost: Pipecat itself is free and open source (BSD license). You pay only for the providers you use and your transport layer. Using Daily as the transport costs $0.01 per participant-minute. Total per-minute cost with commodity providers: $0.03 to $0.06. Self-hosting the transport (using a raw WebSocket or your own WebRTC server) brings the framework cost to zero.

WebRTC: Pipecat supports WebRTC through Daily's transport layer, which works well but ties you to Daily for that piece. There is also a community WebSocket transport and experimental support for LiveKit as a transport. The WebRTC story is less mature than LiveKit's native implementation. If WebRTC is central to your product, LiveKit has the edge.

Self-hosting: The framework runs anywhere Python runs. Your laptop, a Docker container, a Kubernetes cluster, a Raspberry Pi. The transport layer determines your self-hosting options. WebSocket transport is trivially self-hosted. Daily transport requires Daily's cloud service. The LLM/STT/TTS providers can be self-hosted (Whisper, Ollama, XTTS) or cloud-hosted, your call entirely.

Let us put hard numbers side by side. These benchmarks come from our production deployments in early 2026, not synthetic tests.

Voice-to-Voice Latency (p50 / p95)

• OpenAI Realtime API: 250ms / 450ms. The fastest option by a meaningful margin. No pipeline to chain.
• LiveKit Agents (Deepgram + GPT-4o-mini + Cartesia): 550ms / 850ms. Solid for conversation. Users rarely notice.
• LiveKit Agents (Groq + Deepgram + Cartesia): 380ms / 620ms. Groq's inference speed closes the gap with OpenAI Realtime.
• Pipecat (Daily + Deepgram + Claude 3.5 + ElevenLabs): 600ms / 950ms. ElevenLabs TTS adds latency. Swapping to Cartesia drops this by 100ms.
• Pipecat (WebSocket + Deepgram + Groq Llama + Cartesia): 400ms / 650ms. Fastest Pipecat configuration we have tested.

Cost Per Minute of Conversation

• OpenAI Realtime API: $0.30/min all-in. No way to optimize without reducing conversation length.
• LiveKit Agents (Cloud): $0.04 to $0.08/min depending on provider choices. LiveKit Cloud adds $0.02/min.
• LiveKit Agents (Self-hosted): $0.02 to $0.06/min. Server costs are negligible at scale.
• Pipecat (Daily transport): $0.03 to $0.07/min. Daily adds $0.01/min.
• Pipecat (Self-hosted transport): $0.02 to $0.05/min. Lowest possible cost.

Model Swap Flexibility

• OpenAI Realtime: None. GPT-4o only. Cannot swap STT, LLM, or TTS independently.
• LiveKit Agents: Full flexibility. 15+ provider integrations maintained by LiveKit team. Hot-swapping providers requires a config change, not a code rewrite.
• Pipecat: Full flexibility. 20+ provider integrations. Community-contributed processors extend this further. Slightly more code required to swap compared to LiveKit's plugin system.

The pattern is clear. OpenAI Realtime wins on latency and loses on everything else. LiveKit and Pipecat are close on cost and flexibility, with LiveKit having a stronger WebRTC story and Pipecat offering more pipeline customization. For a broader look at where these tools fit into real products, see our overview of voice AI applications across industries.

Choose OpenAI Realtime API when: Latency is your top priority and cost is secondary. You are building a consumer product where every 100ms of delay hurts engagement. Your use case works well with GPT-4o's capabilities without fine-tuning. You do not need to self-host. You want the fastest path to a working prototype. Typical fit: consumer voice apps, gaming NPCs, real-time language tutoring, interactive entertainment.

Choose LiveKit Agents when: You need WebRTC as a first-class transport layer. You want to self-host the entire stack for compliance or cost reasons. You need multi-model flexibility and expect to swap providers as the market evolves. You are building a platform where voice is a core feature, not a bolt-on. You want a strong community and commercial support options. Typical fit: telehealth platforms, contact center infrastructure, multi-tenant SaaS with voice features, enterprise apps with data residency requirements.

Choose Pipecat when: You need maximum pipeline customization. You want to insert custom processing steps (sentiment analysis, content filtering, real-time translation, custom business logic) between standard voice AI components. You prefer a lightweight framework over a full infrastructure platform. You are comfortable managing your own transport layer. Typical fit: research prototypes, custom voice AI products with novel processing requirements, teams with strong Python expertise, projects where the voice pipeline is a differentiator.

Hybrid Approaches

You do not have to pick one forever. Several of our clients start with OpenAI Realtime for speed-to-market, then migrate to LiveKit Agents when they need cost optimization or model flexibility at scale. Others use LiveKit for their WebRTC infrastructure but run Pipecat pipelines as the agent logic within LiveKit rooms. The OpenAI Realtime API is available as a provider plugin in both LiveKit Agents and Pipecat, so you can use it as the "brain" while controlling the transport layer yourself.

The migration path matters. Moving from OpenAI Realtime to a pipeline-based approach (LiveKit or Pipecat) is straightforward because you are decomposing a monolith into components. Moving from Pipecat to LiveKit (or vice versa) is moderate effort since the provider integrations are similar. Moving from LiveKit or Pipecat back to OpenAI Realtime means giving up all your custom pipeline logic, which is painful if you have invested in it.

Getting a voice AI demo working is the easy part. Running it reliably at scale with real users is where these frameworks diverge.

Scaling

OpenAI Realtime scales automatically since it is a hosted API. Your only constraint is rate limits, which OpenAI sets based on your tier. At launch they were restrictive (100 concurrent sessions). By 2026, Tier 4+ accounts get 1,000+ concurrent sessions. If you need more, you negotiate directly with OpenAI's sales team. You have no control over geographic distribution or failover.

LiveKit Agents scales horizontally. Each agent process handles one conversation. You run as many agent processes as you need, orchestrated by Kubernetes, ECS, or any container scheduler. LiveKit server handles room routing and load balancing. LiveKit Cloud manages all of this for you. Self-hosted, you are responsible for capacity planning, but the architecture is well-documented and battle-tested from LiveKit's video conferencing roots.

Pipecat scales similarly to LiveKit Agents since each pipeline instance handles one conversation. You containerize the pipeline process and scale with your orchestrator of choice. The framework does not include built-in load balancing or room management, so you need to build or borrow that layer. For teams already running Kubernetes, this is straightforward. For smaller teams, it is extra operational overhead.

Monitoring and Observability

OpenAI Realtime gives you basic usage metrics through their dashboard. No per-conversation latency breakdowns, no audio quality metrics, no custom alerting. You build your own monitoring by logging WebSocket events.

LiveKit provides a dashboard (Cloud) or Prometheus metrics (self-hosted) with per-room, per-participant metrics. Agent-level telemetry includes STT latency, LLM time-to-first-token, TTS latency, and end-to-end voice-to-voice timing. This is the most observable option.

Pipecat added OpenTelemetry support in 2025. You get span-level tracing through each pipeline processor. Integrate with Datadog, Grafana, or any OTEL-compatible backend. Slightly more setup than LiveKit's built-in metrics, but more flexible.

Failure Modes

OpenAI Realtime has had multiple outages affecting all customers simultaneously. When their API goes down, your product goes down. No failover option exists because there is no alternative provider for the same API.

LiveKit Agents lets you build provider failover directly. If Deepgram STT is down, fall back to AssemblyAI. If GPT-4o is slow, route to Claude. If ElevenLabs is degraded, switch to Cartesia. This resilience is a major advantage for production systems.

Pipecat supports the same failover patterns. You can implement retry logic and provider switching at the processor level. Some teams run parallel STT processors and use the first result that returns, reducing both latency and single-provider risk.

After shipping production voice AI on all three frameworks, here is our honest take. If you are a startup building a voice-first product and you need to move fast, start with OpenAI Realtime API. The latency advantage is real, and you will get to user feedback faster than with any other option. Plan your architecture so you can migrate later when cost or flexibility forces you to.

If you are building infrastructure that needs to last, especially if you have compliance requirements, need multi-model flexibility, or expect to handle thousands of concurrent sessions, go with LiveKit Agents. The WebRTC foundation, open-source licensing, and self-hosting option give you the control that production systems demand. The community is active, the documentation is strong, and LiveKit's commercial support is responsive.

If your voice AI product requires novel pipeline processing that goes beyond standard STT + LLM + TTS, or if you want the lightest possible framework with maximum composability, Pipecat is the right choice. It is particularly strong for research teams, custom product experiences, and situations where you need to insert business logic at every stage of the voice pipeline.

For most of the teams we work with, LiveKit Agents hits the sweet spot. You get production-grade WebRTC, full model flexibility, reasonable latency, and a clear self-hosting path. The cost savings over OpenAI Realtime are substantial at scale, often 5x or more per minute. And the ability to swap providers as the voice AI market evolves (it will keep evolving rapidly) protects your investment.

Voice AI infrastructure decisions are hard to reverse once you are in production with real users. If you want help evaluating these options for your specific use case, or you need a team that has shipped on all three, book a free strategy call and we will walk through the tradeoffs together.