OpenAI Introduces Websocket-Based Execution Mode to Reduce Latency in Agentic Workflows
Leela Kumili · rss · 2026-04-20 source ↗ Reader 🖥️
Transport-layer optimization — replacing per-step HTTP round-trips with a persistent WebSocket connection — yields substantial latency reductions (30-40%) in agentic AI workflows, demonstrating that system-level infrastructure changes outside the model itself are now a primary lever for improving real-world AI performance.
practical · reportorial — A trade-press news report on a concrete engineering change (HTTP to WebSocket) in OpenAI's Responses API. Reportorial rather than expository — announces the change, quotes adopters, and presents benchmark numbers without developing an argument or framework of its own.
Structure
| Part | Role |
|---|---|
| Problem statement: HTTP round-trip overhead in agentic systems | Establishes the bottleneck — as inference speeds improved, network latency from repeated HTTP requests became the dominant source of end-to-end delay |
| Solution: WebSocket-based execution mode | Describes the technical change — persistent bidirectional connection replacing request-response pattern |
| Performance evidence from OpenAI | Presents first-party benchmarks — 40% latency reduction, ~1,000 TPS sustained, 4,000 TPS burst |
| Third-party adoption and corroboration | Reports independent latency improvements from Vercel (40%), Cline (39%), and Cursor (30%) |
| Implementation and design considerations | Notes practical trade-offs — connection lifecycle management, backpressure, reliability in distributed systems |
Key terms
- WebSocket-based execution mode
A persistent, bidirectional connection between client and server for the OpenAI Responses API that replaces per-step HTTP requests with a long-lived session supporting continuous data exchange - agentic workflows Agentic Loop
Multi-step AI reasoning processes involving tool calls, intermediate reasoning, and follow-up queries — used in coding agents and real-time AI systems - transport layer (in agentic systems)
The communication patterns and connection management infrastructure between client and LLM server, as distinct from the model itself - cold starts
The latency penalty of establishing new HTTP connections for each step in a multi-tool agent chain
Propositions
- explicitReplacing per-step HTTP round-trips with a persistent WebSocket connection reduces agentic workflow latency by up to 40%.conf: high
- explicitAs model inference speeds have improved, network round-trip time from repeated HTTP requests has become the dominant source of latency in agentic systems.conf: medium
- explicitSystem-level optimizations outside the model itself are increasingly shaping real-world AI performance.conf: high
- explicitThe WebSocket approach enables warm-up of connections by pre-sending system prompts and tool definitions, further reducing latency.conf: medium
- explicitThe WebSocket mode is production-ready, as evidenced by Codex migrating most of its Responses API traffic to it.conf: high
- explicitPersistent connections trade latency gains for new complexity in connection lifecycle management, backpressure handling, and distributed reliability.conf: high
- implicitThe shift from HTTP to WebSocket for LLM agent orchestration is analogous to well-established stateful system patterns in distributed systems engineering.conf: medium
- implicitThe convergence of multiple independent platforms (Vercel, Cline, Cursor, Codex) on immediate adoption suggests WebSocket-based agent communication will become the default rather than an alternative mode.conf: medium
Arguments
Transport-layer optimization is a high-leverage intervention for agentic AI performance. strength: medium
Explicit premises:
- Inference speeds have improved, making network round-trips the dominant latency source.
- WebSocket connections eliminate repeated handshakes and enable streaming.
- OpenAI reports up to 40% latency reduction in production.
- Independent adopters (Vercel, Cline, Cursor) report 30-40% improvements.
Implicit premises:
- The per-step HTTP pattern was not questioned earlier because model inference time dominated total latency.
- Latency reduction at the transport layer compounds across multi-step workflows — each saved round-trip multiplies by the number of steps.
Inferences:
- Therefore, as models get faster, the non-model infrastructure becomes the primary optimization target.
Narratives
- Codex migrating most Responses API traffic to WebSocket mode after a two-month alpha with selected partners
→ Serves as the strongest production-readiness signal — an internal OpenAI product betting on the feature - Vercel, Cline, and Cursor reporting independent latency improvements (40%, 39%, 30%)
→ Third-party corroboration pattern — three well-known platforms independently validating the same claim adds credibility
Questions addressed
- ✅ How can latency in multi-step agentic AI workflows be reduced?
Replace per-step HTTP requests with a persistent WebSocket connection, enabling streaming and eliminating repeated connection setup. - ✅ What magnitude of improvement does the WebSocket approach deliver?
30-40% latency reduction across multiple independent platforms, with throughput of ~1,000 TPS sustained and 4,000 TPS burst. - ⭕ What new engineering challenges does the WebSocket approach introduce?
Mentioned (connection lifecycle, backpressure, reliability) but not explored in depth.
Gaps
- incompleteAll performance numbers (40% latency reduction, TPS figures) come from OpenAI or its integration partners. No independent benchmark, controlled study, or methodology description is provided. The 40% figure is repeated by adopters who may be measuring different things.
Read next: Look for independent benchmarks comparing HTTP vs. WebSocket modes on the same agentic workloads with controlled methodology. - incompleteThe article does not discuss failure modes specific to WebSocket connections in production — reconnection strategies, state recovery after disconnection, or behavior under network instability. These are well-known challenges in stateful distributed systems.
Read next: The linked 'AI Agent Transport Layer' article on InfoQ may address these concerns in more depth. - incompleteNo discussion of whether the latency gains are uniform across workflow types. A two-step tool call chain would benefit differently from a twenty-step reasoning chain. The per-step vs. end-to-end latency distinction is not made explicit.
Read next: Look for detailed breakdowns of where in a multi-step workflow the latency savings actually occur. - unacknowledgedThe article treats WebSocket adoption as purely beneficial without noting the vendor lock-in implications. A persistent-connection API is harder to abstract behind a provider-neutral interface than a stateless HTTP API. This could strengthen OpenAI's platform stickiness.
Read next: Consider how MCP or other agent protocol standards interact with provider-specific transport optimizations.