How Databricks Chose a Sidecarless Traffic Model With Envoy

Service-mesh discussions typically emphasize traffic policy, mutual TLS, retries, telemetry, and policy enforcement. Databricks prioritized delivering essential service components without deploying a proxy with each workload.

On the Software Plaza podcast, Twain Taylor interviewed Rohit Agrawal, Senior Staff Engineer at Databricks, about the company’s use of Envoy, xDS, and a sidecarless approach to internal traffic management. The discussion showed how a large platform can customize routing, endpoint discovery, security, and cost decisions instead of defaulting to a full service mesh.

Databricks first used a monolithic proxy, later replacing it with Envoy at the edge for authentication and rate limiting. The company did not begin with a service mesh.

The initial Envoy deployment used static configurations and Lua, with a Scala sidecar for advanced logic. Although effective, this approach increased runtime and operational costs at scale.

Why a full-service mesh did not fit

The team evaluated Istio, which uses Envoy proxies as sidecars in its data plane. While this model provides service-to-service security, routing, and telemetry, it requires managing a proxy for every workload. Full adoption would have replaced existing systems, such as certificate management and rotation, and increased the operational burden on the small traffic-platform team. Instead of rejecting service-mesh functions, Databricks integrated only the components that fit its current systems.

The final design combined Envoy with internally developed xDS components. xDS is Envoy’s set of discovery APIs for distributing configurations such as clusters, listeners, routes, and endpoints. This approach enabled Databricks to incrementally add routing and discovery features while retaining its certificate systems and service logic, using a common protocol to distribute traffic information.

Load balancing became a capacity problem

Load balancing posed a significant operational challenge. During rolling updates and canary releases, some pods received disproportionately high workloads. Connection-level affinity contributed to this, as long-lived connections could direct future requests to a single backend while other replicas remained underutilized.

The team addressed this by adding capacity headroom. Agrawal noted that Databricks needed about 40% to 50% more CPU capacity to manage uneven traffic during these events. This significant cost was mainly to compensate for an avoidable distribution issue.

The team implemented per-request balancing, routing each request independently instead of binding connections to specific backends. Endpoint Discovery Service (EDS) provided current endpoint information via xDS, allowing traffic components to consider healthy and available replicas. Envoy describes EDS as a way to update cluster hosts without draining unaffected ones. Databricks used the Power of Two Choices algorithm, which selects two candidate replicas and routes the request to the less loaded one. This simple method reduces load imbalance compared to random selection or sticky connections. The priority was to make request distribution a core infrastructure concern, not to find a perfect algorithm.

Moving the proxy work into the client

The next phase was adopting a sidecarless model. Instead of running Envoy with every pod, Databricks enabled clients to consume xDS directly and make load-balancing decisions internally. Agrawal called this a thick-client approach.

This approach worked because Databricks services already managed tasks typically handled by a sidecar, such as certificate rotation, mutual TLS, and telemetry. Adding an Envoy container to every service would have duplicated these functions. By removing the extra process, Databricks kept necessary routing information without redundancy.

This solution is not universally applicable. Sidecars can provide policy enforcement and observability without changing application code, which may be preferable when applications are diverse or when a central platform team needs consistent controls.

Databricks’ design shifted more networking responsibilities to clients managed by the platform team, removing the need for a separate proxy container across the fleet.

Security still depends on the traffic boundary

Eliminating sidecars did not remove the need for strong boundaries between the control plane and customer-facing data-plane workloads. The team maintained mutual TLS between ingress and egress proxy points for requests crossing these boundaries.

Some data-plane clients cannot initiate mutual TLS. In these cases, Databricks uses an mTLS tunneling pattern, encapsulating the original client request within a payload between proxies. The outer connection uses mTLS between proxy endpoints, while the original request keeps its own TLS layer.

This approach shows that service-mesh security functions do not need to reside in the same place in every architecture. Key considerations include where identity is established, where encryption ends, and which component enforces policy. For Databricks, these controls remained at ingress and egress points instead of adding a proxy to every application.

Where the work is heading

The Traffic Platform Team at Databricks manages both ingress and egress systems and contributes upstream when tool changes are needed. The episode highlighted work in Envoy and Argo Rollouts, which provide Kubernetes resources for canary releases, blue-green deployments, and analysis-driven progressive delivery. Envoy’s AI Gateway project and its MCP work demonstrate how these concerns are increasingly shaping proxy and gateway design. 

Databricks’ traffic design serves as a case study in selective adoption. The team used Envoy and xDS for routing and discovery, retained effective existing systems, and moved some balancing logic into clients to reduce the cost of per-pod sidecars. Network boundaries, connection behavior, rollout patterns, and existing service capabilities all influence whether this trade-off is appropriate. 

For a more detailed technical discussion, including the team’s load-balancing and extensibility work, listen to the Software Plaza episode with Rohit Agrawal.