Read as markdown

Fun and games with Istio in production

Written on 2026-11-10

This is a post detailing some interesting things you can do with Istio and Envoy, and includes a small discussion on Istio's performance.

Istio and Envoy

Istio is a control plane for controlling Envoy proxies, with some additional infrastructure like istio-csr for PKI used for mutual TLS, as well as some observability integrations.

Istio has its own API objects (most often in the form of Kubernetes custom resources), like VirtualService, ServiceEntry, and DestinationRule.

It also exposes EnvoyFilter (reference) which can be used to change Envoy's underlying configuration directly. This is useful, as Envoy's configuration surface is very broad, and you can create some intricate and high performance configurations.

EnvoyFilter example

A simple example, lifted directly from the Istio docs, would be to configure rate limiting:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: domain-match-example
  namespace: myns
spec:
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      context: GATEWAY
      routeConfiguration:
        vhost:
          domainName: 'foo.com'
    patch:
      operation: MERGE
      value:
        rate_limits:
          actions:
            - request_headers:
                header_name: "authorization"
                descriptor_key: "jwt"
            - request_headers:
                header_name: ":path"
                descriptor_key: "path"

Istio endpoint propagation performance

Istio configures Envoy to stream configuration updates from Istio's control plane: istiod. Envoy can receive dynamic configuration through a custom protocol, xDS

xDS has many stream types, like RDS (route discovery), EDS (endpoint discovery), CDS (cluster discovery) and so forth. ADS is "aggregated discovery" which bundles multiple configuration types together. The purpose of ADS is to allow the control plane to sequence updates properly. For example, sending a route update which routes to a new cluster, before the cluster configuration is sent, will cause traffic to be dropped with errors.

With an SLO of "99.95% of endpoint updates propagated within 30s", we saw SLO violations beyond tens of thousands of pods, and scaling Istiod horizontally or vertically produced diminishing returns. Under high load we saw the SLO degrade to 50-60s. This 30s SLO is important because it is the contract between mesh consumers and producers: endpoints become unready and stick around for 30-60s to finish serving in-flight traffic. Delayed endpoint updates may result in a client sending traffic to an endpoint that no longer exists.

Our mesh setup predates Istio and we have many endpoints outside of Kubernetes, and so we choose to store endpoints in ZooKeeper (Kubernetes, bare-metal, VMs, etc), and then these were propagated to Istio. The high churn in the fleet (ie developers deploying thousands of services across regions) generated churn in endpoints seen by Istio, and the sequencing and propagation via ADS became a performance bottleneck.

We created a dedicated service, with the sole function of propagating endpoints from ZooKeeper. This is minimal and so cannot do safe update sequencing (which we have to solve elsewhere), but it means that performance is very good, and we can comfortably propagate endpoints and meet our SLO.

How can we get Istio configured Envoy proxies to consume this stream? With an EnvoyFilter of course. For each service in the mesh we create an EnvoyFilter which overwrites the ADS configuration for endpoints, and it looks something like:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ads-override-for-my-service
  namespace: istio
spec:
  workloadSelector: { ... }
  configPatches:
    - applyTo: CLUSTER
      match:
        cluster:
          service: my-service.env.cluster
        context: SIDECAR_OUTBOUND
      patch:
        operation: MERGE
        value:
          eds_cluster_config:
            eds_config:
              api_config_source:
                api_type: GRPC
                grpc_services:
                  - envoy_grpc:
                      cluster_name: your-eds-stream.cluster
                transport_api_version: V3
            service_name: my-service.env.cluster

Creating the EDS service and configuring the Envoy bootstrap to include the static EDS service's cluster is left as an exercise for the reader.

This setup, as a whole, is not necessary for a normal Istio deployment. Istio is very good in a multi-cluster setup where Istio clusters can send traffic to each other. Getting a small number of Istio clusters to discover and route traffic for a single large network is where performance will degrade.

Rate limits and retries

We use open source metrics/logs collectors, and these have retry behaviours that cannot be configured. While we are waiting on upstreaming patches to improve this, we can partake in the tradition of using networking components to fix software problems.

The use-case is simple: when the collector is itself being rate limited (receives a 429 from the storage engine), we should not retry. The collector has a buffer of telemetry it is sending, so data will not be lost, just sent in the next tick.

When the collector receives an error (ie HTTP 500, with a specific header value), we should retry this request.

This can be done using Lua, which is embedded in Envoy. Although this does have a performance impact.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: fiddle-with-otel-retries
  namespace: opentelemetry
spec:
  workloadSelector: { ... }
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        routeConfiguration:
          vhost:
            domainName: my-otel.service
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.lua
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
            default_source_code:
              inline_string: |
                function envoy_on_response(response_handle)
                  local status = tonumber(response_handle:headers():get(":status"))

                  -- reject telemetry instead of retries
                  if status == 429 then
                    -- avoid a 5xx status code because this is not a failure
                    -- and a 5xx would show up in the SLO metrics
                    response_handle:headers():replace(":status", "428")

                  -- force retries when database returned 500 status code
                  elseif status == 500 then
                    -- a 502 status code will be retried
                    response_handle:headers():replace(":status", "502")
                  end
                end

This behaviour used to be a custom patch, and updating the fork with latest code changes was toil. Now this behaviour can be customised while the upstream packages are used directly. This behaviour can be extended to additional systems without new packages.