Automation goes brrr (2026 update)
Written on 2026-10-30
This is an update post covering 18 months since the original post.
Cluster growth
The sidecar manager is now running in the base configuration of our Kubernetes clusters, and is deployed on over 100 clusters.
Patches
Between Q4 2025 and the end of Q3 2026 the updates-per-day metric grew from 2M to just under 4M updates per day, which ends up being 8M image patches per day at peak.
The human-authored changes (ie which matchers run and which patches should be applied by the automation) are at least 1 per calendar day.
Features added
- Ability to remove/replace container configurations (rather than just patching)
- Support for Kubernetes native sidecar (initContainer that persists)
- Improved matching to distinguish patches depending on the business criticality of the workload, and other business attributes/identifiers
Use-cases demonstrated
- Sidecar migration (replace old service mesh sidecar with Istio sidecar), percentage-based rollout across the whole fleet, service by service, by criticality
- Security patching: being able to ensure all pods with a vulnerable sidecar version were patched within 48h (>100k pods).
Validation of the approach
The core design of the sidecar manager using an embedded expression language has kept up with the changing/increasing organisational requirements. Using an expression language for matching running workloads allows for precise targeting of patches, which has proven that fleet-wide patching can be done safely at scale.
The fleet-wide patching has improved the security posture of the Kubernetes estate, which is increasingly important given the step change in CVEs over the last 2 quarters due to AI-enabled research. This is also a validation of the sidecar container model, versus embedded libraries, because sidecar containers are more legible to the platform operators, and are easier to "patch" during pod replacements.