If you serve LLMs on Kubernetes without inference-aware routing, your load balancer is likely wasting inference capacity. Generic HTTP traffic management blindly routes requests, assuming the backends in your cluster are interchangeable. But your model-serving backends are stateful and unevenly prepared to handle any given request. As a result, requests are often routed to the backend that’s not the one best suited to respond.

Migrating to Gateway API gives you a more capable foundation for traffic management and opens the door to inference-aware routing. The Kubernetes Gateway API’s Inference Extension routes requests based on backend serving state, which tends to make better use of cluster capacity and reduce request latency.

In this post, we’ll look at how the Inference Extension works, the routing strategies it enables, and the signals you can use to monitor whether inference-aware routing is behaving as intended in production.

Spontaneous swarming of responders might seem like a nuisance that breaks our tidy mental models of incident response, but it's actually very powerful.

We investigated why firmware updates were causing our core servers to take four hours to reboot.

How Airbnb built a Kubernetes sidecar to deliver dynamic configuration reliably at scale.

Gamedays are one of the most effective ways we proactively uncover gaps in our systems and processes. At Datadog, we regularly run a variety of gamedays to intentionally stress our platforms and learn how our systems and teams respond under real-world conditions. These exercises help us surface hidden vulnerabilities, strengthen our operational readiness, and continually raise the bar for our infrastructure.

During one such gameday, a simulated zonal failure introduced targeted disruptions in an availability zone on a staging environment by inducing network latency, which exposed a weakness in our PostgreSQL architecture. Several of our Kubernetes-based PostgreSQL clusters had primary or writer nodes running in the affected availability zone. As network latency spiked, those primaries could no longer communicate reliably with their replicas. Replication lag quickly grew, writes stalled, and applications began serving stale data. Because no replica was sufficiently up to date, failover wasn’t safe and the clusters were effectively stuck.

We rely on PostgreSQL as the backend database for many Datadog products, and this architecture has served us well under normal conditions. But the gameday revealed an uncomfortable truth: In the face of certain network failures, our setup prioritized availability over durability in ways that left us with no safe recovery path.

In practice, this meant the primary continued accepting writes even while replication to replicas was delayed due to elevated network latency. The system remained writable, but replication lag continued to grow, and replicas drifted further behind the primary. As a result, failover candidates could no longer be promoted safely without risking data loss. We were left with only one viable option: wait for latency to subside and for replicas to catch up.

We set out to fix this failure mode. Our goal was to make failover both automatic and safe, without compromising PostgreSQL’s performance characteristics more than necessary. To do this, we rearchitected our PostgreSQL deployment to use synchronous replication for failover candidates, coordinated by Patroni, an open source high-availability manager.

In this post, we’ll walk through how we redesigned our Kubernetes-based PostgreSQL clusters for failover safety, how we balanced durability against latency, and what we learned while validating this approach through benchmarking and failure testing.

A command-line tool for querying databases

Distributed terminal-native issue tracker backed by Git.

kage (影, "shadow") clones a website into a folder you can browse offline, with every script stripped out. It opens each page in real headless Chrome, waits for the page to settle, snapshots the DOM a human would have seen, then deletes all the JavaScript and pulls the CSS, images, and fonts down to local paths. What lands on disk looks like the live site and runs no code.

There’s a particular kind of infrastructure bug that’s deeply annoying to debug: the kind that only appears when your cluster is under load. Everything runs fine in steady state. Then you push a release, or traffic spikes and your autoscaler kicks in, and suddenly you’re chasing 502s that resolve on their own a few minutes later. Your first instinct is to blame the app. Then the deployment. Then the cluster. Then yourself.

We hit exactly this pattern at Simon AI. The culprit turned out to be something fundamental about how nginx handles DNS — and fixing it properly required waiting for a feature that was finally open-sourced at the end of 2024.

Here’s what we found, what we tried, and how we solved it.

A practical guide to building the technical-financial baseline that tells you whether autoscaling will actually save money — and how much.

How we built an intelligent 'finally task' that turns 170,000 lines of logs into a 10-line diagnosis in under 30 seconds.

How treating Service Level Objectives as declarative infrastructure changed the way I think about reliability.

In this article we deploy ArgoCD in a bare-metal Kubernetes cluster built on Proxmox.

Image Trust with Cosign and Kyverno

In this guide, I'll explain how to collect Prometheus metrics from an application running in a Kubernetes cluster and push the data to a Prometheus instance on a VM in a different geographic region using Prometheus's remote write endpoint.

Docker-compatible container CLI built on Apple's Containerization framework. Same commands, same flags — mocker run, ps, stop, build, compose, stats — all working on macOS 26.

eBPF egress firewall for Kubernetes

Cluster API is a Kubernetes subproject focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.

Started by the Kubernetes Special Interest Group (SIG) Cluster Lifecycle, the Cluster API project uses Kubernetes-style APIs and patterns to automate cluster lifecycle management for platform operators. The supporting infrastructure, like virtual machines, networks, load balancers, and VPCs, as well as the Kubernetes cluster configuration are all defined in the same way that application developers operate deploying and managing their workloads. This enables consistent and repeatable cluster deployments across a wide variety of infrastructure environments.

A few years ago, Roland Huss and I wrote Kubernetes Patterns, a catalogue of repeatable solutions for building cloud-native applications on Kubernetes. The book covers patterns across six categories: foundational, behavioral, structural, configuration, security, and advanced, everything from health probes and init containers to controllers, operators, and elastic scaling. Recently, Roland and Daniele Zonca co-authored a new book, Generative AI on Kubernetes, which covers the operational side of running LLMs on Kubernetes. I was a reviewer on that book, and the more I read through it, the more I recognized patterns from our original work, just applied to a very different class of workload. Deployments, StatefulSets, Init Containers, DaemonSets. They were all there. Just with bigger numbers.

Here is a high-level summary of how Kubernetes patterns apply to LLM workloads, which Roland and I will cover in more detail in our upcoming KubeCon + CloudNativeCon Europe 2026 talk in Amsterdam.