A few weeks after publishing the 1M token/s post, I spent a weekend helping my good friend Milko Ilari set up vLLM on his shiny new DGX Spark with Gemma 4. My first in-person reaction was “It’s Champagne” (from the old days of PC Perspective) The Spark is a wild little machine — 128 GB of unified memory in a box you can hold with one hand, running the same Blackwell architecture as the datacenter B200s. But its memory bandwidth is 273 GB/s. The B200s in our cluster do 8,000 GB/s. Almost 30x less.

Watching the numbers on that tiny machine got me thinking. The benchmark I ran on GKE Autopilot with 96 B200 GPUs had reported 4.4% FLOPS utilization. 10.9% memory bandwidth. Tensor cores active 1.5% of the time. The GPUs looked almost idle while pushing a million tokens per second. Was something wrong?

No. And honestly, figuring out why turned out to be more interesting than the benchmark itself.

That first post covers the journey — every optimization, and many failure 🫠. This one covers the physics.

GKE pod IP exhaustion is one of the few failure modes that gives you no warning before it goes terminal. I recently stepped into a war room where a client’s primary scaling group had flatlined — workloads cordoned, deployments stuck in Pending, and the estimated cost of the stall nearing $15k per hour in lost transaction volume. The culprit wasn’t traffic. It was a /20 subnet that had quietly run out of address space, and a set of GKE allocation defaults nobody had questioned at design time.

For the last decade, Kubernetes has been the backdrop to most of my work: operating clusters, helping build hosted Kubernetes, and writing Kubernetes operators. At PlanetScale, that now means running stateful systems like Postgres and MySQL in production. Kubernetes has many faces, but here I want to talk about one face only: why it is so good at running workloads at scale.

People ask me what an operator actually does. The canonical answer is: "it reconciles desired state." This is correct, but it also tells you almost nothing.

An operator is a feedback controller. It's the same closed loop that runs a thermostat or keeps your car at a fixed speed on cruise control. In our case, the thing being controlled is a database. I have been building these loops for years, and the best way I know to make them click is to ignore Kubernetes at the beginning. Kubernetes is full of control theory, even if we don't call it that in the day-to-day.

Before we look at a single line of Kubernetes, we're going to run a production database by hand and slowly let the feedback loop appear on its own. Then we'll map that loop to Kubernetes, with the pieces production needs: a store, watches, queues, retries, and more. At the end, we'll look at what one of these loops looks like in a real operator.

This article documents a real production investigation into latency variance in Python-based microservices running on Kubernetes, revealing how CPU throttling amplifies GIL contention into unpredictable response time spikes.

KubeUser is a kubernetes-native operator that turns user management into a declarative code experience. No more manual certificate juggling — just apply a YAML file, and the operator handles the rest.

There is a lot of noise right now about letting AI "fix" your infrastructure. When production is acting up, you need to maintain a complete mental model of the system. If you let the AI be the driving force, you lose the overview.

I've spent the last few months working on what should have been a solved problem: letting applications running in our on-premises Kubernetes clusters access Google Cloud services. What I found instead was an industry-wide workaround culture built on security anti-patterns, and a surprisingly elegant solution hiding in plain sight.

Moving from Slurm to Kubernetes doesn't have to mean losing the workflow you know. Here's how SkyPilot brings Slurm-like simplicity to K8s.

In cloud environments this problem is usually solved by a managed load balancer. In a self-hosted Proxmox setup we need an equivalent component that sits on the boundary between the public network and the private cluster network. Here, HAProxy plays that role.

Thankfully on Kubernetes, updates to the ephemeral containers feature and the addition of the --profile and --target options to kubectl debug have greatly simplified this process. There is now a fairly easy means of creating a privileged process context that shares a pid namespace with a specific target container.

Luxury Yacht is a cross-platform GUI desktop app for managing Kubernetes clusters and resources.

It's a plugin for kubectl that gives you a UNIX find-like experience. Find resource based on name regex, age, labels, status, node name, restarts, image name, or a jq filter — and then print, patch, annotate or delete any.

The Operator is designed to unlock seamless and effortless distributed performance testing in the cloud and enable continuous integration for CI/CD. By design, the entire system is cloud native and focuses on automation and CI practices. One strong feature about the system is its ability to horizontally scale to meet any required performance demands.

Most of what's been written about blameless post-incident reviews is about managers not blaming engineers, and engineers not blaming each other, because blame shuts down learning. What many miss is that engineers still blame themselves, and the damage is the same.

Every GCP resource and API have quotas. In a big organization, you can start having production incidents due to hitting quotas you didn't know about in projects you have never touched before.