Forwarded from KubeFM
Media is too big
VIEW IN TELEGRAM
Tanat Lokejaroenlarb, Staff Site Reliability Engineer at Adevinta, explains how his team conducted experiments to measure the impact of node rebuilds on Service Level Objectives (SLOs) during Kubernetes upgrades.
He compares two approaches: a faster but more disruptive non-serialized strategy (rebuilding all nodes simultaneously) versus a slower but safer serialized approach (rebuilding one node pool at a time). Tanat shares how they identified affected components like ingress controllers and DNS, then fine-tuned configurations with proper graceful shutdown periods and Pod Disruption Budgets to minimize service impact. While Karpenter wasn't mature enough during these initial experiments, Tanat mentions they've since fully adopted it to improve their upgrade process.
Watch the full episode: https://kube.fmhttps://ku.bz/VVHFfXGl_
He compares two approaches: a faster but more disruptive non-serialized strategy (rebuilding all nodes simultaneously) versus a slower but safer serialized approach (rebuilding one node pool at a time). Tanat shares how they identified affected components like ingress controllers and DNS, then fine-tuned configurations with proper graceful shutdown periods and Pod Disruption Budgets to minimize service impact. While Karpenter wasn't mature enough during these initial experiments, Tanat mentions they've since fully adopted it to improve their upgrade process.
Watch the full episode: https://kube.fmhttps://ku.bz/VVHFfXGl_
π1
Forwarded from Kube Builders
This case study explains how cURL 65 errors and DNS resolution failures on AWS EKS were caused by Linux kernel network limits being exceeded, resolved by increasing
More: https://ku.bz/VMSf7zX6P
netdev_budget, netdev_budget_usecs, and netdev_max_backlog parameters.More: https://ku.bz/VMSf7zX6P
This case study shows how a single RAG chunk size change collapsed vLLM prefix-cache hit rate from 85% to 4%, triggering an 80% GPU replica increase while latency stayed flat.
It also includes the fix: adding a two-phase cache replay gate in CI.
More: https://ku.bz/7hB5K3Wkn
It also includes the fix: adding a two-phase cache replay gate in CI.
More: https://ku.bz/7hB5K3Wkn
We published a Kubernetes production-readiness checklist for teams preparing workloads for production.
The checklist is designed to help platform and application teams review the Kubernetes-specific behavior that affects an application before it goes live.
It includes:
- An interactive checklist
- Detailed explanations for each production-readiness check
- A downloadable PDF worksheet
It walks through five areas:
- The contract between your application and Kubernetes
- The manifests that define how Kubernetes should run it
- The workload security posture
- Scaling behavior under load
- Operational checks after launch
Open the checklist:
https://learnkube.com/production-best-practices
If you want a guided review, LearnKube also offers a Kubernetes Production Readiness Review with one of our instructors:
https://learnkube.com/production-readiness-review
The checklist is designed to help platform and application teams review the Kubernetes-specific behavior that affects an application before it goes live.
It includes:
- An interactive checklist
- Detailed explanations for each production-readiness check
- A downloadable PDF worksheet
It walks through five areas:
- The contract between your application and Kubernetes
- The manifests that define how Kubernetes should run it
- The workload security posture
- Scaling behavior under load
- Operational checks after launch
Open the checklist:
https://learnkube.com/production-best-practices
If you want a guided review, LearnKube also offers a Kubernetes Production Readiness Review with one of our instructors:
https://learnkube.com/production-readiness-review
Forwarded from KubeFM
Media is too big
VIEW IN TELEGRAM
Developers do not need to become Kubernetes experts to ship good systems.
Joel Vasallo argues for a middle ground: application engineers should understand the basics, like pods and CrashLoopBackOffs, but platform teams should not make Kubernetes knowledge a hiring gate. The platform should guide people toward good defaults instead of forcing every developer to learn every abstraction layer.
Watch the full interview: https://ku.bz/zTPZwj-__
Joel Vasallo argues for a middle ground: application engineers should understand the basics, like pods and CrashLoopBackOffs, but platform teams should not make Kubernetes knowledge a hiring gate. The platform should guide people toward good defaults instead of forcing every developer to learn every abstraction layer.
Watch the full interview: https://ku.bz/zTPZwj-__
Valkey Operator is a Kubernetes operator that automates deployment and lifecycle management of Valkey clusters and instances with features like automated installation and configuration management.
More: https://ku.bz/M2q9_T15T
More: https://ku.bz/M2q9_T15T
Forwarded from KubeFM
Media is too big
VIEW IN TELEGRAM
Brian Stack from Render explains why Kubernetes scaling can break along a dimension most teams ignore: namespaces.
At Render scale, hundreds of thousands of namespaces made common DaemonSet patterns expensive. Calico and Vector were list-watching namespace data across every node, multiplying memory usage and putting pressure on the API server during restarts and rollouts.
You will learn:
- Why namespaces can become a hidden scaling bottleneck
- How DaemonSets multiply memory and control-plane pressure
- How profiling, staging clusters, and upstream collaboration freed 7 TiB
- Why pushing from an 80% fix to a complete fix can make teams faster
Watch (or listen to) it here: https://ku.bz/0mrvCsXrV
π This episode is brought to you by LearnKube β comprehensive Kubernetes training. https://learnkube.com/training
With @Birthmarkb
At Render scale, hundreds of thousands of namespaces made common DaemonSet patterns expensive. Calico and Vector were list-watching namespace data across every node, multiplying memory usage and putting pressure on the API server during restarts and rollouts.
You will learn:
- Why namespaces can become a hidden scaling bottleneck
- How DaemonSets multiply memory and control-plane pressure
- How profiling, staging clusters, and upstream collaboration freed 7 TiB
- Why pushing from an 80% fix to a complete fix can make teams faster
Watch (or listen to) it here: https://ku.bz/0mrvCsXrV
π This episode is brought to you by LearnKube β comprehensive Kubernetes training. https://learnkube.com/training
With @Birthmarkb
This article explains that BuildKit is not just Dockerβs build engine but a general-purpose framework that can turn custom frontend definitions into images, tarballs, local artifacts, and package outputs.
More: https://ku.bz/mLZp6ZLys
More: https://ku.bz/mLZp6ZLys
π1
This week on Learn Kubernetes Weekly 183:
π₯ Autoscaling Hid Our LLM Cost Regression (85% β 4% Cache Hit Rate)
π₯ Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
ποΈ DocumentDB on Kubernetes: Resilient, Highly Available Databases with Automatic Failover
π‘οΈ We Brought Skew Protection to Your Kubernetes
π Keeping Your Security Model Intact When Running VMs in Kubernetes
Read it now: https://kube.today/issues/183
βοΈ This newsletter is brought to you by LearnKube β master Kubernetes with hands-on training designed for engineers who want to learn the smart way https://ku.bz/hypSbyc-V
π₯ Autoscaling Hid Our LLM Cost Regression (85% β 4% Cache Hit Rate)
π₯ Mount Mayhem at Netflix: Scaling Containers on Modern CPUs
ποΈ DocumentDB on Kubernetes: Resilient, Highly Available Databases with Automatic Failover
π‘οΈ We Brought Skew Protection to Your Kubernetes
π Keeping Your Security Model Intact When Running VMs in Kubernetes
Read it now: https://kube.today/issues/183
βοΈ This newsletter is brought to you by LearnKube β master Kubernetes with hands-on training designed for engineers who want to learn the smart way https://ku.bz/hypSbyc-V
k10s is a terminal dashboard for watching multiple Kubernetes clusters at once, with side-by-side views, health signals, warnings, and recent logs in one screen.
More: https://ku.bz/CpbrdbBG0
More: https://ku.bz/CpbrdbBG0
Forwarded from Kube Architect
This case study shows how a team traced repeated pod OOM kills in ASP.NET Core to native memory growth from zombie SignalR connections, glibc fragmentation, and kernel socket buffers.
More: https://ku.bz/0KH7ncbBR
More: https://ku.bz/0KH7ncbBR
Forwarded from KubeFM
This media is not supported in your browser
VIEW IN TELEGRAM
William Morgan, CEO at Buoyant, addresses how teams should approach cross-zone traffic costs in Kubernetes clusters.
He emphasizes that when finance teams flag these expenses as problematic, teams should evaluate two key solutions: HAZL (High Availability Zonal Load Balancing) and Topology-aware routing. His main message is that these costs are avoidable β teams can implement cost-saving measures while maintaining reliability by choosing the right solution for their specific use case.
Watch the full episode: https://ku.bz/CBwn51pl-
He emphasizes that when finance teams flag these expenses as problematic, teams should evaluate two key solutions: HAZL (High Availability Zonal Load Balancing) and Topology-aware routing. His main message is that these costs are avoidable β teams can implement cost-saving measures while maintaining reliability by choosing the right solution for their specific use case.
Watch the full episode: https://ku.bz/CBwn51pl-
This article explains five Ingress-NGINX behaviors that can break migrations, including path-matching differences, regex quirks, rewrite behavior, and annotation mismatches when migrating to another ingress solution.
More: https://ku.bz/KXFB4qzH6
More: https://ku.bz/KXFB4qzH6
Forwarded from KubeFM
This media is not supported in your browser
VIEW IN TELEGRAM
Most infrastructure visibility is still fragmented across tools and devices.
Phil Trickovic describes the before-and-after: instead of stitching together disconnected views, teams can trace workload consumption from creation through delivery, even across large endpoint fleets.
That shift is what makes cost and resource tracking practical.
Watch the announcement: https://ku.bz/fNT9T1FZZ
Read the announcement: https://ku.bz/Y-Hr_rhZk
Phil Trickovic describes the before-and-after: instead of stitching together disconnected views, teams can trace workload consumption from creation through delivery, even across large endpoint fleets.
That shift is what makes cost and resource tracking practical.
Watch the announcement: https://ku.bz/fNT9T1FZZ
Read the announcement: https://ku.bz/Y-Hr_rhZk
eksup analyzes your EKS cluster and generates a step-by-step upgrade playbook, flagging deprecated APIs, add-on version mismatches, and node group issues before you upgrade.
More: https://ku.bz/2B_b2k4F4
More: https://ku.bz/2B_b2k4F4
π New on LearnKube: "Kubelet Metrics: How cAdvisor and CRI Collect Kubernetes Stats."
Kubernetes metrics often look like a Prometheus topic, but the data originates much lower in the stack.
This guide explains how kubelet collects and exposes pod, container, node, and resource metrics, and how that path changes when stats move from cAdvisor to the container runtime through CRI.
You will learn:
- how Linux cgroups provide the raw counters behind container metrics
- where cAdvisor fits inside kubelet
- what kubelet exposes through /metrics, /metrics/cadvisor, /metrics/resource, and /stats/summary
- how containerd and CRI-O can return pod and container stats through CRI
- why the same kubelet endpoint can hide a different internal collection path
Read the full article:
https://learnkube.com/kubernetes-metrics-cadvisor-kubelet-cri
Kubernetes metrics often look like a Prometheus topic, but the data originates much lower in the stack.
This guide explains how kubelet collects and exposes pod, container, node, and resource metrics, and how that path changes when stats move from cAdvisor to the container runtime through CRI.
You will learn:
- how Linux cgroups provide the raw counters behind container metrics
- where cAdvisor fits inside kubelet
- what kubelet exposes through /metrics, /metrics/cadvisor, /metrics/resource, and /stats/summary
- how containerd and CRI-O can return pod and container stats through CRI
- why the same kubelet endpoint can hide a different internal collection path
Read the full article:
https://learnkube.com/kubernetes-metrics-cadvisor-kubelet-cri
β€3
Forwarded from LearnKube news
π New on LearnKube: "Kubelet Metrics: How cAdvisor and CRI Collect Kubernetes Stats."
Kubernetes metrics often look like a Prometheus topic, but the data originates much lower in the stack.
This guide explains how kubelet collects and exposes pod, container, node, and resource metrics, and how that path changes when stats move from cAdvisor to the container runtime through CRI.
You will learn:
- how Linux cgroups provide the raw counters behind container metrics
- where cAdvisor fits inside kubelet
- what kubelet exposes through /metrics, /metrics/cadvisor, /metrics/resource, and /stats/summary
- how containerd and CRI-O can return pod and container stats through CRI
- why the same kubelet endpoint can hide a different internal collection path
Read the full article:
https://learnkube.com/kubernetes-metrics-cadvisor-kubelet-cri
Kubernetes metrics often look like a Prometheus topic, but the data originates much lower in the stack.
This guide explains how kubelet collects and exposes pod, container, node, and resource metrics, and how that path changes when stats move from cAdvisor to the container runtime through CRI.
You will learn:
- how Linux cgroups provide the raw counters behind container metrics
- where cAdvisor fits inside kubelet
- what kubelet exposes through /metrics, /metrics/cadvisor, /metrics/resource, and /stats/summary
- how containerd and CRI-O can return pod and container stats through CRI
- why the same kubelet endpoint can hide a different internal collection path
Read the full article:
https://learnkube.com/kubernetes-metrics-cadvisor-kubelet-cri
Forwarded from KubeFM
This media is not supported in your browser
VIEW IN TELEGRAM
Artem Lajko shares how to prevent wasted resources in observability implementations using a label-based approach.
The model introduces three tiers: low tier (Prometheus + Grafana), medium (adding Loki and alerting), and high (including Jaeger tracing). Teams can progress through the tiers as their needs and capabilities grow, with GitOps and ArgoCD managing the dynamic deployment of tools.
Watch the full episode: https://ku.bz/9sGxhmm8s
The model introduces three tiers: low tier (Prometheus + Grafana), medium (adding Loki and alerting), and high (including Jaeger tracing). Teams can progress through the tiers as their needs and capabilities grow, with GitOps and ArgoCD managing the dynamic deployment of tools.
Watch the full episode: https://ku.bz/9sGxhmm8s
This article introduces ctx_, a CLI tool that switches an entire DevOps working context at once, including Kubernetes context, cloud credentials, environment variables, VPN, SSH tunnels, secrets, and browser profile.
More: https://ku.bz/-BG8_C5W2
More: https://ku.bz/-BG8_C5W2
β€1π1π₯1
Forwarded from KubeFM
Media is too big
VIEW IN TELEGRAM
John Ford from Scout24 SE explains how Scout24 turned a forced OS migration into a chance to rethink Kubernetes autoscaling, node provisioning, and infrastructure efficiency.
You will learn:
- Why two-minute node provisioning forced a 25% capacity buffer
- How Karpenter made the Bottlerocket migration safer
- What broke around EC2 metadata, AWS SDKs, and cgroups
- How the new foundation enables Spot, ARM, and GPU workloads
Watch (or listen to) it here: https://ku.bz/DdmVC2_7v
π This episode is brought to you by LearnKube β get started on your Kubernetes journey through comprehensive online, in-person or remote training: https://learnkube.com/training
With @Birthmarkb
You will learn:
- Why two-minute node provisioning forced a 25% capacity buffer
- How Karpenter made the Bottlerocket migration safer
- What broke around EC2 metadata, AWS SDKs, and cgroups
- How the new foundation enables Spot, ARM, and GPU workloads
Watch (or listen to) it here: https://ku.bz/DdmVC2_7v
π This episode is brought to you by LearnKube β get started on your Kubernetes journey through comprehensive online, in-person or remote training: https://learnkube.com/training
With @Birthmarkb
Cluster Agent Swarm Skills is a collection of specialized AI agent skills for Kubernetes and OpenShift operations, covering cluster management, GitOps, security, observability, incident response, and platform workflow orchestration.
More: https://ku.bz/n9K3N9JBq
More: https://ku.bz/n9K3N9JBq