CatOps
5.06K subscribers
94 photos
5 videos
19 files
2.66K links
DevOps and other issues by Yurii Rochniak (@grem1in) - SRE @ Preply && Maksym Vlasov (@MaxymVlasov) - Engineer @ Star. Opinions on our own.

We do not post ads including event announcements. Please, do not bother us with such requests!
Download Telegram
Unless you're super diligent with deprecation, you may be in a situation right now, when you need to migrate away from NGINX ingress.

Here's a great article that explains new Kubernetes API objects related to the GatewayAPI project that is here to replace Ingress.

Ingress API is not deprecated itself, but it won't be further developed either.

This article confuses the names for the community-led Ingress Nginx and the F5 NGINX ingress controller, but so do many of us: there are way too many nginx's in this world.

#kubernetes #networking #nginx
👍21
​​A case study from Amazon, how science solves actual engineering problems that later translate in money savings (likely millions on the Amazon scale).

How a Slack shout-out, a dusted-off academic theory, and a spaghetti monster led an AWS team to crack an elusive code—and deliver greater reliability and performance for customers is a story about AWS realigning their network around the random graph theory.

P.S. I always feel excited about the networking stories, because I studied them in the university. Even though I haven’t worked closely with them since many years ago, and forgot almost everything about them.

#aws #networking
👍91🔥1
​​How Do You Fit a Trillion-Parameter Model Into a Kubernetes Cluster? is an interesting article about how one should change their perspective when reasoning about running LLMs in Kubernetes compared to usual web apps.

It’s an interesting read, especially, if you don’t work with this stuff every day. The biggest takeaway here is that in the case of models, a “replica” doesn’t mean a pod in most of the cases, it’s a distributed system on its own that should behave as one. This article also explains, how exactly things are distributed within a replica, and what are the low level system parameters to pay attention to.

#kubernetes #ai #llm
👍32
​​For today's Donations Monday, I'd like to share with you requisites of a friend of mine, who volunteers for AFU since the beginning of the full-scale invasion.

Here's a page with all the possible ways to donate. You can also find links to the current goals, and reports for previous fundraises there.

Here's their Monobank jar that supports Apple Pay, if you'd better have a direct link:

https://send.monobank.ua/jar/BQjWbpver

#donations #Ukraine
3
​​How much do amd64 microarchitecture levels help in Go? is a benchmarking article that shows the compute time improvements you can get if you'd build your apps for modern x64 processors only. You likely use modern processors already and do not plan to run your apps on the decade old hardware.

Still, it's important to remember that while such articles are nice; your real applications probably don't just calculate bit vectors all day. It's much more likely your real performance bottleneck is I/O and not the fact that your apps are built with the support for old hardware. Still, you can get some easy wins here by just adding a compilation flag, if you're using Go.

#performance #go #programming
👍1
A Reddit thread with some useful tools for Kubernetes and kubectl plugins0.

Some things there are well-known, but you may find some new interesting things there. I did :)

#kubernetes
👍1
​​For today's Donations Monday, I'd like to share with you a fundraiser that our friends at DOU started for the 2nd separate corps of the National Guard of Ukraine «Хартія». The goal of this fundraiser is to buy heavy bomber drones "Vampire" for the Kupiansk direction.

Monobank jar: https://send.monobank.ua/jar/26mrQPQ3PZ

#donations #Ukraine
3
I will post AI-related articles this week, because why not?

The first one is from Charity Majors called AI demands more engineering discipline. Not less, in which she follows up on her another article.

This one is on technical aspects of moving to the disposable code. It also has a lot of links to other articles, which is also cool.

#ai
👍1
​​Harness engineering for coding agent users is a new guest article in Martin Fowler's blog that summarizes approaches to improve AI output and make it more manageable.

If you're actively using AI agents day-to-day, things described in this article won't be news to you, but it helps to structure one's thoughts.

#ai
2👎1
Continuing with our AI week.

AI in SRE: What's Actually Coming in 2026 is telling a story of AI coming for help with incident response.

The article suggests trying an AI tool for real investigation or data collection for postmortems. To clarify this, in my experience, you don’t need to have a dedicated tool, a general purpose AI agent with some harness (skills and scripts) would do. You should try it! AI does the job of data gathering incredibly well. Yet, the results are indeed not perfect.

Another good point in this article is data quality. AI results are as good as context you provide. I witnessed two prominent failure modes so far:

1. Inference on incomplete data: a person with limited access (typically a developer) asks their agent to investigate an alert. The agent comes to some conclusion. At the same time, a person with elevated access (typically a systems engineer) asks their agent to investigate the same alert and gets a different result, likely because some data is only available via kubectl events, etc. The fix for that is not to allow everyone to do everything, the fix is to revisit your observability pipelines and ensure that you ship all the relevant data, which is easier said than done.
2. Agent that cries "wolves": if you have a pollutant in your logs, or simply an event that happens very often, agents like to correlate it with everything. If your clusters are elastic, an agent could blame node count fluctuations for every error. The problem here is that once node count fluctuation actually causes a problem, you will be the one to ignore this hint from an agent, because it suggests it every single time.

If you are ready to share more AI failure modes specifically related to SRE in Ukrainian, welcome to our chat.

#ai #sre
👍2🤡1
So, that's for AI in the companies, but what about AI in the wild i.e. in open source?

We have cases like curl, that had to take down their bug bounty program due to the influx of slop bug reports. Yet, the industry adapts.

Here's a study by Redmonk on the stance of various foundations and standalone open source projects on AI, including their major concerns, and openness to AI-generated contributions.

#ai #open_source
🔥1