CatOps
5.1K subscribers
94 photos
5 videos
19 files
2.7K links
DevOps and other issues by Yurii Rochniak (@grem1in) - SRE @ Preply && Maksym Vlasov (@MaxymVlasov) - Engineer @ Star. Opinions on our own.

We do not post ads including event announcements. Please, do not bother us with such requests!
Download Telegram
​​​​We continue with a fundraiser a pickup truck for the 423rd battalion.

Right now, one jar has 218/350k raised, and the other one 37/50k.

Jar 1:

https://send.monobank.ua/jar/2aMqtZT592

Jar 2:

https://send.monobank.ua/jar/8oTyqJUjPV

#donations #Ukraine
5
Being a good communicator is becoming more and more important in the world, where more and more code is generated.

15 principles for managing up is an article with some tips that can help you not only get on a better foot with your manager, but also to become a better colleague in general.

I personally so not like the term “managing up”, but I guess you have to call it somehow.

#culture #mgmt
👍4
You likely already know about the recent security incident that involves OpenAI and Hugging Face, when a model escaped its containment to cheat an evaluation test and hacked Hugging Face in the process. Now, postmortems are available from both OpenAI and Hugging Face.

- Initial incident disclosure by Hugging Face
- Postmortem by Hugging Face
- Postmortem by OpenAI

This is quite interesting read. On one hand, in the words of Hugging Face: "AI-driven offensive tooling is no longer theoretical. It lowers the cost of running a broad, patient, multi-stage campaign, and it operates at machine speed. Defending an online platform now means treating the data and model surface as a first-class attack surface, and using AI on defense to keep pace."

But also: "...we had no admission policy rejecting privileged or hostPath pods, and the CSI driver's ClusterRole granted pod creation cluster-wide..."

This highlights not only the need to develop new defense methods and strategies, but also the fact that the good-old multi-layer security approach is important, and old boring things work as well.

#ai #security
👍72
An interesting article by Coroot on reproducing split brain on CloudNativePG.

Regardless of whether or not you use CloudNativePG, it's a good example of chaos testing. Chaos testing is about an experiment: here they define a clear gap in certainty (what would happen if?) and tested it.

The article also has some recommendations on CloudNativePG configuration, if you're using it.

#databases #postgres #kubernetes
👍1🔥1
​​For today's Donations Monday, I'd like to remind you about a big fundraiser by DOU for the 2nd separate corps of the National Guard of Ukraine «Хартія».

The goal of this fundraiser is to buy heavy bomber drones "Vampire" for the Kupiansk direction.

Monobank jar: https://send.monobank.ua/jar/26mrQPQ3PZ

#donations #Ukraine
6
An article from Anthropic on the AI agents evaluation strategies, basically test strategies for AI agents.

It's an interesting read, even if you do not create your own agents. There are also some interesting points in the difference of evaluating a coding agent vs a communication agent (customer support, etc.), as well as examples of the metrics you can use for that.

Besides, you can use the ideas from this article not only for the agents, but also for skills, if you develop any.

#ai #testing
👍3
An interesting article that explains how Karpenter works.

It does have flaws, though. Apart from useless footnotes, it doesn’t elaborate on how did they address the behavior, only some abstract suggestions, some of which kinda contradict what Karpenter actually is.

Spoiler: their fix is a separate node pool. You can learn it from this thread on Reddit.

Otherwise, it’s a good article and a good thread.

#kubernetes #aws #karpenter
4👍1
Two bundles in one post.

- Hands-on DevOps bundle - these are courses, so if you buy them, make sure to claim them before 2027-09-01!
- Software Engineering books by Manning. Just check if you have those already, because some of these books were parts of other bundles.

#books
1
An article that asks an important question: what happens when you are given a vibe-coded project by a manager and how to deal with it?

This article is partially behind a paywall, but the question is important nevertheless. Moreover, I feel like in the platform field it's even more visible, since you are handed not only projects by your managers, but also vibe-coded projects by developers.

Here, two things become important. First, "partial work has no value", and the second is that a handover is a handover - it doesn't matter who or what created the code, all the existing production-readiness checks should be there.

#ai #culture
👍3
Argo For Kubernetes: From Argo CD to Workflows and Image Updater | A complete Guide is a nice article about building a complete CD workflow with Argo products in a home lab.

I would still not call it a "complete guide", because it's only scoped for local development, but it's a nice entry point, if you want to learn more about Argo on a home server, or create a small PoC.

#cicd #kubernetes #argo
🤝2
The Hidden Cost of Postgres Constraints at Scale is a description of a bottleneck foreign keys can create in a write-heavy PostgreSQL table. It also has recommendations of what you can do to overcome that.

Now, I would say, it's a corner case for many people. The author works in a company that develops their own time series solution based on Postgres, so for them, it makes sense.

For the vast majority of people, I would argue, it makes more sense to offload the integrity checks to the database, rather than trying to implement those in-house. Of course, it may be different for you, but no one knows your system better than you do.

#db #postgres
👍2
A postmortem for the recent GitHub incident. Regarding the cause:

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery. 


#postmortem #github
👍12
University of Colorado published an article about a study of 7700 employees that shows that Remote workers report the highest well-being.

The study simply analyzed the self-reported data without any judgments on why exactly this is the case, but one of the authors points out that:

The study did not explore why remote workers reported higher well-being, but Johnson points to a growing body of research on autonomy and flexibility.

One explanation is that remote workers have greater control over their work setup and daily schedule, she said.

"If you have control over your environment, you tend to have more positive outcomes," she said.

Working from home can also eliminate many everyday stressors.

"Spending a lot of time in traffic is negatively related to well-being," Johnson said. "There are so many little stressors associated with being in the office."

Those stressors can include arranging child care, hiring help for pets or managing the logistics of getting to and from work, she said.

Johnson said the study points to a broader lesson for employers navigating return-to-office debates. Rather than focusing only on where employees work, organizations may get better results by investing in employee well-being.


I know that this has nothing to do with the immediate topic of systems / platform / reliability engineering, but I still find this article amusing, since we all are doing our jobs in one way or another: in an office or remotely.

Now, you also have a study to point to, if anyone brings up the topic of the remote work and well-being. However, it's important to keep in mind that the employee well-being is not necessary the metric that is being optimized when the RTO directives are issued.

#culture
🔥4🤗1
​​Happy Independence Day of Ukraine! 💛💙

It always starts with small steps, but every step counts. So today, I'd like to get back to small fundraisers that also matter!

Here, a friend of mine is raising funds for a pickup track for the Zaporizhzhia direction.

You can find more info in this Instagram post (in Ukrainian).

You can donate to this Monobank jar:
https://send.monobank.ua/jar/8Mjotj4Uei

#donations #Ukraine
🎉151