CatOps
5.03K subscribers
94 photos
5 videos
19 files
2.61K links
DevOps and other issues by Yurii Rochniak (@grem1in) - SRE @ Preply && Maksym Vlasov (@MaxymVlasov) - Engineer @ Star. Opinions on our own.

We do not post ads including event announcements. Please, do not bother us with such requests!
Download Telegram
​​Support a friend of mine on the Frontline!

Last year, she chose the tough path: Combat Medic.
Now, she needs our help to secure critical medical supplies that can't wait for paperwork.

No donation is too small. Let’s help her save lives!

- Mono Jar: https://send.monobank.ua/jar/75jQXw6aYq
- Mono: 💳: 4874100025644306
- Privat: 💳: 5168745027810065

#donations #Ukraine
5
​​For those of you, who're into MySQL.

There's an open letter to Oracle to establish a foundation to take care of MySQL.

https://letter.3306-db.org/

You can subscribe to make your voice heard. I would say, it's an important thing to do, because, you know, community matters. Also, it doesn't require much work from your side at this point.

More information is available via that link above.

#databases #mysql
3
Collaboration sucks is a nice Friday evening read about the ways we work together.

I think, this article has interesting thoughts, but as usual, you need to use your own judgement to understand the environment you’re in.

For example:

You’re the driver” is a key value for us at PostHog. We aim to hire people who are great at their jobs and get out of their way. No deadlines, minimal coordination, and no managers telling you what to do.
In return, we ask for extraordinarily high ownership and the ability to get a lot done by _yourself._ Marketers ship code, salespeople answer technical questions without backup, and product engineers work across the stack


This works great until end up with the codebase that has a unique flavor of the same wheels at every corner. Sure, there are ways of dealing with that, but you have to have those constraints beforehand.

However, the idea of limiting your collaboration and inviting only the relevant people into the decision making process, makes total sense. If your company growths, at some point it will grow beyond the point a single person can understand every aspect of your system. When it happens, sharing proposals to everyone wouldn’t, indeed, improve your collaboration and the team spirit; it would just generate noise. And when there is too much noise, it’s easy to lose important signals.

#culture
👍4
Apparently, AWS had at least two recent outages due to AI. It was originally reported by Financial Times, but their article is behind a paywall. If you’re subscribed, you can read it here.

Me seeing these news surprisingly coincided with me seeing this post on Reddit: Vibe coders passing responsibility on code reviewers.

And this is kinda true, scary, and reassuring at the same time.

True because it’s indeed very easy these days to generate a lot of code in almost any language.

Scary, because the meme about 5000+ lines PRs with LGTM stands true. While AI code reviewers can quite effectively catch typos and style issues, that humans kinda suck in catching; overly complex logic is usually Ok for them. Thus, we will face more outages in the nearest future, in my opinion.

Reassuring, because it means that those of you who “keep the lights on” are not going anywhere because of AI. In fact, quite the opposite.

#ai
👍11🔥3😁32
For today's donations Monday, I'd like to share once again the standing Monobank jar for FPV equipment.

This jar is for the unit in which a guy from my wife's hometown serves.

https://send.monobank.ua/jar/4WLw91UqFe

#donations #Monday
👍1
An article from OpenAI on how they created a complete project without any human-written code.

This is, of course, kind of marketing material for OpenAI, but it also has interesting points:

 code throughput increased, our bottleneck became human QA capacity.


 management is one of the biggest challenges in making agents effective at large and complex tasks. One of the earliest lessons we learned was simple: give Codex a map, not a 1,000-page instruction manual.


 the agent’s point of view, anything it can’t access in-context while running effectively doesn’t exist. Knowledge that lives in Google Docs, chat threads, or people’s heads are not accessible to the system. Repository-local, versioned artifacts (e.g., code, markdown, schemas, executable plans) are all it can see.


And the most important point, in my opinion:

 kind of architecture you usually postpone until you have hundreds of engineers. With coding agents, it’s an early prerequisite: the constraints are what allows speed without decay or architectural drift.
...
In a human-first workflow, these rules might feel pedantic or constraining. With agents, they become multipliers: once encoded, they apply everywhere at once.


In any case, it's an interesting read. Obviously, it's all related to a completely green field project. So, your mileage for decade-old monoliths may vary.

P.S. Also, according to the diagrams in this article, OpenAI uses VictoriaMetrics, which is also cool :)

#ai #programming
👍5🔥2🤮2
Why etcd breaks at scale in Kubernetes is a great article that describes, what etcd is, how it works in nutshell, what are its limitations; and, most importantly, why does it all matter to Kubernetes.

I had an etcd cluster becoming read-only due to the backend quota. The fact that this happened on the 1st of January added insult to injury.

P.S. LearnKube is a great resource for all the Kubernetes related topics. I highly recommend reading their other articles as well, if you want to get more familiar with the technology!

#kubernetes #etcd
🔥11
​​For today’s Donations Monday, I’d like to ask you to donate the downing of russian UAVs.

This is a fundraiser by Come Back Alive and they support different payment methods!

#donations #Ukraine
8
An article from Alex Ewerlöf on how the Staff Engineer Archetypes can backfire.

Honestly, it’s not the best of his articles, but it’s a good illustration that you cannot “lock yourself” just on a single aspect of your work. Especially, once you’ve gotten high enough to levels such as a Staff Engineer.

#culture #career
🔥5👍1
A story from OpenAI on how they scale Postgres.

While this is an interesting read, and you can definitely borrow some ideas from there; I got an impression that OpenAI is moving towards Azure’s CosmosDB from Postgres.
If a new feature requires additional tables, they must be in alternative sharded systems such as Azure CosmosDB rather than PostgreSQL.
—-
While we’re happy with how far PostgreSQL has taken us, we continue to push its limits to ensure we have sufficient runway for future growth. We’ve already migrated the shardable write-heavy workloads to our sharded systems like CosmosDB. The remaining write-heavy workloads are more challenging to shard—we’re actively migrating those as well to further offload writes from the PostgreSQL primary.

Cosmos DB, from my understanding, is a document-oriented DB from Microsoft, similar to MongoDB.

#databases
6👍4
These days all the talks are about AI.

My (hypothetical) SRECon26 keynote is an article from Charity Majors with her advice on how SREs should approach AI.

BTW, I’ve heard good things about SRECon. I don’t know if the upcoming SRECon Americas is the same as SRECon in Dublin in October, but maybe I should check that one out!

#sre #ai
1👍1
4 ways to use Argo CD and Terraform together is an article on how to pass data between Terraform (or OpenTofu) and ArgoCD (or any other GitOps tool for that matter). For example, if you're creating a new infrastructure component and need to pass its endpoints to an app.


They pointed out quite explicitly, that you shouldn't just pass raw secrets around. However, they didn't mention any secret storage for whatever reason. For example, you can use vault_generic_secret resource to store credentials in Vault and then something like the External Secrets Operator to fetch them from there. I am not saying, you should, but you can.

P.S. This article was shared in our chat. Come join us! The chat is in Ukrainian, and it's usually fun!

#terraform #argocd #cd #kubernetes
👍13👎1
So, Amazon pushed back on the Financial Times report about AI causing outages. This is not news.

However, they now require a senior engineer's approval on the AI generated code pushed by non-senior engineers, apparently. It's not clear, who should review AI code generated by seniors, though.

- Financial Times (paywalled).
- Opinion on Xitter.
- Discussion on Hacker News.

This is an interesting twist on the whole AI adoption, and it would be very interesting to see where it would go from here. At least, at Amazon.

P.S. As a bonus, here's a book bundle about LLMs and some related stuff, so you could review AI-generated code more efficiently!

#ai #aws #culture
😁81
A colleague of mine wrote an article on using Cloudflare Tunnels to securely connect to your self-hosted things. It specifically covers quirks of connecting mobile apps, since not all of them can handle auth redirects correctly.

This is a nice read if you have a home lab or anything self-hosted. However, you can also use Cloudflare Tunnels for your business cases, like exposing your staging backend to test mobile devices, etc.

#security #cloudflare
🔥8👎1
​​For today’s Donations Monday, I’d like to remind you about the UA Responders foundation that raises money for the rehabilitation of Ukrainian veterans.

#donations #Ukraine
🫡2
I guess many of you are familiar with the concept of OKRs - Objective-Key-Results. OKRs have been around for quite some time. So, of course, there is a book about it.

Here is a short summary of this book by someone on the internet, alongside with their rating and recommendation for whom this book may be interesting.

Now, you can easily generate a book summary using AI these days. The summary itself is not the reason I want to share it with you. I think, writing such summaries is a great way of conceptualizing books for yourself in the first place. I keep telling myself, I should do this as well. Unfortunately, I am lazy :\

#books #okr #management
4
A former colleague of mine wrote an article on how to write better tests with AI.

I recall, there were debates, what should a human write: tests or the implementation. Now, there are debates on whether a human should open their IDE at all.

This article is front-end focused, but it has some actionable and more or less universal advice on how to make AI do tests better. At the end of the day, AI is just another tool and the whole trick is in how good do you apply it.

#ai #programming
3