​​Постмортем о недавнем инциденте AWS.

Там же описаны некоторые зависимости внутренних сервисов компании.

#aws #postmortem

Recent Google incident post-mortem: https://status.cloud.google.com/incident/zall/20013#20013004

tl;dr: wrong quota applied to the Google User ID Service

#postmortem

​​Sometimes it is worth getting your head from clouds down to Earth.

Here is a great post-mortem story of a failed Ceph cluster.

The investigation led them down to more “invisible” underlying layers rather than just Ceph itself, but I won’t spoil more. This is an interesting and not that long read, so you can go through it yourself. Also, at least for me, every post-mortem looks like a detective story, not just a technical article.

P.S. I haven’t worked much with Ceph myself. When I was a very junior engineer, we had a few small Ceph clusters in a company I worked for. I was not involved in that project, though. However, I remember that once we had an issue with one of the clusters and my colleague spent a night fixing it.

The next day he said: “We didn’t quite lose the data. We just cannot retrieve it”. I think from that time this became a strong association for me with Ceph, even though Ceph is usually not the case.

#postmortem #ceph #linux

Amazon has published a public postmortem for the recent issues on Friday. However, it went through a little bit unnoticed because of the Log4j story (see one of the previous posts).

So, the original issue is happened to be a cascading failure, which led to congestion in AWS internal networks. This is an interesting part, because it puts some light on AWS internals.

So, the internal monitoring system as well as parts of control plane for EC2 reside in the internal network, which experienced issues. That's why AWS team was operating with partial visibility of their systems, which impacted the speed of resolution.

Customer services were still running, but their control APIs were impacted. For example, your existing EC2 machines were there, but you could neither describe them, not start a new one. These matters happened to be more critical for certain services within AWS line API Gateways and Amazon Connect.

The interesting thing is that these events were caused by the code that was there for years (according to AWS). Unfortunately, an unexpected behavior was revealed during an automated scaling event.

To mitigate such issues in future, AWS switched off automatic scaling in us-east-1. They claim that they have enough capacity already. As well as they're working on a fix for the part of code that caused the co congestion in the first place. I assume, there are many other internal action items from this outage as well.

#aws #postmortem

If you haven’t read a Roblox’s postmortem on October‘s 73-hour outage, you definitely should!

Even though this event happened in October, the postmortem was released just a few days ago. And in this case, this is a very good decision! Especially, because this write up provides a detailed analysis on what happened at that time and what chain of events caused that.

It‘s cool to read postmortem the next day after an outage - we are all curious human beings. Unfortunately, those postmortems usually they are lacking many details. This is understandable: it‘s not enough time for a thorough analysis, also your team is probably already tired.

In this case, though, you can have a detailed overview of what happened as well as plans to prevent this chain of events happening again. Moreover, with some plans already implemented.

It‘s a pity that not may companies do similar postmortems. And I must say that this is probably in their disadvantage either. After reading this document I have a feeling that Roblox is a cool place to work, TBH.

#postmortem #hashicorp #consul

On Thursday, November 18, 2021, Dropbox did not go down. It sounds like a beginning of some modernist novel. Yet, this is rather a post-mortem on an incident that never happened.

This is a great story of leadership and dedication, which lasted for a few years. Results? Dropbox were able to literally pull the network cord from their data center and “do not go down on 18 of November, 2021”. And not just a data center, but their main one.

I really enjoyed this story because it proves once again a few basic things that we prefer not to think about:
- Any project take time and effort. Big projects take a lot of time and effort. If you’re looking into rebuilding your system from scratch, that won’t take a week or two.
- Big projects require dedication. You cannot just add them as a side hassle for your existing team and expect them to deliver everything with the highest quality.
- Iterative improvements. Apollo 11 was the first mission to reach the Moon not because 11 is a pretty number.
- Test and exercise. It’s not enough just to “implement the best practices”, you have to validate if those are actually working as expected. And if there’s a process involved, you have to repeat it frequently enough to not to get rusty.

#infrastructure #culture #postmortem

I love reading postmortems. A good postmortem usually unveils a set of problems some of which you can have in your company as well. As they say: there is never a single root cause.

Here is a postmortem from Reddit about their Pi-day outage.

It has everything you love: complex systems, legacy software, processes that were not tested that well, sacred knowledge that is long gone, etc.

Don’t get me wrong, I’m saying that not to shame Reddit. In fact they did a great job highlighting all the problems. It’s much harder and takes more courage than just say: Calico broke - Calico bad.

Also, I have similar problems at my place as well and I bet you have too. This why it’s important to recognize the importance of such “low priority tech debt”. Cleaning that out may save your company’s ass someday.

#kubernetes #networking #postmortem

Avery nice read for the weekend - Postmortem of a 2005 Flickr Outage Modernized for Today.

It’s cool because it’s not just a postmortem from a well-known, but this piece also contains some history! Trust me, you will like the root cause of this one ;)

#postmortem #history

Fixing a memory leak in Kubernetes with a single line of code.

I love such debug stories because you can read them almost as a thriller or a detective literature. Also, this case is very interesting indeed and it took that team a couple of months to figure it out.

A takeaway for myself:
- It‘s useful to have a dashboard (and potentially a monitor) that shows the difference of the node memory usage and the memory usage by pods.

#kubernetes #postmortem

Resend had a 12 hour outage on the 21st of February.

tl;dr:
> The database migration accidentally deleted data from production servers…
> … we performed a database migration command locally, but it incorrectly pointed to the production environment instead…

You can read it in more details is the article, but here are some of the action items from this postmortem:

- No accessible user role should have write privileges on the production database.
- Improve local development to reduce risks related to database migrations.
- Create redundancy to preserve sending function even during a database outage.
- Increase cadence for disaster recovery tests.
- Implement incident banner on Resend dashboard to inform users quickly.

So, I dunno, check your database. Maybe, you have such a risk as well.

Also, it’s kinda strange that people rarely talk about network isolation not only between their production and non-production environments, but also between their local environment and production. Make production access conscious. Put it on a separate role/VPN. Add some friction accessing it.

Moreover, for the love of god, validate your DB backups.

#postmortem #databases

Yet another explainer of what has happened to CrowdStrike on July 19th and more importantly, how.

tl;dr: config changes. Config changes can be dangerous too. Despite there were successful deploys between the update of CrowdStrike Scanner and the outage, it seems like a new type of config was deployed which caused the entire clusterfuck.

This line is also interesting:

 June 4th, Red Hat released a KB relating to kernel panics that were caused by the Crowdstrike sensor 
process. This was a bug in the Linux kernel itself, that the sensor was 
triggering and wasn’t Crowdstrike’s fault. However it does prove that config that has passed the Content Validator can cause kernel panics.

UPD: I think the most important take-away here is not what caused the outage or how the deployment process at CrowdStrike looks like. It's the fact that problems can be obscure enough. When something goes wrong big times, it's easy to "blame" a "big thing": the whole deployment process, or code quality, or people behind the software. This is much more comforting than the idea that any small change can cause a butterfly-effect and take your whole system down. This was true for CrowdStrike and this is true for you as well.


#postmortem #crowdstrike #windows

Breaking down OpenAI's outage: How to avoid a hidden DNS dependency in Kubernetes - in this article, folks at Render analyze the recent OpenAI outage and provide some insights on the possible cause of the issue.

This article also serves as a reminder that you'd better run DNS and the Control Plane on separate nodes and also that the data in ETCD is organized by namespace, thus listing "all" resources is very costly, especially in large clusters, since you need to scan everything.

#kubernetes #postmortem

Spotify has released a postmortem for their outage that happened on 16th of April, and was almost global.

In nutshell, it was a combination of a bug, and a cascading issue caused by user retries. Here's an interesting bit:

> This change was deemed low risk and as such we applied it to all regions at the same time.

This is something what burned a lot of engineers. So, the take-away is probably never consider any change low-risk, especially if you already have the architecture for gradual rollouts. However, it's much easier to be said than done.

#postmortem #sre

Who would win: a null pointer crash loop or a multi-billion dollar corporation?

And here we have a postmortem of the recent Google’s outage.

P.S. Kudos to Google for releasing this postmortem so quickly!

#postmortem

A postmortem from Cloudflare for yesterday’s outage is now available.

tl;dr:
>>>
The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind.Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.
<<<

Another interesting thing:
>>>
Unrelated to this incident, we were and are currently migrating our customer traffic to a new version of our proxy service, internally known as FL2. Both versions were affected by the issue, although the impact observed was different.
Customers deployed on the new FL2 proxy engine, observed HTTP 5xx errors. Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero. Customers that had rules deployed to block bots would have seen large numbers of false positives. Customers who were not using our bot score in their rules did not see any impact.
<<<

So, if you were not affected yesterday, you know why now.

#postmortem #cloudflare

​​I don't know, when is the point, where we can all collectively agree that front-end frameworks have gone too far in their complexity.

Yet, here you are with the Cloudflare preliminary postmortem:

>>>
A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today.
<<<

https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

#cloudflare #postmortem

At least Cloudflare is fast in sharing their postmortems.

https://blog.cloudflare.com/5-december-2025-outage/

A curious thing is this:

>>>
Customers that have their web assets served by our older FL1 proxy AND had the Cloudflare Managed Ruleset deployed were impacted. All requests for websites in this state returned an HTTP 500 error, with the small exception of some test endpoints such as /cdn-cgi/trace.
<<<

IIRC, in the previous incident on Nov 18, only the customers on the newer proxy version were impacted. So, one could say that Cloudflare had a single time-distributed total outage.

Another important thing:

>>>
Before the end of next week we will publish a detailed breakdown of all the resiliency projects underway, including the ones listed above. While that work is underway, we are locking down all changes to our network in order to ensure we have better mitigation and rollback systems before we begin again.
<<<

Honestly, looking forward to seeing the write-up. I can only imagine how stressed their team is after taking down a big chunk of the Internet twice in less than 30 days.


#cloudflare #postmortem

Starting a new year with a postmortem, eh?

There was a prolonged incident with Kafka at Honeycomb last month. Here you can find a preliminary postmortem for this incident.

"Preliminary" means that there is no root cause analysis yet, but there's already the timeline and the remediation steps.

#postmortem

A post from Cloudflare about a low-level race condition they tracked down and fixed in the Rust Hyper library.

I like reading such detective stories. Also, I recall times, when people would regularly ask about strace on the interviews. I am not sure if this is still the case. At least, I wasn’t asked about strace for a long time.

#programming #postmortem

A postmortem for the recent GitHub incident. Regarding the cause:

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery. 

#postmortem #github