Work Culture & Behind the Scenes

How Google SRE Handles Incidents: IC, CL and OL Roles and Blameless Postmortems

Google SRE's incident management guide sets out the full approach to service disruptions, from alerts and on-call to role assignment during a crisis and blameless postmortems.

By Oemah Solution Editorial 5 min read

This article was prepared with AI assistance from the official sources listed below, and checked automatically before publication.

Service disruptions are bound to happen in any sufficiently complex system. This applies equally to e-commerce applications and to digital public services run by government agencies. The problem is that many organisations still handle disruptions in an ad hoc way. Everyone panics, it is unclear who is in charge, and once the service is restored, the focus turns to finding who is at fault. Google's Site Reliability Engineering (SRE) team published the Incident Management Guide, which covers a complete process for preventing that kind of chaos. The guide was written by Adam Crume, Alex Cepoi, Chelsea Granados, Roxana Loza, Steve McGhee, Svetlana Gites, Trevor Mattson-Hamilton, and Vrai Stacey.

What Google SRE-Style Incident Management Is

According to the guide, a well-defined plan helps teams in three ways: reducing the impact of disruptions on users, coordinating the response so that mitigation is faster, and learning from events so they do not recur. The process is divided into three main stages:

  • Preparation: reliable alerting mechanisms, on-call rotations, and playbooks.
  • Response: clear roles while an incident is under way.
  • Recovery and learning: blameless postmortems.

How the Practice Works

1. The right alerts. Google SRE lists several characteristics of a good alert:

  • it fires in time, before the impact on users grows;
  • it covers all the important functions that users rely on;
  • it is based on symptoms that users experience, not on the internal behaviour of the system;
  • it is actionable, because an alert that cannot be acted on is just noise.

The guide recommends alerts based on SLOs (Service Level Objectives) as a good way to meet the first three characteristics. Preventive alerts based on internal metrics are still needed in certain cases, for example when a resource quota is nearly exhausted. However, the general rule is to avoid alerts tied to implementation details.

2. Playbooks and exercises. An up-to-date playbook contains instructions for debugging and mitigating problems, so that responses can be much faster. A playbook is only useful if the on-call engineer knows what is in it. For this reason Google runs a regular exercise called "Wheel of Misfortune". The exercise keeps the team's knowledge fresh and gives less experienced engineers a safe space to hone their skills.

3. Roles during an incident. Google's incident response system, IMAG, is based on the Incident Command System (ICS), the emergency response standard in the United States for events such as wildfires or earthquakes. Its focus is on the "3Cs": coordinate, communicate, and control. There are three main roles:

  • Incident Commander (IC) coordinates the overall response.
  • Communications Lead (CL) provides regular updates to stakeholders and acts as the point of contact for incoming communications.
  • Operations Lead (OL) focuses on mitigating and resolving the technical problem.

These roles do not follow the organisational hierarchy. Assignments are based on knowledge and the context of the incident.

4. Blameless postmortems. After an incident, the team writes a postmortem that focuses on improving systems, procedures, and training, not on individual mistakes. Completion targets for follow-up actions are agreed with stakeholders in the form of SLOs. After that, the follow-up actions go into the team's backlog and are prioritised alongside feature work.

Common Pitfalls

  • Focusing only on the technical fix. Google SRE stresses that keeping users and leadership informed is just as important as technical mitigation. Updates should cover what is affected, how severe it is, any workarounds available, and the estimated time to recovery.
  • Alerts based on internal metrics. Alerts like these do not reflect real impact on users and break easily when the system's implementation changes.
  • Playbooks that are never practised. A document that the on-call engineer does not know will not help in a crisis.
  • Letting the response run on its own. The guide warns that chaos will take over if the response is not actively managed. Incident response needs to be treated as a project in its own right, complete with planning, deciding who is involved, and documenting actions.

How Teams and Clients Can Apply It

This pattern is relevant not only to giant companies. Small teams and government agencies can also start with simple steps:

  1. Identify a few key functions that users rely on, then set up alerts based on the symptoms they experience.
  2. Write short playbooks for the most frequent disruptions, then practise them regularly through simulations.
  3. Agree in advance on who can serve as IC, CL, and OL. In a small team, one person may hold more than one role, but the separation of duties remains clear.
  4. If you are a service owner or client, ask for regular updates during an incident instead of pressing the technical team directly, because that communication channel is exactly the function of the CL.
  5. Make blameless postmortems a habit, and make sure follow-up actions actually enter the backlog and are given priority, not merely recorded.

With clear roles, alerts that reflect the user experience, and a culture of learning without looking for scapegoats, every disruption can become material for improvement rather than a source of finger-pointing.

#incident management#sre#postmortem#on-call#slo

Sources & references

This article is a summary and interpretation by the Oemah Solution team of the sources below. Refer to the original sources for full details and context.

  1. Google SRE - Learn sre incident management and response Google Site Reliability Engineering (sre.google) · sre.google