Site Reliability Engineering (SRE) Guide: SRE vs DevOps, SLOs, and Error Budgets
Site Reliability Engineering (SRE) is what happens when you ask a software engineer to design an operations function. Coined by Google in 2003, SRE applies software engineering principles — automation, measurement, iterative improvement — to infrastructure and operations. It is not a rebranding of DevOps. It is a specific implementation of DevOps principles with concrete practices, metrics, and organizational structures.
SRE vs DevOps
| Aspect | DevOps | SRE |
|---|---|---|
| Origin | Cultural movement | Google engineering practice |
| Focus | Breaking silos, collaboration | Reliability as a feature |
| Measurement | Deployment frequency, lead time | SLOs, error budgets |
| Key practice | CI/CD, IaC, monitoring | Error budgets, toil reduction |
| Team structure | Embedded in dev teams | Dedicated SRE team |
| Philosophy | “Everyone is responsible for ops” | “Software engineers do ops, using software” |
They are complementary. DevOps is the philosophy. SRE is one way to implement it.
SLIs, SLOs, and SLAs
SLI — Service Level Indicator
A quantitative measure of a service’s behavior. The raw metric.
| SLI Type | Formula | Example |
|---|---|---|
| Availability | successful_requests / total_requests |
99.95% of requests return non-5xx |
| Latency | requests_under_threshold / total_requests |
95% of requests complete in < 300ms |
| Throughput | requests_processed / time_period |
10,000 requests per second |
| Error rate | error_requests / total_requests |
< 0.1% of requests return errors |
SLO — Service Level Objective
The target for an SLI. This is the reliability target your team commits to.
SLO: 99.9% of requests to the payment API will return
successfully (non-5xx) over a 30-day rolling window.
SLA — Service Level Agreement
A contract with consequences (usually financial) for missing the SLO. SLAs are for customers. SLOs are for internal engineering.
Key insight: Your SLO should be stricter than your SLA. If your SLA is 99.9%, your SLO should be 99.95%. This gives you a buffer.
Error Budgets
The error budget is the amount of unreliability you can tolerate.
Error Budget = 1 - SLO
If SLO = 99.9%:
Error Budget = 0.1% = 43.8 minutes of downtime per month
= 8.76 hours per year
How Error Budgets Work
- Budget remaining: Ship features. Deploy aggressively. Take risks.
- Budget exhausted: Freeze feature releases. Focus on reliability work only.
This creates a self-regulating system:
- Developers want to ship features → they need error budget → they care about reliability
- SREs want reliability → they allow feature releases when budget is available → they support velocity
Error Budget Policy
When error budget is:
> 50% remaining → Ship freely, deploy daily
25-50% remaining → Deploy with extra testing, no risky changes
< 25% remaining → Deploy only reliability fixes
0% remaining → Feature freeze until budget replenishes
Toil: The Enemy of SRE
Toil is manual, repetitive, automatable work that scales linearly with service growth and has no enduring value.
Examples of toil:
- Manually restarting pods that crash weekly
- Hand-creating user accounts
- Copy-pasting the same Terraform for every new environment
- Manually checking certificate expiration dates
Not toil:
- Incident response (unpredictable, requires judgment)
- Writing automation (enduring value)
- Architecture reviews (strategic value)
The 50% rule: SREs should spend no more than 50% of their time on toil. The rest goes to engineering work that reduces future toil.
Measuring Toil
Track time spent on toil weekly:
| Week | Toil Hours | Engineering Hours | Toil % |
|---|---|---|---|
| 1 | 25 | 15 | 63% ⚠️ |
| 4 | 20 | 20 | 50% |
| 8 | 12 | 28 | 30% ✅ |
| 12 | 8 | 32 | 20% ✅ |
Incident Management
Incident Lifecycle
Detection → Triage → Mitigation → Resolution → Post-mortem
Severity Levels
| Level | Impact | Response |
|---|---|---|
| SEV1 | Service down, revenue impact | All hands, 15-min response |
| SEV2 | Degraded performance, partial outage | On-call + backup, 30-min response |
| SEV3 | Minor impact, workaround available | On-call, next business day |
| SEV4 | No user impact, internal only | Track in backlog |
Blameless Post-Mortems
Every SEV1 and SEV2 incident gets a post-mortem within 48 hours.
Template:
- Summary: What happened, when, impact.
- Timeline: Minute-by-minute events.
- Root cause: The underlying issue (not “human error”).
- Contributing factors: What made it worse.
- Action items: Concrete fixes with owners and deadlines.
- Lessons learned: What we will do differently.
Critical rule: No blame. If a human made a mistake, the question is “Why did our system allow this mistake to happen?” — not “Who made the mistake?”
On-Call
Sustainable On-Call
- Minimum 2 people in rotation. No single point of failure.
- 8-12 hour shifts, not 24-hour shifts.
- Compensation: Time off after on-call shifts, or additional pay.
- Escalation path: If the on-call engineer cannot resolve in 30 minutes, escalate.
- Runbooks: Every alert has a runbook with diagnosis steps and resolution procedures.
- Target: < 2 pages per 12-hour shift. More than that means your system needs reliability work.
Building an SRE Practice
Start Small
You do not need a dedicated SRE team to start. Adopt SRE practices incrementally.
Month 1-2: Define SLIs and SLOs for your most critical service.
Month 3-4: Implement error budget tracking. Dashboard the error budget burn rate.
Month 5-6: Introduce blameless post-mortems for all incidents.
Month 7-8: Measure toil. Set a target to reduce it by 20%.
Month 9-12: Formalize on-call rotation. Create runbooks for every alert.
SRE Team Models
| Model | Structure | Best For |
|---|---|---|
| Embedded | SRE sits within a dev team | Small org, 1-3 SREs |
| Dedicated | Separate SRE team | Medium org, 4-10 SREs |
| Consulting | SRE reviews and advises | Large org, SRE expertise as a service |
Conclusion
SRE is not about preventing all failures — it is about measuring reliability, setting targets, and making data-driven trade-offs between reliability and velocity. Error budgets give teams a shared language: “We have budget, so let us ship.” Or “We are out of budget, so let us fix reliability.” Start by defining SLOs for your most important service, track the error budget, and run blameless post-mortems. The rest follows naturally.
Related posts:
