The Apollo | What Went Wrong | | Ep 189

Reducing Alert Fatigue in On-Call Rotations

2:06:50 – 2:14:307:40 long

Elena outlines actionable practices for managing engineering on-call rotations to prevent developer burnout.

In the final topic of this segment, Elena and Smac address the human element of software reliability by discussing on-call engineering rotations. Elena opens at 36:50 by stating that alert fatigue is one of the leading causes of voluntary engineering turnover. She explains that non-actionable notifications sent during night hours erode trust in monitoring systems and lead to delayed responses during real system outages.

Smac adds at 39:15 that alerts should strictly correspond to conditions requiring immediate human intervention. He argues that automated self-healing scripts or delayed morning notifications should handle non-critical warning thresholds. Elena supports this view, asserting that if an engineer cannot take a direct, concrete action upon receiving an alert, that notification should not trigger an urgent page.

The two conclude by discussing structural approaches to schedule rotations and team compensation at 42:10. Elena recommends implementing secondary backup roles on shifts and providing compensatory time off following high-incident shifts. Smac concurs, noting that sustainable operational health requires treating on-call duties as a formal, well-supported component of engineering work.

Watch the full episodeOpen on YouTube

More from this episode

2:20Tribute to Beloved Chat Moderator LSMac Wop pays respect to LS, a longtime moderator for The Apollo stream who passed away at the start of the year.9:58Debating SZA's Concert Mishap in MinneapolisMac Wop and Keem analyze viral footage of a background dancer almost dropping SZA during a performance.25:00Keem Breaks Studio Equipment During Bag BanterKeem questions Mac Wop's custom bag before accidentally snapping a studio microphone stand.35:00Comparing Streaming Culture on Kick and TwitchThe Apollo hosts debate the differences in content guidelines and creator culture between Kick and Twitch.43:40TF Showcases His Marathon Running MedalsTF enters the Apollo studio wearing medals earned from completing local marathons and 15k races.46:30Grid bottleneck risks and industrial load growthThe panel debates whether current power distribution networks can support massive increases in baseline electricity demands.53:20Regulatory friction and small modular reactor viabilityThe speakers explore the structural challenges and potential benefits of scaling small modular nuclear reactors.1:01:15Storage economics and the limits of intermittent powerThe conversation addresses battery technology limitations and the economic realities of solar and wind grid penetration.1:09:00Critical mineral supply chains and geopolitical riskThe group analyzes the geographic concentration of battery metals and the policy efforts to domesticate processing.1:35:12Evaluating Microservices Versus Monolithic ArchitectureElena argues that early startups frequently adopt microservices prematurely, while Smac highlights the operational overhead involved.2:19:15Debating Open Source versus Proprietary ModelsSmac and the guest explore whether open-source software provides better long-term security and reliability compared to proprietary solutions.2:34:10Evaluating AI Automation and Workforce ImpactsThe speakers analyze how artificial intelligence tools are reshaping entry-level software engineering roles and corporate hiring practices.3:05:30Trade-offs in Automated Quality AssuranceThe host and guest argue over whether end-to-end automated testing can entirely replace human quality assurance teams.3:21:40Preventing Burnout During High-Pressure ReleasesThe panel discusses leadership approaches to protecting developer well-being during intense project deadlines.3:55:30Evaluating the Open Weights Model vs Proprietary APIsThe host and guest discuss whether open-source AI models present a net benefit to society or introduce uncontrollable safety hazards.4:17:40Addressing Hardware Bottlenecks and Energy DemandsThe conversation turns to the physical infrastructure required to sustain rapid AI scaling, including microchip manufacturing and nuclear power.4:33:15The Sustainability Crisis in Open-Source InfrastructureSmac and Alex debate whether corporate sponsorship models ultimately distort open-source developer priorities. Alex argues that reliance on single-entity grants creates systemic vulnerabilities for critical tools.4:40:10Building Resilient Digital Media Systems for the Next DecadeThe discussion shifts to practical steps for independent media creators aiming to secure their archives. Smac proposes redundant local backups alongside open protocols to prevent platform lock-in.