This is a continuation of a series called Tales from the Ops Side. Back in November 2016, a seemingly innocuous piece of code that has been running for years starts throwing a very strange and nonsensical exception. This strange exception seemed to be thrown completely at random, could not be reproduced in any other environment, and would also self-heal! This talk tells the story of a baffling bug hunt spread over many weeks, involved all members of the company, including the CEO and CTO, and lead us deep into the guts of CPython and countless dead-ends.
Hany Fahim is the Founder and CEO of VM Farms, a 24×7 Operations Team and Cloud Provider. Hany has over 10 years of operations experience, both in building and supporting private and public cloud platforms, as well as Linux and OSS environments. You can reach Hany at @iHandroid and the folks of VM Farms @vmfarms.