This oncall cycle, I observed my first real hardware failure. A machine critical to the deployment pipeline spontaneously shutdown and failed to reboot, problems with mounting RAID, etc. #fail