• 5 Posts
  • 90 Comments
Joined 3 years ago
cake
Cake day: June 15th, 2023

help-circle








  • I’m a system administrator for a relatively big SaaS for a relatively big company.

    I was attempting to configure a new inbound email protocol in the TEST environment. I needed to reference how it was configured in PROD so I could copy certain settings into TEST, so I had both up on my screens (you can see where this is going). But first I needed to clear out the old cloned configurations in TEST.

    …And not even half a second after I deleted those config files did I realize what I’d done: deleted inbound email configuration on PRODUCTION. In the middle of the day. Killed inbound email for the system for the whole company.

    The whole reason I had PROD open was because I didn’t have good documentation of how it was configured (what was cloned into TEST was old and not correct). So now I was seriously fucked. I had to redo the config from scratch live in PROD. Even tried backing out the deletion but that failed.

    Had to advise my boss of the fuckup. He was new. Had only been with the company for like 2 weeks. I had to assure him that I am NOT actually a total fuckhead and I usually don’t do stupid shit like this. I felt terrible, because I knew he’d have to give the post-failure analysis review on this to his new bosses, and that would definitely be awkward.

    Anyway, he was cool with it. Assured me that he’d done something similar at a previous job. He let me do my thing to fix it, which I did. Inbound email was only down for …well, a long time (3-4 hours), because once I was able to get the config working, I had to re-register all of the inbound email addresses to use it -which had its own sordid story. That took some time.

    After the fact, we modified the TEST environment to have a different “theme” (header colors) so it wasn’t identical to PROD on-screen, and I documented the email configuration thoroughly.