Loadingā¦
0
Almost lost a client because of a false positive alert at 2am
I was sitting in my home office in Austin last Tuesday when our SIEM flagged a login from Russia on a server that's been decommissioned for 6 months. Spent an hour trying to trace it before realizing the alert rule never got updated after the migration. Anyone else dealing with alert fatigue from rules that nobody cleans up?
2 comments
Log in to join the discussion
Log In2 Comments
barnes.jamie3mo ago
About a year ago I had a false positive from a stale rule at 3am for a server that was literally unplugged in a closet. It took me two hours to drive to the office and check it myself before I realized the rule was pointing at an IP that hadn't been used in months. The fix that actually worked for us was setting up a quarterly rule review session with the team where we go through every alert and kill the ones tied to old infrastructure. We put it on the calendar as a recurring thing and it cut our false positives by like 60 percent in the first round. It is a boring task but way better than getting woken up for a ghost server.
10
susan3503mo ago
Eh, 60 percent is a bit generous, I'd say closer to half from our own quarterly cleanups.
3