Saw this breakdown on the math behind testing multiple metrics at once. It's pretty wild how easy it is to accidentally hit a 46% chance of a false positive just by running four variants against three different success metrics. If you're looking at twelve separate comparisons and still using that standard 5% threshold, you aren't actually testing anything. The Bonferroni correction helps fix the family-wise error rate, but it makes the significance bar so much harder to hit. I'm always worried about being too strict and missing real wins, but seeing those numbers makes me realize how often people ship changes that don't actually do anything. It's basically a
math trap for anyone who doesn't account for their total number of tests.
article:
https://www.convert.com/blog/a-b-testing/bonferroni-test/