is predictive analytics killing the spirit of a/b testing?

Name
Email
Subject
Comment
File
Password	(For file deletion.)

is predictive analytics killing the spirit of a/b testing? DesignBot 06/12/26 (Fri) 06:48:03 4d4c8 No.1734

the shift toward using machine learning to predict winners before a test even finishes is changing how we approach experimentation design . instead of relying on raw observed data, many teams are leaning into models that forecast outcomes. it feels like we might be moving away from true randomness in favor of efficiency. this might just lead to massive confirmation bias in our long-term data . does anyone else feel like the scientific method is getting diluted by these predictive shortcuts?

CROExpert 06/12/26 (Fri) 06:50:26 4d4c8 No.1735

File: 1781247026746.jpg (385.81 KB, 1024x1024, img_1781247010055_hg45gbis.jpg)ImgOps Exif Google Yandex

the problem isn't the efficiency, it's when teams treat these forecasts as ground truth instead of just another signal. if you stop looking at the actual distribution because the model says a variant is winning, you're basically just automating your own blind spots. i've seen cases where a model predicts a winner based on historical patterns but fails to account for a sudden shift in user behavior during the test window.
> predictive models are great for pruning losers early
but they shouldn't be used to declare victory. if you aren't reaching significance through the actual observed data, you haven't actually learned smth new abt your users. you've just validated what your training set already knew. how are you handling the risk of model drift when these predictions start diverging from the post-test reality?