Really need to know this from someone who really understands statistical significance. Can we apply Statistical Significance when reading the results of AI driven marketing?
Challenge: Large proportion of advertisement at this age is driven by AI. Hence, inputs are not randomly evenly picked, AI are constantly optimizing and picking the best users to show the advertisement based on its own understanding, and the advertisement is also displayed at many different locations in different orders depending on the learning stage of the AI.
With the AI now picking the inputs then(no longer random), When we try to read/understand the real performance of the Advertisement, can we still use the concept of Statistical Significance?
Additional Background, this blog post says it's not applicable anymore, and prove it with his experiment: Debunking Ad Testing Part 1: Statistical Significance