I'm running an A/B test.
Let’s say we’re measuring the following:
number of links clicked per user
This means that we’ll have a sample point for each of our users. This can make it hard to get statistical significance since the number of sample points is bounded by the number of users we have.
I’ve seen folks conduct experiments with a modified measurement, like this:
number of links clicked per user per day
The per day modification leads to many more sample points. The longer you run the experiment, the more sample points you’ll get. This makes it easier to obtain statistical significance.
However, when analyzing the data, I saw that a t-test requires sample points to be independent of each other. Since we’re using multiple sample points from a given user, doesn’t that invalidate this requirement?
Does the per day modification above break the validity of the t-test?