0
$\begingroup$

I'm running an A/B test.

Let’s say we’re measuring the following:

number of links clicked per user

This means that we’ll have a sample point for each of our users. This can make it hard to get statistical significance since the number of sample points is bounded by the number of users we have.

I’ve seen folks conduct experiments with a modified measurement, like this:

number of links clicked per user per day

The per day modification leads to many more sample points. The longer you run the experiment, the more sample points you’ll get. This makes it easier to obtain statistical significance.

However, when analyzing the data, I saw that a t-test requires sample points to be independent of each other. Since we’re using multiple sample points from a given user, doesn’t that invalidate this requirement?

Does the per day modification above break the validity of the t-test?

$\endgroup$
1
$\begingroup$

It probably does break the assumptions of the t-test. On the other hand, it is not clear that is easier to achieve significance this way, because the standard deviation of the result will also increase and make the result less significant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.