Paired t-test with multiple observations per pair

Question

I have a bunch of websites websiteA....websiteZ for example. Now I have an optimization technique that possibly improves the performance of these websites. I want to check whether this technique actually significantly effects the performance. So my dependent variable is apply_technique with the treatments enabled and disabled.

So for each website I measure the performance with and without the optimization technique applied. So I get results like this:

website | technique disabled | technique enabled
websiteA      20 seconds          17 seconds
.....
websiteZ      45 seconds          39 seconds

etc.

However, to account for possible fluctations I measured each (website, treatment) combination 5 times so for websiteA I have 5 measurements with the technique disabled and 5 with the technique enabled. This then results in 26 websites * 2 treatments * 5 repetitions = 260 measurements.

My question is, when I want to do a paired t-test do I first need to average the performance over these 5 trials or not? I might lose some information when I average it right?

Could I also decide to not average them? So I have 130 technique_disabled observations and 130 technique_enabled observations and I then simply use these to do a paired t-test? Would that be acceptable?

EDIT: I see that the performance differences (technique_enabled - technique_disabled) are not normally distributed. So I will probably use a Wilcoxon signed rank test. However here the same question applies? Should I average the observations over the 5 trials or not?

Answer 1

Averaging the data will result in a loss information and statistical power, so it is best avoided.

Since you have repeated measures for websites, you can account the differences between websites (or equivalently, the non-independence of observations within each website, since observations on one website are more likely to be similar to each other than to observations on other websites), by fitting random intercepts for website ID in a regression model (a mixed effects regression model). This would look something like:

 apply_technique ~ treatment + (1 | website_ID)

and you could fit such a model using lmer from the lme4 package. This way you will make maximal use of the data.

Answer 2

If each group has the same number of members, then the mean of the group means is the same as the mean over all the observations. The sd can be more complicated, depending on what you're taking as your null hypothesis. One of the simplest null hypotheses is that each observation is equal to $\mu(W)+\epsilon$, where $\mu$ is some function of the website (that is, $\mu: \text {set of websites} \rightarrow \mathbb R$), and $\epsilon$ is normally distributed with zero mean and sd that is independent of the website. A more complicated null hypothesis would have the sd of $\epsilon$ depend on the website (there is no need to consider the possibility that the mean is nonzero or depends on the website, as that is absorbed into $\mu(W)$).

If your null hypothesis has constant $\sigma$, then $s$ should be calculated on the set of observations as a whole. However, if your null doesn't have a single $\sigma$, then you won't be able to calculate a single $s$. You'll have to calculate a different $s$ and thus $p$ for each website, then use some method of combining them. For this to be rigorous, you should decide on the combination method before collecting any data.

Now, if you reject this null, then you are rejecting the hypothesis that the statistic is normally distributed and the mean for the two conditions is the same. So the null being false means that the means are different or the statistic isn't normally distributed. If you think, looking at the data, that the statistic isn't normally distributed, then testing whether the means are the same requires an adjustment, but that runs into the issue of HARKing.

Answer 3

Here is another approach to address this problem by means of combining p-values

1. You can perform one-sided t-tests individually for each website, obtaining p-values. Here, significant difference would imply rejection of the null hypothesis (that application of the technique has no effect) in favour of the alternative hypothesis (that application of the technique improves performance measure). Note that if you would choose to stop here, all p-values would need to be corrected for multiple testing, for example using a Bonferroni correction. Since you only have 10 datapoints per website and 26 websites, it is likely that this approach would not find any significant changes because we ask too many questions
2. Instead of doing a Bonferroni correction, we can combine p-values of all of the above tests (e.g. using Fisher's method), resulting in a single p-value. In this case, we are effectively performing only a single hypothesis test, so there is no need for multiple comparisons. In this case, the null hypothesis is that the treatment has no effect over all websites, and that any significant effects in individual websites are due to chance.

I think this approach answers precisely the question that you are asking. In practice it will perform similarly to the model-based approach proposed by @RobertLong.

Last thing to mention is the choice of the test. The combination of p-values will work the same way if you choose Wilcoxon test. However, with 10 datapoints, it will be more of a test of consistency of the effect across websites than a test of magnitude. For example, if the effect is very strong in only one website, then the combined t-test would find the overall result significant even if there are no significant changes in the rest of the websites, whereas Wilcoxon might not, as getting 5 random points ranked higher than 5 other random points by chance is unlikely but not impossible.

I think you should pick between t-test and Wilcoxon depending on the exact question you are trying to answer. I think it is ok to use a t-test even if its assumptions are not perfectly met. If you observe large differences in magnitude and get a p-value of 1.0E-10, then it does not matter if it is wrong by an order of magnitude. If you get a p-value of 0.01 with any method, you always have to question whether you have gathered enough data, regardless if the method is perfectly accurate.