I have a bunch of websites websiteA....websiteZ for example. Now I have an optimization technique that possibly improves the performance of these websites. I want to check whether this technique actually significantly effects the performance. So my dependent variable is apply_technique with the treatments enabled and disabled.

So for each website I measure the performance with and without the optimization technique applied. So I get results like this:

website | technique disabled | technique enabled
websiteA      20 seconds          17 seconds
websiteZ      45 seconds          39 seconds


However, to account for possible fluctations I measured each (website, treatment) combination 5 times so for websiteA I have 5 measurements with the technique disabled and 5 with the technique enabled. This then results in 26 websites * 2 treatments * 5 repetitions = 260 measurements.

My question is, when I want to do a paired t-test do I first need to average the performance over these 5 trials or not? I might lose some information when I average it right?

Could I also decide to not average them? So I have 130 technique_disabled observations and 130 technique_enabled observations and I then simply use these to do a paired t-test? Would that be acceptable?

EDIT: I see that the performance differences (technique_enabled - technique_disabled) are not normally distributed. So I will probably use a Wilcoxon signed rank test. However here the same question applies? Should I average the observations over the 5 trials or not?


Averaging the data will result in a loss information and statistical power, so it is best avoided.

Since you have repeated measures for websites, you can account the differences between websites (or equivalently, the non-independence of observations within each website, since observations on one website are more likely to be similar to each other than to observations on other websites), by fitting random intercepts for website ID in a regression model (a mixed effects regression model). This would look something like:

 apply_technique ~ treatment + (1 | website_ID)

and you could fit such a model using lmer from the lme4 package. This way you will make maximal use of the data.


If each group has the same number of members, then the mean of the group means is the same as the mean over all the observations. The sd can be more complicated, depending on what you're taking as your null hypothesis. One of the simplest null hypotheses is that each observation is equal to $\mu(W)+\epsilon$, where $\mu$ is some function of the website (that is, $\mu: \text {set of websites} \rightarrow \mathbb R$), and $\epsilon$ is normally distributed with zero mean and sd that is independent of the website. A more complicated null hypothesis would have the sd of $\epsilon$ depend on the website (there is no need to consider the possibility that the mean is nonzero or depends on the website, as that is absorbed into $\mu(W)$).

If your null hypothesis has constant $\sigma$, then $s$ should be calculated on the set of observations as a whole. However, if your null doesn't have a single $\sigma$, then you won't be able to calculate a single $s$. You'll have to calculate a different $s$ and thus $p$ for each website, then use some method of combining them. For this to be rigorous, you should decide on the combination method before collecting any data.

Now, if you reject this null, then you are rejecting the hypothesis that the statistic is normally distributed and the mean for the two conditions is the same. So the null being false means that the means are different or the statistic isn't normally distributed. If you think, looking at the data, that the statistic isn't normally distributed, then testing whether the means are the same requires an adjustment, but that runs into the issue of HARKing.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.