5
$\begingroup$

This could be a pretty basic question, I'm a little rusty on my stats knowledge.

Background: I am monitoring website load time performance. To do so, I have a script running and capturing data points (About 400) on load time through various Agents. Every Agent is located in different geographic locations, but they measure the same steps.

I would like to determine if there is statistical difference between the agents. So if one is consistently reporting slower load time performance I would like to know if its because of the Agent or not. I would include images but I need 10 reputation points and I just found out about this website.

Problem: I have two sets of data from different agents measuring the seconds it takes a website to download, both are bell-shaped but are heavily skewed to the right. Can I still perform ANOVA to determine if there is difference, even though they are skewed?

Thanks in advance

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ You'll be doing ANOVA on the means for each subject, not on each data point. Are the means skewed? $\endgroup$
    – John
    Commented Jun 17, 2013 at 20:32
  • 1
    $\begingroup$ @John the uniform distribution of p-values under the null is based on the assumption that the data is normal, not the means - it relies among other things on the sums of squares of residuals being a scaled chi-square and on the independence of numerator and denominator in the F; if the OP is interested in those p-values behaving as claimed the skewness should not simply be ignored. $\endgroup$
    – Glen_b
    Commented Jun 18, 2013 at 0:19
  • $\begingroup$ I think you're taking it as I was suggesting the means of each condition. I was thinking repeated measures where the CLT does help if you collect enough data per cell. Perhaps this isn't, it isn't clear. In the repeated measures case the individual values usually** never enter into or matter for the ANOVA. Mewhort has a paper on this demonstrating that correcting times for skew does not influence ANOVA negatively when there are enough trials / cell. He doesn't say it but the CLT normalizes the distribution that's analyzed. $\endgroup$
    – John
    Commented Jun 18, 2013 at 3:27
  • $\begingroup$ ** unless you collect one sample / cell / subject. $\endgroup$
    – John
    Commented Jun 18, 2013 at 3:28
  • 2
    $\begingroup$ (oops, subjects when I meant agents). No, my thinking was that the data is collected in a repeated measures design such that each agent has several loads of each page design. If you don't care about the agents, only the page design, what you can do is average each design within each agent and do a repeated measures ANOVA. 30 might be too few to eliminate skewness of the kind you're likely observing. But you can look at the distribution of resultant means and see. $\endgroup$
    – John
    Commented Jun 19, 2013 at 20:20

2 Answers 2

Reset to default
6
$\begingroup$

If the distributions are similar (in particular have the same variance) and the group sizes are identical (balanced design), you probably have no reason to worry. Formally, the normality assumption is violated and it can matter but it is less important than the equality of variance assumption and simulation studies have shown ANOVA to be quite robust to such violations as long as the sample size and the variance are the same across all cells of the design. If you combine several violations (say non-normality and heteroscedasticity) or have an unbalanced design, you cannot trust the F test anymore.

That said, the distribution will also have an impact on the error variance and even if the nominal error level is preserved, non-normal data can severely reduce the power to detect a given difference. Also, when you are looking at skewed distributions, a few large values can have a big influence on the mean. Consequently, it's possible that two groups really have different means (in the sample and in the population) but that most of the observations (i.e. most of the test runs in your case) are in fact very similar. The mean therefore might not be what you are interested in (or at least not be all you are interested in).

In a nutshell, you could probably still use ANOVA as inference will not necessarily be threatened but you might also want to consider alternatives to increase power or learn more about your data.

Also note that strictly speaking the normality assumption applies to the distribution of the residuals, you should therefore look at residual plots or at least at the distribution in each cell, not at the whole data set at once.

Improve this answer
$\endgroup$
1
  • $\begingroup$ After three days of reading all comments & looking at the data I think I finally understand. Basically, I should make sure my samples for both Agents are the same size and see if their variance is similar, if I have this then I can do ANOVA to determine if there is difference between the agents. I need to be comparing apples to apples essentially, in order to determine if there is indeed difference in what is being measured. Also, I can't do the ANOVA straight up on the data points, but rather bucket them into time intervals and average those because of CLT, which gives me normal distribution. $\endgroup$
    – jpbehrens
    Commented Jun 20, 2013 at 14:16
5
$\begingroup$

There is a degree of sensitivity to heavy skewness in ANOVA

Times, in particular, tend to be much more skewed than speeds (inverse-times) and log-time. If your question of interest could be stated in terms of one of those (and they are less skew) then you may not have to rely on an assumption that doesn't hold.

Further, and even more critically, you also tend to get strongly different spreads with times (rather than either log-times or with speeds).

What does your data look like? Have you got some by-group displays (e.g QQ plots, box plots) and summaries (e.g. mean, sd, median, quartiles) ?

Improve this answer
$\endgroup$
2
  • $\begingroup$ I tried pasting the summarized data but could not format it correctly (sorry still learning the ropes of this community) . But I do believe I am dealing with Times... although not exactly sure what this means. Maybe a reference I can read on can clarify my mind and I can come back with a better question? $\endgroup$
    – jpbehrens
    Commented Jun 20, 2013 at 14:05
  • 1
    $\begingroup$ Jeampz - If you paste your data from an ascii file in what looks like a good format before you paste (hint: avoid tabs), once it's pasted in, select the whole table and click the {} button above the edit window. Now save. $\endgroup$
    – Glen_b
    Commented Jun 20, 2013 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.