3
$\begingroup$

I have two samples, my hypothesis is that both of them are equal. The number of experiments is different (N), the means are very similar and the medians are 5 and 4.

First I calculate descriptive statistics to find:

N1: 1077  Mean1: 8.39  Median1: 5  StDev1: 23.14  Skewness1: 8.20  Kurtosis1: 101.241
N2: 729   Mean2: 8.37  Median2: 4  StDev2: 16.62  Skewness2: 21.18 Kurtosis2: 568.66

First of all, I think the population is not normal, it is skewed to the left close to 0.

Next, I made the box plot of both samples to visually compare them, they looked quite similar.

Finally I did a two sample t-test, 95% CI and resulted in 0,98 p-value. The p-value being so close to 0, I could say that the samples are equal right?

However, the samples are not normally distributed so I am not sure if I can use the t test.

Did I proceed correctly? How should I proceed otherwise?

Edit (more info about the test):

Basically, I had quite an active forum which I changed in a way that... the users were not so happy about. Now, the amount of activity in terms of new topics and new replies has decreased but what I want to compare is, if a new topic is posted is it likely to have the same, less or more replies now than before the change.

$\endgroup$
3
$\begingroup$

Your samples are normally large enough for the CLT to hold but in this case they are characterized by an extraordinary degree of skewness, which makes convergence slower. Thus, in lieu of the t-test, you might want to use a non-parametric alternative, such as the Mann-Whitney test.

I want to be clear on what this test shows though. It does not test for equality of samples-the samples are of course not equal- but whether the location of the population distributions is identical. This is an important distinction.

Now, if you want to test for normality or equality of distributions, this is an entirely different matter. There are many tests for normality and I am not convinced of their usefulness to be honest since in large samples they will overwhelmingly reject the normality hypothesis. A QQ plot is more useful in that respect, although it carries a certain degree of obejectivity with it. Lastly, if you want to test for identical distributions and not for a single parameter, a Kolmogorov-Smirnov test is one way to accomplish this.

Hope this clears it up a bit.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The summary statistics show these samples are extraordinarily skewed. We should hesitate to conclude that the CLT has much to say. $\endgroup$ – whuber Sep 22 '15 at 11:55
  • $\begingroup$ @whuber Even with 1000 and 700 observations respectively? $\endgroup$ – JohnK Sep 22 '15 at 11:55
  • 2
    $\begingroup$ Yes, even with that many. The sampling distribution of the difference of means will have some noticeable skewness (make a simulation and draw a histogram). It's not likely to be terrible, but it may be enough to suggest using an alternative way to compute the p-value of the t statistic, especially if a one-sided test is needed. On the other hand, the sample means are so close compared to the SDs that it is obvious there isn't enough data here to detect a difference in underlying means. For detecting other differences, the huge skewnesses in the data may become even more important. $\endgroup$ – whuber Sep 22 '15 at 12:08
  • $\begingroup$ Hi JohnK and whuber, thank you very much for replying. Maybe if I describe what's the problem that we want to answer you can give me better guidance. Basically I had quite an active forum which I changed in a way that... the users where not so happy about. Now, the amount of activity in terms of new topics and new replies has decreased but what I want to compare is, if a new topic is posted is it likely to have the same, less or more replies now than before the change? $\endgroup$ – gstats Sep 22 '15 at 12:43
  • $\begingroup$ @gstats Is that what your data represents then? Average number of replies per topic, before and after the change? $\endgroup$ – JohnK Sep 22 '15 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.