2
$\begingroup$

I am trying to do a two samples t-test. The null hypothesis is there is no difference in mean runs scored by two teams, the alternative is there is a difference.

sample 1 size = 104

sample 2 size = 150

mean of sample 1 = 236

mean of sample 2 = 184

significance of the test= 0.05, Test statistics = 2.46, P value = 0.014, cohen's d = 0.31(small)

How to interpret this result? The t-test says that on average team1 makes more runs than the team2 and the result is statistically significant. But the effect size is small, d=0.31. So what should I conclude from this result? Should I conclude that there is a difference between two samples and team1 perform much better than team2 or should I reject the this claim as the effect size is small? What to do in these type of situations and How do you describe the result to others?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ "much" does not look justified in your conclusion $\endgroup$
    – Henry
    Commented Oct 4, 2021 at 8:03
  • 1
    $\begingroup$ Your questions answers itself. The implications depend on whether the smallness of the effect size limits the practical implications of the effect in the situation. $\endgroup$
    – ReneBt
    Commented Oct 4, 2021 at 8:22
  • $\begingroup$ We don't know a lot about your data but it seems that you have important variability in your data. You might also be interested in performing a power analysis to validate your study... $\endgroup$
    – Pitouille
    Commented Oct 4, 2021 at 10:19
  • 3
    $\begingroup$ On what basis do you consider $d=0.31$ to be "small"? Remember, Cohen characterized values as "small," "medium," or "large" with great diffidence, recognizing that such qualitative evaluations could not possibly be general and were specific to some forms of observation and experimentation in the "behavioral sciences." In particular, you should never take these as rigorous definitions. $\endgroup$
    – whuber ♦
    Commented Oct 4, 2021 at 13:40

1 Answer 1

Reset to default
5
$\begingroup$

The conclusion is exactly what the test says.

There is a small difference between the two groups, and it is unlikely to have arisen by chance. That is, you can have confidence that the one group really is higher on average than the other. However, the difference is $0.31$, which might not be enough for you to care about the difference. It is okay to conclude that a statistically significant difference lacks practical significance. (Perhaps your business requires a difference of at least $0.5$ to be able to profit from the new technique; you don't have enough of an improvement to profit.) Always remember that hypothesis tests are---and should be---sensitive to sample size, with large sample sizes being able to reject small differences that really are there but might be so small that they do not warrant our attention.

As mentioned in the comments, however, $0.31$ does not have to be small. That is up to your knowledge of the subject of the opinions of subject matter experts (physicians, engineers, etc).

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ You write as if Cohen's d had a practical meaning. Usually it does not: it is a standardized effect--a difference relative to a measurement of dispersion in the population. It is most useful when the measurements themselves have no inherent meaning, such as scores on a psychological test. $\endgroup$
    – whuber ♦
    Commented Oct 4, 2021 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.