# T-test shows no differences, but the experiment group shows tendency more benefit in all variables measured than control group

I've just finished an animal experiment. I compared 1 control group and 1 experimental group, the only difference between the two is type of diet. For statistical analysis I used the independent groups t-test, and the result showed no significant differences between the two groups. However, the data shows the tendency that the experimental group has more benefit in all variables measured. So, what should I say about my data? All data are normally distributed.

My supervisor said that maybe because I used very small sample (each group n=8) that I could not find any significant differences. He suggested me to do some "probability test" or something to extrapolate my data (unfortunately, I don't have any clue what he was talking about).

So, is there any statistical analysis that I can use like what my supervisor told me to do?

• How many variables did you measure / test? Do you think of them as related to each other, or are they independent? Commented Jun 13, 2013 at 5:13
• With the 'probability test', do you think he might be referring to Fisher's method? If so, you'd need the various response variables to be independent, which I would usually doubt. Commented Jun 13, 2013 at 5:39
• You say "T-test" in your title but "t-test" in your body text. Did you do several univariate two-sample t-tests, or did you do a single multivariate T-test? (See also this and this and this) Commented Jun 13, 2013 at 5:40

Calculate an effect size (such as Cohen's d). Effect sizes are not as influenced by sample size as the test statistic is.

• Effect sizes aren't, but the accuracy of inferential estimates thereof is. Low power probably means a large confidence interval for your d.
– jona
Commented Nov 11, 2013 at 1:54

Your supervisor may very well be right and the small sample size is the problem. You might want to do a bit of reading on Power Analysis. An introductory paper is that by Cohen (1992).

In short, there is a relation between sample size, effect size and power (which is the probability that the test detects a significant effect assuming that there is one). For example, if you have an estimate of the effect size you're looking for (in your example the difference between the means of the two groups) and you want to obtain a statistically significant result regardings this effect with a certain error probability (the $\alpha$-Level), then you can compute the size of the sample that is neccessary. Generally, when you have two of the numbers, you can compute the third one.

The difficult part is probably to get an idea of the effect size before doing the analysis. After all, ususally that is what one wants to find out about. An interesting discussion on this can be found on the Cognitive Sciences SE site.

One piece of free software to do power analysis is G Power. There is also the pwr-package for R.

References:

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155.