2
$\begingroup$

I've a table as the following that contains the results of one sample t-test on four independent collections for the null hypothesis that the mean is 0:

Collection  t-Statistic p-value     Cohen's d   Interpretation
A           8.17        2.31E-13    0.40        Medium effect size
B           9.30        9.94E-14    0.30        Small effect size
C           2.17        7.99E-06    0.18        Small effect size
D           18.23       6.78E-89    0.21        Small effect size

According to the p-value and t-statistic, I can reject the null hypothesis for all the four collections. However, the question I am trying to answer is, which one is more significant than the other one? or, can I make such inferences on this data?

For instance, can I say mean is more significantly different than 0 in collection B than in collection C based on the p-value and t-statistic. And, the mean in collection A is more significantly different from 0 than in collection D because Cohen's d in collection A has medium effect size while in collection D with it has small effect size. Does making such inferences make sense (or are accurate)?

$\endgroup$
2
$\begingroup$

In trying to compare two population means by looking at the difference in sample means, there may be three criteria to consider.

1) Practical importance. If you're wondering whether a method of helping people lose weight is effective, you might have a huge number of subjects. Then you might get a very small P-value, and so a "highly significant" difference. But if the actual average weight loss of a group of subjects who used the method for a month was 1/2 lb (or 1/4 kg), you might decide that even though the effect of the method may be significant, it is too small to be of practical importance. Whether a real difference is of practical importance is a judgment to be made by people who are familiar with the measurements and with the practical effect of a change.

2) Statistical significance. Statistical significance is often judged by looking at the P-value of the test. It is the probability of a result more extremely different from the null hypothesis. If this probability is very small (say, below 5% or 1%) you can say that the departure from the null hypothesis is 'statistically significant' (at the chosen level). In the weight loss example, you might have a small group of subjects with an average weight loss of 22 lbs (10 kg). If real, that amount of weight loss might be of practical importance. But for a small group of people, the significance level might not show a significant decrease in weight. Then you might have anecdotal evidence that the method worked well for a few people, but not enough evidence to be generally convincing to people who care about statistical significance.

3) Measures of effect size. Cohen's $d$ is one of a number of proposed measures of effect size. Roughly speaking it measures the difference obtained by looking at a difference between two means in terms of the number of standard deviations that difference represents. It is often used in two-sample tests, so that $d = \frac{\bar X_{\mathrm{trt}}-\bar X_{\mathrm{ctr;}}}{S},$ where $s$ is an estimate of the standard deviation based on the two samples.

In (1) "practical importance" may be judged by experts on the kind of data involved. Cohen's $d$ can be used as a fixed standard of what practical importance may mean. If the observed difference between treatment and control groups is half a standard deviation $(d \approx 1/2),$ then the effect may be interpreted as moderate in size; if $ d \approx 1,$ then the effect may be interpreted as large or very large.

In a weight loss study, subjects who are carefully screened and about equally motivated may yield very similar weight losses, giving a small $s$ and inflating $d.$ Then people who know and care about typical weight losses may have differences of opinion about the usefulness of Cohen's $d.$

Consider a two-sample t test on (simulated) data for $n_1 = n_2 = 500$ treatment and control subjects with weight losses x.trt and x.ctrl.

        Welch Two Sample t-test

data:  x.trt and x.ctrl
t = 24.559, df = 995.12, p-value < 2.2e-16
alternative hypothesis: 
   true difference in means is not equal to 0
95 percent confidence interval:
 4.446439 5.218732
sample estimates:
mean of x mean of y 
 14.83626  10.00367

Here the observed difference in sample means is about 4.83 lbs., and Cohen's $d$ is about 1.5.

The difference is very highly significant, and Cohen's $d$ is large, but people who know and care about weight loss may argue whether a difference in weight loss of less than 5 lbs. is of practical importance.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ So, would it be fair to argue that the message all these numbers convey depends on the interpretation of the critiques in that domain? $\endgroup$ – Hamed May 15 at 20:42
  • $\begingroup$ Discussion about the distinction between statistical significance and practical importance often involves differences of opinion. Cohen's $d$ is sometimes useful as an objective measure of effect size, but not always. $\endgroup$ – BruceET May 15 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.