1
$\begingroup$

I have variables a1 and a2 with 57 observations each, and variables b1 and b2 with 29 observations each.

I compare a1 with a2 by means of a paired-samples t-test, this gives p<0.001. Next, I compare b1 with b2 by means of a paired-samples t-test, this gives p>0.05 (actually p>0.10).

I compared the p-values, and concluded that I do find a significant effect for the larger set of 57 observations, and do not find a significant effect for the smaller set of 29 observations.

I am happy with that result, but got the criticism that the p-value of the first t-test cannot be compared to the one of the second test, because of the different sample sizes (57 versus 29).

What solution would you recommend to solve this problem?

$\endgroup$
  • 2
    $\begingroup$ Welcome to Cross Validated! Could you edit the question to explain what comparisons you're making between the results of the two tests? $\endgroup$ – Scortchi - Reinstate Monica Jun 26 '17 at 11:25
  • $\begingroup$ Thanks! Note that the bare statement "I compared the p-values, and concluded that I do find a significant effect for the larger set of 57 observations, and do not find a significant effect for the smaller set of 29 observations" is unexceptionable. $\endgroup$ – Scortchi - Reinstate Monica Jun 26 '17 at 15:40
1
$\begingroup$

I agree with @ConnorG that you should use a standardized effect size, which is independent of sample size. However, the equation he presented is for between-subjects tests. Calculating a Cohen's d is trickier, but possible, doing paired-samples t-tests. There are multiple ways of calculating it, but I believe the simplest is Cohen's dz. See here for an illustration of multiple formulas.

You can then see which sample had a bigger effect size dz.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

If your goal is to say that the difference you found in group a is greater than that in group b then you are going to need a formal test of the interaction. The easiest way to do this would be to do a repeated measures analysis of variance with group (a versus b) as the between participants factor and before ad after as the within subjects factor.

these three brief articles in the British Medical Journal explain in more detail what the issues are: http://www.bmj.com/content/313/7055/486 http://www.bmj.com/content/313/7060/808 http://www.bmj.com/content/313/7061/862

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.