2
$\begingroup$

I will defend my Master's thesis on Thursday, and I have a doubt about an analysis that I have to present. In my experiment, I had two independent variables:

  1. Age category (SU = senior unemployed subjects and YU = young unemployed subjects)
  2. Experimental condition (ST = stereotype threat; NST = No threat condition). My dependent variable is the performance achieved by subjects on a memory test.

My hypothesis was that there would be an effect of Experimental condition only on SU subjects. Consequently, I performed a two-way 2x2 ANCOVA and expected the interaction between Age category and Experimental condition to be significant. Unfortunately, the interaction turned out to be non significant (p=0.615).

However, when I looked at pairwise comparisons, only SU subjects' performance varied (almost) significantly from one experimental condition to another (p=0.058), whereas YU did not vary significantly across experimental conditions (p=0.213). This helped me confirm my hypothesis and conclude that there was an effect of the experimental condition only on SU participants.

However, I'm not quite sure about the conditions under which one is allowed to look directly at pairwise comparisons (as I did) and overlook the global interaction is not significant. I don't have any textbook at hand, so if anyone could indicate me a stat published article arguing in favor of this method and indicating the conditions under which it is doable, I'd be very grateful!

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I gather your main effects were non-significant, was the global F significant? What was your N? $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 10, 2013 at 23:36
  • $\begingroup$ Yes, giving us your sample size would really help, because power is very much dependent upon sample size. One thing that you might do is looking at partial eta squared's ($\eta^2$), which represent proportions of variance explained in your DV by your effect, and see if they are at least somewhat large or not. With small sample sizes, medium or even large effect sizes can go undetected. $\endgroup$
    – Patrick Coulombe
    Commented Sep 10, 2013 at 23:46
  • 1
    $\begingroup$ Hi, thanks for answering so quickly. N=80 (40 SU participants, 40 YU participants, 20 in each experimental condition). The global was equal to F=0,255. The partial η2 was: η2=0.003, so really not large! $\endgroup$
    – Oriane
    Commented Sep 11, 2013 at 11:17

1 Answer 1

Reset to default
1
$\begingroup$

To paraphrase part of your approach, your are doing two pairwise comparisons, resulting in two p-values, say p1 and p2. If p1 is small and p2 is not small, you would like to conclude that the difference in the first pairwise comparison is statistically somehow more significant than the difference in the second pairwise comparison.

Unfortunately, as pointed out by Andrew Gelman, interpreting differences between p-values as being statistically significant is not generally valid, at least not without the careful development of a new procedure for comparing p values: http://www.tandfonline.com/doi/abs/10.1198/000313006X152649#.Ui_P2N_ztTM

I suppose that I have addressed only one of the multiple facets of your question.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Hi! Actually, I did not intend to compare the relative sizes of p1 and p2. Rather, p1 and p2 respectively reflect the significance of the 2 comparisons that answer my experimental hypothesis (out of 4 possible comparisons -- yet the last two ones are of no interest for my research). That is, for my project, looking at comparisons 1 and 2 (SU and YU participants' performances across experimental conditions) seems to be more informative that looking at the global interaction (since it reflects all 4 comparisons). Does it seem correct to you to do so then? $\endgroup$
    – Oriane
    Commented Sep 11, 2013 at 11:22
  • 1
    $\begingroup$ Just to be clearer... The four comparisons are as follow: comparison 1 = looking at SU participants' performance across experimental conditions (interesting); comparison 2 = looking at YU participants' performance across experimental conditions (interesting); comparison 3 = inside the NST experimental condition, comparing SU and YU participants' respective performances (of no interest); comparison 4 = inside the ST experimental condition, comparing SU and YU participants' respective performances (of no interest). Hope this helps! Thanks a lot. $\endgroup$
    – Oriane
    Commented Sep 11, 2013 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.