In my design, I have 3 binary items for each condition (4 conditions total) meant to measure the same thing. All participants were in all conditions (repeated measures, 12 scores each).

I made a composite score for each participants for each condition (yes, I know, not advisable but this is the way I am doing it so work with me, please). I coded the items such that 0= small and 1= large. I then interpreted the composite score to mean the likelihood of choosing the large item. (With me so far?)

I ran a rANOVA using my 2 IV's as within subjects factors.

The output says that there are 2 main effects and an interaction (I can't post the image but I am basing this off of the table called "multivariate tests" with Wilks' Lambda, which is p<0.05 (as are all the other tests there).

However, when I look at some of the levels (either in the main effects or in the interaction), I see that the confidence interval contains 0.5 (which to me means that we the level does not provide us any more info about how likely it is to choose "large.")

Does the significance in the table I have shown here mean that the effects do exist?

Should I interpret the effect by looking at the individual levels? (i.e. <0.5 means more likely to choose large) and see if the confidence intervals overlap?

What does it mean if the table says a main effect with 2 levels is significant but one of the 2 levels has a confidence interval that contains 0.5?

Thanks so much!

  • $\begingroup$ Why can't a difference of 0.5 be significant? it should depend on the variance of the estimates. If the standard error of the estimates are well below 0.5 say 0.1 or less then a p-value less than 0.05 is consistent with this result. $\endgroup$ – Michael R. Chernick Jun 12 '12 at 23:21
  • $\begingroup$ I understand that it is significant- I just don't know how to interpret that in light of the meaning of my proportion (aka likelihood to choose large). I thought that if the confidence interval were significantly over 0.5 then I would be able to say there is a preference for large (and vice versa for <0.5). However, 0.5 itself is problematic in those terms. Did I clarify my question? $\endgroup$ – Emily Jun 12 '12 at 23:24
  • 1
    $\begingroup$ In that case (responding to your comment) the issue you are addressing is not statistical. It has to do with what you would call a large enough difference to be important. This is very much a subject matter and data dependent issue. $\endgroup$ – Michael R. Chernick Jun 12 '12 at 23:29
  • 2
    $\begingroup$ Yes. It means repeated measures ANOVA. @Michael- maybe you could answer the question then of how I deal with this subject matter in interpreting my results in an answer below? Thank you! $\endgroup$ – Emily Jun 12 '12 at 23:44
  • 1
    $\begingroup$ @Michael Nothing prevents you to leave a comment first, and to convert it as a reply after clarification or confirmation from the OP. Often, this may help clarifying or refining a question so as to provide a better answer or arrange multiple replies into one. Thank you to remember these rules of use in future. $\endgroup$ – chl Jun 13 '12 at 6:37

The way to treat this problem taking what is considered to be an important difference into account is in the initial design where the sample size is chosen. Do not choose the sample size to be so large that the power to detect a difference of 0.5 is high. Define the importance difference (maybe that is 0.9). Then choose the sample size large enough to have power say of 95% to detect a difference of 0.9 but no larger. Then the test will not have much chance of detecting unimportant differences such as 0.5. The idea is that in the hypothesis testing framework you can incorporate what you consider to be a significant effect size into the design for the test.

  • $\begingroup$ I don't understand what you mean. My DV values are between 0 and 1 (percentage of people who picked "large")- how could I get a difference of 6? Also, I already ran the study with 34 participants. $\endgroup$ – Emily Jun 13 '12 at 0:11
  • $\begingroup$ Sorry. I lost the context of the problem. I will edit that to say 0.9. My point is that you look at this in terms of hypothesis testing rather than confidnce intervals. if in the context of your problem 0.5 is not a meaningful proportion then set up the design to have power to detect a sgnificant proportion (perhaps 0.9 or some other high value). Then the chance of the test calling 0.5 significant is decreased. $\endgroup$ – Michael R. Chernick Jun 13 '12 at 0:17
  • $\begingroup$ @Emily I know that you did the experiment with 34 participants and got the results you described. I was addressing the question as to how you might have initially set up the problem so as to detect only differences that you would consider meaningful. This might have led you to choose fewer than 34 subjects for the experiment. $\endgroup$ – Michael R. Chernick Jun 13 '12 at 0:45
  • $\begingroup$ Thanks for the answer. I'm just not sure how to proceed from here. $\endgroup$ – Emily Jun 13 '12 at 5:29
  • $\begingroup$ Given the data as it is about all you can say is that a difference of 0.5 is statisticlly significant and the confidence interval provides a ange of likely values for the proportion. If you like my answer you can approve it and upvote it. $\endgroup$ – Michael R. Chernick Jun 13 '12 at 6:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.