2
$\begingroup$

Disclaimer: I have a very minimal stats background and am just getting into this stuff. Corrections to any terminology I use or an improved rephrasing would be super appreciated.

I have a simple science/perception experiment that I've run, testing a single group of subjects on 2-dimensions of conditions: speed (3 possible values) and "encoding type" (3 possible values). On a given trial, a subject is presented with a stimulus of a random "encoding type" presented at a random speed. I have a small number of subjects (~10). For each subject, I've calculated their performance (an observation) for each speed+encoding type combination.

I'm interested in only seeing if there is a significant difference in performance of of "encoding type" restricted to a given speed (e.g. is encoding type 1 at speed 1 different from encoding type 2 at speed 1). I don't care of comparisons across speeds (e.g. I don't care if encoding type at speed 1 is different from encoding type 1 at speed 2). I'm doing 9 comparisons total (3 for each speed). So far, I've only run some paired ttests (would this even be the right thing to do?). I am getting some significant results without correction (p values ~0.03).

  • Is a paired ttest the best test to use?
  • Should I be correcting at all?
  • Is there something other than bonferroni/sidak I should try? I've attempted to use bonferroni correction, where I took 0.05/9 = 0.0055. This throws out a number of results. Should I be dividing by 9 (the total number of tests I've performed) or by 3, the total number of tests constrained to the speed condition?
Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ "The researcher armed with a confidence interval, but deprived of the false respectability of statistical significance, must work harder to convince himself and others of the importance of his findings. This can only be good." Michael Oakes $\endgroup$
    – rolando2
    Aug 23, 2012 at 3:00

2 Answers 2

Reset to default
0
$\begingroup$

It sounds like you should do a 2 way ANOVA. Is your outcome measure (DV) continuously measured (Interval or ratio)?? you have two factors (IVs), Speed and Encoding. You would then just look for a signficant main effect of encoding (if you are not interested in speed). This is an omnibus test. Meaning that the multiple comparisons are not an issue (unless you go on to deconstruct this analysis further by doing pairwise comparisons).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Sorry i think i misread --- looks like you've done this and now want to deconstruct analysis by looking at 'simple effects'. the difference between encoding at each level of speed. I assume your comparisons are not 'planned comparisons'. post hoc analysis corrections would be appropriate in this case. you would need to /9 for bonferroni correction. Sidak is a little less strict. $\endgroup$
    – user41270
    Mar 9, 2014 at 19:14
0
$\begingroup$

To follow up: I ended up using a Friedman test, which is non-parametric doesn't make assumptions about equal variance. To test for equal variance in the first place, I used Levene's test, which ruled out using ANOVA as an appropriate test.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Re the equal variance assumption, look at this question. $\endgroup$
    – Silverfish
    Sep 29, 2015 at 20:25
  • $\begingroup$ On the matter of whether you should run one test to decide whether to perform another, see here and here. Using the phrase "ruled out" is very strong: in large samples, even a negligible difference in variance could be highly significant (p-value very close to zero), and yet ANOVA could still be perfectly appropriate. $\endgroup$
    – Silverfish
    Sep 29, 2015 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.