2
$\begingroup$

I have two test groups that conducted an online task measuring response times (avg, avg(congruent), avg(incongruent)). I expected one group to be faster than the other but it turned out exactly the opposite way (significant). So now I'm trying to find one or more factors that are responsible for this unexpected outcome.

My test data looks like this: participants conducted a Simon task and I measured their response times for each trial. In SPSS I have 3 variables, one for their average response times over all trials, one with their average response time for the trials that were congruent (place on screen and direction match), one with their average response time for the trials that were incongruent (place on screen and direction do not match).

My groups are monolingual and multilingual.

Literature suggests that multilinguals would score either higher averages overall, or higher averages for just the incongruent trials. My outcomes show that the monolinguals have a higher overall average and that the difference is the same between congruent and incongruent trials for both groups.

For each group I know the following variables:

  • age
  • education level
  • accuracy%
  • sex
  • lurking variables (yes|no)
  • average response times (total, congruent trials, incongruent trials)
  • first language

Each variable in itself seems to be significant, but none of the variables seem to have a significant influence in combination with the original groups.

I want to ensure that my test was not faulty and verify the validity of the test method. Maybe one other factor I didn't account for in my hypothesis is causing these results. It would either help me substantiate that my test is correct and that the results are valid, or provide a new angle for future research.

I'm not sure what test to use. For my first comparisons I used Spearman's correlation. I was told once that I may not use partial correlations when using Spearman's correlation. Can anyone help me how to proceed from here?

$\endgroup$
  • 1
    $\begingroup$ explain the differences between your three outcomes: avg, avg(congruent), avg(incongruent). $\endgroup$ – AdamO Jun 14 '13 at 18:13
  • $\begingroup$ @AdamO does this help? $\endgroup$ – Charlotte Houwing Jun 14 '13 at 18:43
1
$\begingroup$

You can probably look at your data in another way. Start with a t-test of the difference in average response time between the groups (that would be equivalent to a test that the Pearson – not Spearman – correlation between group membership and average time is 0). From there, you can easily add other variables in the model (i.e. turn it into an ANOVA or linear regression), consider transformations, rank-based statistics or generalized linear model if needed, etc.

Because of the specifics of this type of experiments, it's also standard practice to analyze only successful trials and to exclude large response times before computing the means (there are better and more principled ways to deal with this problem but you definitely need to do something about it).

It could also be more appropriate to analyze the response time to individual trials directly, using a multilevel model (see the literature on the “language-as-fixed-effect fallacy” in psycholinguistics).

@AdamO is right, thought, doing all this after the fact is at best suggestive. If you fiddle with the model until you get something you like, p-values become meaningless. Also, you might have heard of the mounting debate on reproducibility within psychology. I personally think that the ease with which we explain away unexpected results through ancillary variables or details of the procedure is part of the problem. The effect might not be what you expected but a second look at the literature might reveal that it was not as strong as it first seems. If that's the case, the “disappearance” of the effect really does not need any explanation.

$\endgroup$
  • $\begingroup$ Succesfull trials: check. Outliers removed: check. Other strange results removed: check. I'm aware of all the other factors other than language that might have affected my results. Since it's for a bachelors thesis, not my masters thesis, I don't have the time and resources to setup a multi-level trial. $\endgroup$ – Charlotte Houwing Jun 15 '13 at 14:31
  • $\begingroup$ I am not sure I understand you correctly but it seems that you already have multilevel data. I am just suggesting another way to look at it that would readily accommodate the factors you mentioned yourself (e.g. instead of a correlation you can fit a linear model of the relationship between group membership and response time, add age or gender as predictors etc.) $\endgroup$ – Gala Jun 15 '13 at 14:55
  • $\begingroup$ Now, that said, I am not suggesting that many factors might have influenced your results, quite the contrary, I am suggesting that we often use this sort of excuses to explain away failures to replicate whereas the real problem might be that there was no meaningful effect in the first place. $\endgroup$ – Gala Jun 15 '13 at 14:58
3
$\begingroup$

By attempting several tests to produce a desired outcome you are greatly inflating your type I error rate. In fact, because the direction was opposite what you expected (or considered a significant result, but not significant in the right direction), in a way you're inflating type I and type II error rates simultaneously. Statistics do not produce expected results, they measure the results you have. Perhaps your expectations are miscalibrated.

$\endgroup$
  • 1
    $\begingroup$ I understand what you mean. I just want to ensure there is no other significant factor that is influencing my results in order to substantiate my findings. $\endgroup$ – Charlotte Houwing Jun 14 '13 at 20:26
  • $\begingroup$ Again, this is not something that is tested for in confirmatory analyses, but is stated explicitly using causal modeling and a conceptual framework for the association of interest. With multivariate regression models, it is tempting to adjust for things simply because they can. One must resist all urges to do so. Carefully adjust for stratifying effects and confounding factors according to their position in causal diagrams. ref: "Regression Methods in Biostatistics" Vittinghof et al. $\endgroup$ – AdamO Jun 14 '13 at 22:46
  • $\begingroup$ Also to substantiate the validity of the test or to look for alternative angles for future research? If I can exclude the other factors, then that would confirm the validity of the test. If I can find a factor that has significant impact, it could be a target for future research right? $\endgroup$ – Charlotte Houwing Jun 15 '13 at 9:44
  • $\begingroup$ If your objective is to do a confirmatory data analysis and you find evidence contradicting what you believe, you report the evidence you find. This is the cardinal rule of science. $\endgroup$ – AdamO Jun 17 '13 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.