2
$\begingroup$

I ran an experiment where I expected that reaction times in two different conditions would differ, and I estimated that one condition would always be associated with longer reaction times.

After collecting the data, I realised that there did appear to be systematic differences between the two conditions, but that individual differences (between participants) seemed to dictate the direction of the difference (i.e., some people responded systematically slower in one condition, and other people vice versa). I mostly just care that differentiation is happening, not so much whether reaction times are shorter or longer in one condition or the other.

What I would like to do is calculate the relative distinction (in reaction times) between conditions. I don't want to use the absolute or unsigned difference in milliseconds, given that different participants will have different reaction time speeds in general.

In my manuscript, I ended up dividing the mean reaction time of condition A by the mean reaction time of condition B, and then used the unsigned difference between that ratio and 1 as the new dependent variable:

the absolute value of 1 - [condition A/condition B]

I received back reviewer comments and they didn't feel I named or communicated this term very well. I said "ratio difference" but I can see why that would be confusing. So I need a standard term, or a more concise and clever definition than I can come up with. Is there a commonly accepted statistical term I have missed?

The other criticism was a lack of significance testing. How can I establish, for each participant, that this ratio is significant, before adopting it as a dependent variable? For each condition (A or B), for each participant, for each block, I have at minimum 30 reaction times. So perhaps I could test, within each block and participant, between the two conditions, as is suggested by this answer. In that case, should I use the test statistic as a DV, or simply report it as validation for using the ratio difference? And I presumably would then correct each participant test's p-value for multiple comparisons.

Edited to add that I have been looking at questions already posted here concerning the use of ratios in regression, but it seems that this is an issue where the ratio is constituted by an interaction, especially if the ratio is meant to be interpreted as a predictor. What I am chiefly concerned with is how unequal the two conditions are, if that makes sense--not necessarily the relative contribution of one factor over the other.

Edited again to add: for anyone else reading this question with a similar issue, these previous questions do a much better job articulating what my original research goal was:

Extracting slopes for cases from a mixed effects model (lme4)

Test whether random slopes are significantly different from 0 for individual subjects

$\endgroup$

1 Answer 1

2
$\begingroup$

First, your formula can be written in several different ways:

$$ \left| \frac{A}{B}-1\right|=\left|\frac{A-B}{B} \right|=\frac{\left|A-B \right|}{|B|}$$

with the last being the form for one definition of (absolute) relative difference.

Second, that form places a particular importance on Condition B as the reference. If that's the case your formula might make some sense. But if Condition A is really the reference you might want to replace the denominator by |A|. If neither is a reference, you might be better off using the maximum of A and B or their average for a measure of relative difference.

Third, your use of this measure on the averages as the outcome for each participant might be getting in the way of a statistical approach to test your hypothesis. In general, the closer you get to the original observations, the better. The multiple trials for each participant suggests that you might be better off with going back to individual data rows for each trial.

Each data row then represents a single trial with the reaction time, the Condition, the ID of the participant, and any other covariate values you might have. If you think that these are relative differences rather than absolute differences you might want to work with a log scale for reaction times, or use a generalized linear model with a log link between the predictors and the outcome.

The two Conditions are a fixed effect, with one chosen as the reference. The random intercepts take into account participant differences in the reference-condition reaction times, and the random slopes would represent the further participant differences between the two conditions. The magnitude of the variance among those random slopes seems to be what you are interested in.* For an introduction to testing significance of random effects, you could start with this page.

Finally, I'm a bit worried by the way you have approached this. You started out with an hypothesis about a particular directionality of difference between the two Conditions, and only came up with this outcome measure after you found that hypothesis to be unsupported. That's not generally good practice, as hypothesis testing fundamentally assumes hypotheses that were formulated without looking at the data.


*Generally one would expect a distribution of differences between 2 conditions among individuals just at random. I expect that you have some variability of a particularly large magnitude in mind, or perhaps a difference that in turn is associated with some covariate for each individual.

$\endgroup$
5
  • $\begingroup$ Thanks! I know very little about testing random effects in this way, so I will check out the resource you mention. From your explanation here, I am not sure I follow, but perhaps furthering reading will help. Here is a little more detail, to explain the hypothesis testing - I understand it sounds dodgy, and this is of course something the reviewers picked up on. But essentially what I am looking for, and what the point of the experiment is, is to see participants expressing a pattern. I don't care how they express the pattern, just that they don't treat the two conditions as equivalent. $\endgroup$
    – stck8888
    May 2, 2021 at 12:29
  • $\begingroup$ I had theoretical reasons to think reaction times would be shorter in one condition, and some participants indeed performed the task this way. But other participants "marked" the difference in conditions by instead producing much later or slower reaction times - you can think that some participants were faster, and other participants may have made a more emphatic (and slower) response, but in both cases, they have made a distinction. I hope that clears up my thinking, somewhat. $\endgroup$
    – stck8888
    May 2, 2021 at 12:31
  • $\begingroup$ @stck8888 a couple of other resources on testing random effects: from the R companion website, and from the University of Wisconsin. You have to be very careful in just what you mean by "participants expressing a pattern," and formulate that general idea in a testable way. $\endgroup$
    – EdM
    May 2, 2021 at 15:48
  • $\begingroup$ Thanks for these! Do you think it would be better to simply include a descriptive report but without any kind of hypothesis testing? I appreciate the philosophy of hypothesis testing in principle, but it can feel like reviewers won't consider work worth considering at all unless there are p-values < 0.05. My own thought was to clearly identify this analysis as exploratory for this manuscript, and then confirm with further data collection when that's possible again. $\endgroup$
    – stck8888
    May 2, 2021 at 18:19
  • 1
    $\begingroup$ @stck8888 that depends on what is expected in your field. Most conservative would be to use your work so far as preliminary data to support an application for a grant to support testing a specific interesting hypothesis in the future, using your data as a guide to designing the sample size needed to get adequate power. If what you seem to have found is interesting enough, then a descriptive report might be called for, with illustrations of effect sizes tempered by the recognition that this is based on what you found after collecting the data. $\endgroup$
    – EdM
    May 2, 2021 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.