1
$\begingroup$

RQ: analyzing the effect of completing a strenuous vs. more relaxing warm-up on reaction speed.

Methodology:

  1. Randomly assigned the population to two conditions: complete a strenuous warm-up or a more relaxing one
  2. All participants took the reaction time test 3 times before being exposed to their condition to establish a benchmark.
  3. participants took the reaction time test 3 times again after being exposed to their condition

My problem is that the group that completed the more strenuous warm-up has a faster reaction speed for the benchmark so my groups aren't equal at the beginning. Is there any test I can do to test the statistical significance of the change in the mean reaction speeds? Would ANOVA work?

$\endgroup$
2
  • $\begingroup$ Treatment effects are identified even if the groups are not balanced on observables; after all, you can make a Type 1 error. If you are worried then you can randomize again and go for it but if that is too costly then you can proceed under the assumption that the true average reaction times in the two groups are the same at baseline. $\endgroup$ Commented Feb 24, 2023 at 6:17
  • $\begingroup$ Typically, what you'd do in this case is use a regression and include the pre-treatment covariates. $\endgroup$
    – num_39
    Commented Feb 28, 2023 at 20:59

2 Answers 2

0
$\begingroup$

There is no perfect way to deal with a study looking for a treatment effect when the baselines of the intervention groups are different. There are a few approaches that are common, but none can be used to support inferences without lots of thought. I'll give you a few scenarios that might help you see the issues.

Scenario 1: Imagine that the intervention increases the average reaction time by about 4 units in every individual. If you subtract the baseline from everyone then you get a good measure of the effect of the intervention. Easy. That is what is assumed implicitly when people reach for the change from baseline as their index of response. In this case it will not matter if the two experimental groups have different baselines because the effect you are interested in is independent of the baseline as long as you analyse the change from baseline. However, if you express the changes as a fraction of the individual baselines then you will disadvantage the group with the higher average pre-intervention average baseline.

Scenario 2: Imagine that the intervention increases the reaction time by about 10% of the baseline. Now you should ideally express the responses as a fractional change. If you use change from baseline then your index of response will have extra noise from the influence of the variation in the individual baselines. In this case it will not matter quite a lot that the two experimental groups have different baselines unless you express the data as a fraction of the pre-intervention baselines.

Do you know if your intervention is likely to have an additive or multiplicative effect? If not then you might wish to examine the data both as change from baseline and as fractional change. If the two expressions support the same conclusions then it might be OK to make an inference, but if they point in opposite directions then you need a better study. (It is, of course, possible that the effect of intervention is a mixture of additive and multiplicative, or that the arithmetical nature is better expressed in some other way.)

Scenario 3: Imagine that your intervention is roughly additive but that there is a ceiling to the biological phenomenon that you are measuring. (No-one can have a zero reaction time, for example.) In that case when the group with the longer pre-intervention reaction times has the opportunity to change more than the group with the shorter pre-intervention times.

Scenario 4: Imagine that the change to average reaction times is mostly due to changes to the times of the individuals who are normally slow to react, or those who are normally fast to react. It is common to assume that a population-level change is due to a roughly equivalent change in all members of the population, but biology doesn't often work like that. Individuals can be more or less susceptible to interventions of all kinds. In this scenario the difference in pre-intervention baselines should be thought about as a difference in distribution of fast and slow individuals. You might look for the nature of the correlations between pre- and post-intervention reaction times.

If you have lots of data then you can probably explore the relevance of the various scenarios that I have listed (and others), but I doubt that your study is large enough. If it is large enough for such an exploration to be helpful then it is unlikely that you would have found a substantial difference in group baselines in actually randomly assigned groups. (I would worry about the randomisation procedures, keeping in mind that haphazard allocation is not random allocation.)

If you have relatively little data then you have done a preliminary study that can serve to help you design a better subsequent study. Do not make firm conclusions at this stage. Examine the data with a variety of graphs, making sure that the baseline and post-intervention values are visible and connected.

$\endgroup$
0
$\begingroup$

I'll provide a computational response and a conceptual response. Computationally, I agree with the Yashaswi that findings account for baseline measures when those baseline measures are included in the analytic model. If a direct analysis of the difference between the post-minus-pre averages of the two groups is desired, a difference test (e.g., t-test) could be performed. However, conceptually, because there was not sampling (i.e., it sounds like there was random assignment to treatment but not random sampling of individuals from the population to participate in the study), there is not a need for inferential statistics. That is, the population values are known and so do not need to be estimated. Similarly, differences among groups can be calculated through basic arithmetic and do not need to be inferred using standard errors (which is a sampling statistic). Findings can be reported in terms of average change in strenuous versus relaxed groups; the impact of the groups not being equivalent prior to treatment is not known. Replications and alternative designs could provide insight into unanswered questions.

Re: @num_39's assertion that this response is incorrect: I disagree but would agree that industry standard would be to attempt to generalize from the individuals in the study by making strong assumptions that either the specific individuals are representative of all possible individuals who could have been selected for the study or the specific combination of individuals assigned to treatment and control conditions are representative of all possible combinations. In this study, my bet would be that neither of those assumptions hold. So I would recommend describing the actual findings, rather than speculating about what would happen with different individuals or combinations of individuals. See the replication crisis literature in any field for the problems with relying on strong assumptions and the rampant over-generalization of findings.

$\endgroup$
1
  • $\begingroup$ This is incorrect. Even without random sampling, you still have random assignment, and for this you need inferential statistics, see potential outcomes framework. $\endgroup$
    – num_39
    Commented Feb 28, 2023 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.