1
$\begingroup$

I am in the last phases of my master’s thesis, and came across a question regarding my data analysis results that I cannot seem to find a clear answer for.

For context, my data is from a randomised, double-blind, placebo-controlled, between-subject study, in which 50 healthy subjects were randomly assigned to receive either four 15μg microdoses of LSD or four doses of placebo over the course of two weeks (4 test days; baseline, dose 1, dose 4, follow-up). I used mixed model repeated measures ANCOVA with Drug (PLA, LSD) as between-subject factor, and Testday (dose 1, dose 4, follow-up) as within-subject factor. At recommendation of a statistics professor at my uni, I included the scores from the baseline test day as covariate for each outcome measure.

While the covariate itself was significant in all my analyses, the covariate:Testday interaction was not signficant for most of my variables. Except one – number of attention lapses in the PVT.

Edit as requested by @EdM
Definition/explanation of the task of the outcome variable:
[The Psychomotor Vigilance Task (PVT) measures sustained attention by assessing the reaction time in response to a visual stimulus (Dinges & Powell, 1985). Participants had to press a response button as quickly as possible when a visual stimulus occurred. The stimulus consisted of a counter that increased by 1 each millisecond (ms), and was presented a total of 100 times at random time intervals over a period of 10 minutes. Average reaction time in ms and the number of attention lapses were recorded as outcome measures. An attention lapse consists of failure to react or a reaction time greater than 500 ms.]

I used anova_test() and get_anova_table from the rstatix package to run the ANCOVA analyses (based on this guide).
The relevant outcome measure for my question is lapses, i.e. the number of times a subject failed to react or had a reaction time of over 500ms. Condition refers to the group (placebo or LSD), Testday is, well, the test day (within-subject factor as described above), and PPNR is short for "participant number" which was the case identifier. Finally, lapses_t01 is a separate variable that contains lapses scores from the baseline test day; it was made specifically to serve as baseline covariate.

The ANCOVA for lapses gave the following R-output:

R-output of ANCOVA for number of attention lapses.

My question is now: What does the significant interaction effect of the covariate (lapses_t01) and time-factor/repeated measures variable (Testday) mean?

I have searched this site repeatedly already, but was unable to draw a firm conclusion from the explanations on any similar questions I could find. A few even seem to contradict each other; some say this just means that the influence of the covariate varies depending on Testday, while others say that this indicates a violation of ANCOVA assumptions.

I would be very grateful to learn how to interpret this result correctly.

$\endgroup$
8
  • $\begingroup$ Welcome to Cross Validated! As this site tries to serve as an accessible repository of questions and answers, please edit your question to provide explanations for abbreviations you use that might not be widely known (PVT, PPNR). It would also help if in your editing you could include more details about your data (e.g., number of individuals, nature of your outcome variables like "attention lapses"), the form of the mixed model that you used, and (as this seems to be in R) any non-base packages that you used (e.g., the sources of the anova_test() and get_anova_table() functions). $\endgroup$
    – EdM
    Aug 4, 2022 at 14:23
  • $\begingroup$ @EdM thank you! I adjusted my question, I hope this is what you meant? $\endgroup$ Aug 4, 2022 at 21:05
  • $\begingroup$ Did you want the model to include an interaction between the baseline lapses_t01 and the within-subject factor Testday? $\endgroup$
    – dipetkov
    Aug 7, 2022 at 15:33
  • $\begingroup$ @dipetkov not specifically, but I was under the impression that its inclusion is mandatory if I want to include the covariate. $\endgroup$ Aug 11, 2022 at 10:19
  • $\begingroup$ No, ANCOVA doesn't require to interact the covariates with the categorical variables. The point is that the R package/function made an important decision on your behalf. Two-Way ANCOVA: It is necessary to include a non-significant interaction term in the model? $\endgroup$
    – dipetkov
    Aug 11, 2022 at 19:23

1 Answer 1

1
$\begingroup$

Instead of getting caught up in ANCOVA terminology, remember that it's just a particular form of a linear regression model. With ANCOVA, you have a continuous outcome (here, lapses, but see below), some categorical predictor(s) of primary interest (here, Condition and Testday), and some other (typically continuous) predictor(s) to control for (here, lapses_t01). The model, however, treats all the predictors equivalently.

When you include an interaction term in such a model, you are allowing for the possibility that the regression coefficient for one predictor depends on the level of another predictor (and necessarily vice-versa). A "statistically significant" interaction coefficient suggests that such interdependence of those regression coefficients is the case. In your example, the association of lapses with lapses_t01 depends on the Testday, and the the association of lapses with Testday depends on the lapses_t01 value.

I suppose that significant interaction violates "ANCOVA assumptions" in that you can't evaluate the predictions for combinations of your primary variables of interest, Condition and Testday, without also knowing the value of lapses_t01.

But that doesn't mean that the overall regression model is incorrect. With an interaction like that, you can illustrate the model predictions for relevant combinations of predictors. You can also evaluate the magnitude of the interaction, to determine whether there is a practical significance to the statistical significance.

You can't get those details from the ANOVA table you show. The model, however, contains the information you need in the individual regression coefficient estimates and their covariance matrix. Tools for post-modeling analysis, like the emmeans package, make it straightforward to evaluate the detailed implications of your model.

A couple of thoughts on the model

First, you might discuss with your statistical consultant whether a linear regression model is appropriate for your lapses outcome. As they are counts and not continuous, the assumptions underlying the model might not hold, particularly if the counts are few. Count data can be analyzed with a generalized linear model based on Poisson or negative binomial statistics.

Second, think about the implications of your missing data. You might need to consider multiple imputation to avoid bias if the data aren't missing completely at random. Stef van Buuren's book is a superb explanation of the issues, with tools provided by his mice package and by Frank Harrell's rms package and associated Hmisc package.

It's also possible, however, that the way you structured your ANCOVA exacerbated the problem with missing data. I'm not familiar with the rstatix package that you used; it might have forced you into the form of a repeated measures ANCOVA that requires all subjects to have data for all time points. Depending on the structure of your missing data, a mixed model with the lme4 package or generalized least squares regression might allow you to work directly with all the data that you acquired.

Finally, be wary of Type III tests. See this page for an explanation of the different ANOVA types, and the Warning on the help page for the Anova() function in the car package.

$\endgroup$
8
  • $\begingroup$ rstatix is a supposedly "simple" way to do statistics within the "tidyverse". In practice this means rstatix calls the well-known statistics packages (base stats, emmeans, car) in a obscure way so that it's never clear what model was actually fitted. For example, according to the anova_test docs: "type = 2 is the default because this will yield identical ANOVA results as type = 1 when data are balanced but type = 2 will additionally yield various assumption tests where appropriate. When the data are unbalanced the type = 3 is used by popular commercial softwares including SPSS." $\endgroup$
    – dipetkov
    Aug 7, 2022 at 15:45
  • 1
    $\begingroup$ It's by the author of survminer which speaks for itself. $\endgroup$
    – dipetkov
    Aug 7, 2022 at 15:46
  • $\begingroup$ Thank you for your explanations! Unfortunately, I am not in the position to adjust the analysis at this point. From what I understand based on your explanations, the ANCOVA results for lapses cannot be directly interpreted, is that correct? $\endgroup$ Aug 11, 2022 at 10:18
  • $\begingroup$ @DruidRaves-V the coefficient estimates and their covariance matrix, which should be provided by the modeling function, can let you interpret associations of lapses with the predictors. You just need to specify values of all three predictors--lapses_t01, Testday and Condition--as their interactions mean that the association of each with outcome depends on the levels of the others. Their individual coefficients are for situations when the others are at 0 or reference levels. The Type III ANOVA summary used by get_anova_table() was probably inappropriate for your data. $\endgroup$
    – EdM
    Aug 11, 2022 at 16:34
  • $\begingroup$ Okay... given the available function/output, would you agree with the following though? "Mixed RM ANCOVA revealed a significant interaction effect of the covariate, number of lapses at baseline, and Testday (F2,76 = 3.31, p = 0.04, η2G = 0.03), indicating interdependence of these variables. As this violates the assumptions of ANCOVA, the results for this variable (i.e. number of attention lapses) are not reliable and should not be interpreted." ? Or is this incorrect? $\endgroup$ Aug 11, 2022 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.