Repeated measures ANCOVA interpretation – significant covariate/test day interaction

Question

I am in the last phases of my master’s thesis, and came across a question regarding my data analysis results that I cannot seem to find a clear answer for.

For context, my data is from a randomised, double-blind, placebo-controlled, between-subject study, in which 50 healthy subjects were randomly assigned to receive either four 15μg microdoses of LSD or four doses of placebo over the course of two weeks (4 test days; baseline, dose 1, dose 4, follow-up). I used mixed model repeated measures ANCOVA with Drug (PLA, LSD) as between-subject factor, and Testday (dose 1, dose 4, follow-up) as within-subject factor. At recommendation of a statistics professor at my uni, I included the scores from the baseline test day as covariate for each outcome measure.

While the covariate itself was significant in all my analyses, the covariate:Testday interaction was not signficant for most of my variables. Except one – number of attention lapses in the PVT.

Edit as requested by @EdM
Definition/explanation of the task of the outcome variable:
[The Psychomotor Vigilance Task (PVT) measures sustained attention by assessing the reaction time in response to a visual stimulus (Dinges & Powell, 1985). Participants had to press a response button as quickly as possible when a visual stimulus occurred. The stimulus consisted of a counter that increased by 1 each millisecond (ms), and was presented a total of 100 times at random time intervals over a period of 10 minutes. Average reaction time in ms and the number of attention lapses were recorded as outcome measures. An attention lapse consists of failure to react or a reaction time greater than 500 ms.]

I used anova_test() and get_anova_table from the rstatix package to run the ANCOVA analyses (based on this guide).
The relevant outcome measure for my question is lapses, i.e. the number of times a subject failed to react or had a reaction time of over 500ms. Condition refers to the group (placebo or LSD), Testday is, well, the test day (within-subject factor as described above), and PPNR is short for "participant number" which was the case identifier. Finally, lapses_t01 is a separate variable that contains lapses scores from the baseline test day; it was made specifically to serve as baseline covariate.

The ANCOVA for lapses gave the following R-output:

My question is now: What does the significant interaction effect of the covariate (lapses_t01) and time-factor/repeated measures variable (Testday) mean?

I have searched this site repeatedly already, but was unable to draw a firm conclusion from the explanations on any similar questions I could find. A few even seem to contradict each other; some say this just means that the influence of the covariate varies depending on Testday, while others say that this indicates a violation of ANCOVA assumptions.

I would be very grateful to learn how to interpret this result correctly.

Answer 1

Instead of getting caught up in ANCOVA terminology, remember that it's just a particular form of a linear regression model. With ANCOVA, you have a continuous outcome (here, lapses, but see below), some categorical predictor(s) of primary interest (here, Condition and Testday), and some other (typically continuous) predictor(s) to control for (here, lapses_t01). The model, however, treats all the predictors equivalently.

When you include an interaction term in such a model, you are allowing for the possibility that the regression coefficient for one predictor depends on the level of another predictor (and necessarily vice-versa). A "statistically significant" interaction coefficient suggests that such interdependence of those regression coefficients is the case. In your example, the association of lapses with lapses_t01 depends on the Testday, and the the association of lapses with Testday depends on the lapses_t01 value.

I suppose that significant interaction violates "ANCOVA assumptions" in that you can't evaluate the predictions for combinations of your primary variables of interest, Condition and Testday, without also knowing the value of lapses_t01.

But that doesn't mean that the overall regression model is incorrect. With an interaction like that, you can illustrate the model predictions for relevant combinations of predictors. You can also evaluate the magnitude of the interaction, to determine whether there is a practical significance to the statistical significance.

You can't get those details from the ANOVA table you show. The model, however, contains the information you need in the individual regression coefficient estimates and their covariance matrix. Tools for post-modeling analysis, like the emmeans package, make it straightforward to evaluate the detailed implications of your model.

A couple of thoughts on the model

First, you might discuss with your statistical consultant whether a linear regression model is appropriate for your lapses outcome. As they are counts and not continuous, the assumptions underlying the model might not hold, particularly if the counts are few. Count data can be analyzed with a generalized linear model based on Poisson or negative binomial statistics.

Second, think about the implications of your missing data. You might need to consider multiple imputation to avoid bias if the data aren't missing completely at random. Stef van Buuren's book is a superb explanation of the issues, with tools provided by his mice package and by Frank Harrell's rms package and associated Hmisc package.

It's also possible, however, that the way you structured your ANCOVA exacerbated the problem with missing data. I'm not familiar with the rstatix package that you used; it might have forced you into the form of a repeated measures ANCOVA that requires all subjects to have data for all time points. Depending on the structure of your missing data, a mixed model with the lme4 package or generalized least squares regression might allow you to work directly with all the data that you acquired.

Finally, be wary of Type III tests. See this page for an explanation of the different ANOVA types, and the Warning on the help page for the Anova() function in the car package.