Statistically testing for differential attrition in R?

Question

Context

I have a randomized controlled trial with three groups and three time measurements (pre, post, follow-up). I noticed that from pre to post, and then from post to follow-up, the attrition rates seem different by group. A reviewer asks whether the differential attrition is statistically significant.

Reprex

Here is example data:

data <- data.frame(
  group = c("Control", "Treatment1", "Treatment2"),
  pre = c(150, 150, 150),
  post = c(150, 100, 80),
  follow = c(120, 90, 70)
)
data
#>        group pre post follow
#> 1    Control 150  150    120
#> 2 Treatment1 150  100     90
#> 3 Treatment2 150   80     70

^{Created on 2023-11-10 with reprex v2.0.2}

Question

Can I really do a statistical test with just 6 values at a time? I was thinking of two tests: one for attrition from pre to post, and another test for attrition from post to follow-up. If so, what test?

I read from this source and this source that there are two primary tests used in this context: (a) differential attrition rate tests and (b) selective attrition tests. Are these tests appropriate for my data? And how can I conduct these tests in R?

Answer 1

For the Differential Attrition Rate test, you could consider a Chi-Squared Test of Independence, if you have sufficiently large samples - with small samples, this test has low power. An alternative is Fisher's Exact Test. Here's how we can do a Differential Attrition Rate test for pre to post, and post to follow-up

# Calculate the number of dropouts and completers at each stage
data$dropout_pre_post <- data$pre - data$post
data$dropout_post_follow <- data$post - data$follow

# Create a contingency table for pre to post
# This is a test of attrition from pre to post:
table_pre_post <- matrix(c(data$pre, data$dropout_pre_post), ncol = 2)

# Fisher's Exact Test
fisher.test(table_pre_post)

# And similarly for post to follow-up:
# Create a contingency table for post to follow-up
table_post_follow <- matrix(c(data$post, data$dropout_post_follow), ncol = 2)

# Fisher's Exact Test
fisher.test(table_post_follow)

With only three groups and small numbers, power could be an issue. These tests might not detect a difference unless it is quite large. Be cautious with the interpretation of the p-values, especially with small sample sizes. If possible, supplement these tests with a qualitative assessment of attrition reasons, which might provide additional insights into whether attrition is likely to be differential or selective.

For the Selective Attrition Test, which assesses whether the characteristics of the dropouts are different from those who remained in the study, you could use a logistic regression model which can be used to model the probability of dropout as a function of group and other relevant covariates.

Another, perhaps more appropriate possibility would be a survival analysis, as suggested by Demetri Pananos in the comment to the question. This would likely require a reshaping of your data, but is specifically designed for "time to event" data, which you seem to have.

Answer 2

It was also suggested, in a meeting with a statistician, that one could compare means and use t tests (or Mann–Whitney U tests for small samples where the assumption of normality becomes difficult to assess) to compare dropouts and completers on the dependent variables of interest to assess whether the dropouts were different or not than the completers on those variables.

Example reprex:

library(report)

# Prepare data
data <- mtcars
names(data)[2] <- "group"
names(data)[9] <- "dropout"
data$group <- as.factor(ifelse(data$group == 4, "Control",
                            ifelse(data$group == 6, "Treatment1",
                               "Treatment2")))
data$dropout <- as.factor(data$dropout)

# t test
x <- t.test(data$mpg ~ data$dropout)
report(x)
#> Effect sizes were labelled following Cohen's (1988) recommendations.
#> 
#> The Welch Two Sample t-test testing the difference of data$mpg by data$dropout
#> (mean in group 0 = 17.15, mean in group 1 = 24.39) suggests that the effect is
#> negative, statistically significant, and large (difference = -7.24, 95% CI
#> [-11.28, -3.21], t(18.33) = -3.77, p = 0.001; Cohen's d = -1.41, 95% CI [-2.26,
#> -0.53])

# Could also be checked for a specific group, if desired
data_Treatment2 <- data[data$group == "Treatment2", ]

x <- t.test(data_Treatment2$mpg ~ data_Treatment2$dropout)
report(x)
#> Effect sizes were labelled following Cohen's (1988) recommendations.
#> 
#> The Welch Two Sample t-test testing the difference of data_Treatment2$mpg by
#> data_Treatment2$dropout (mean in group 0 = 15.05, mean in group 1 = 15.40)
#> suggests that the effect is negative, statistically not significant, and very
#> small (difference = -0.35, 95% CI [-2.34, 1.64], t(10.19) = -0.39, p = 0.704;
#> Cohen's d = -0.17, 95% CI [-1.05, 0.71])

^{Created on 2023-11-20 with reprex v2.0.2}