2
$\begingroup$

Context

I have a randomized controlled trial with three groups and three time measurements (pre, post, follow-up). I noticed that from pre to post, and then from post to follow-up, the attrition rates seem different by group. A reviewer asks whether the differential attrition is statistically significant.

Reprex

Here is example data:

data <- data.frame(
  group = c("Control", "Treatment1", "Treatment2"),
  pre = c(150, 150, 150),
  post = c(150, 100, 80),
  follow = c(120, 90, 70)
)
data
#>        group pre post follow
#> 1    Control 150  150    120
#> 2 Treatment1 150  100     90
#> 3 Treatment2 150   80     70

Created on 2023-11-10 with reprex v2.0.2

Question

Can I really do a statistical test with just 6 values at a time? I was thinking of two tests: one for attrition from pre to post, and another test for attrition from post to follow-up. If so, what test?

I read from this source and this source that there are two primary tests used in this context: (a) differential attrition rate tests and (b) selective attrition tests. Are these tests appropriate for my data? And how can I conduct these tests in R?

$\endgroup$
1
  • 2
    $\begingroup$ Why not turn this into a survival analysis problem and use a log rank test? $\endgroup$ Commented Nov 11, 2023 at 4:23

2 Answers 2

3
$\begingroup$

For the Differential Attrition Rate test, you could consider a Chi-Squared Test of Independence, if you have sufficiently large samples - with small samples, this test has low power. An alternative is Fisher's Exact Test. Here's how we can do a Differential Attrition Rate test for pre to post, and post to follow-up

# Calculate the number of dropouts and completers at each stage
data$dropout_pre_post <- data$pre - data$post
data$dropout_post_follow <- data$post - data$follow

# Create a contingency table for pre to post
# This is a test of attrition from pre to post:
table_pre_post <- matrix(c(data$pre, data$dropout_pre_post), ncol = 2)

# Fisher's Exact Test
fisher.test(table_pre_post)

# And similarly for post to follow-up:
# Create a contingency table for post to follow-up
table_post_follow <- matrix(c(data$post, data$dropout_post_follow), ncol = 2)

# Fisher's Exact Test
fisher.test(table_post_follow)

With only three groups and small numbers, power could be an issue. These tests might not detect a difference unless it is quite large. Be cautious with the interpretation of the p-values, especially with small sample sizes. If possible, supplement these tests with a qualitative assessment of attrition reasons, which might provide additional insights into whether attrition is likely to be differential or selective.

For the Selective Attrition Test, which assesses whether the characteristics of the dropouts are different from those who remained in the study, you could use a logistic regression model which can be used to model the probability of dropout as a function of group and other relevant covariates.

Another, perhaps more appropriate possibility would be a survival analysis, as suggested by Demetri Pananos in the comment to the question. This would likely require a reshaping of your data, but is specifically designed for "time to event" data, which you seem to have.

$\endgroup$
4
  • $\begingroup$ Thanks! In the example data I provided, the smallest cell is 70, which is bigger than 5, so technically should the chi squared test be OK instead of Fisher? In your example, I see you compare pre to dropout_pre_post instead of just post, so the smallest value in this contingency table is actually 10, which is a lot lower. Both strategies provide different p values. Why must we use the values of the dropouts rather than of the remaining participants? $\endgroup$
    – rempsyc
    Commented Nov 13, 2023 at 2:03
  • $\begingroup$ Also, in your example, the first test is significant, and the other one isn't. If you would convert this example into a reprex that includes the result, we could also include the interpretation of the result, which I believe would be useful to future readers. If I am not mistaken, p < .05 for the first test means that we reject the hypothesis of independence (of rows and columns), i.e., the attrition rates differ by group (i.e., there is differential attrition), whereas for the second test, p > .05, so there is no differential attrition. Is that correct? $\endgroup$
    – rempsyc
    Commented Nov 13, 2023 at 16:08
  • 1
    $\begingroup$ I met a statistician about this who suggested that the above example is "wrong" because instead of comparing pre to dropout_pre_post, we should be comparing post to dropout_pre_post so that the row total equals the original sample size. I am unmarking this as answer until this is corrected or clarified. $\endgroup$
    – rempsyc
    Commented Nov 20, 2023 at 20:43
  • 1
    $\begingroup$ Agree with @rempsyc that the table to run the test on should count Remaining vs. Not-Remaining for each group, not Total vs. Remaining. This also doesn't strike me as a terribly small sample - with 150 samples in each group, the 95% CI of the true follow-up rate is +/-8% at worst, and only +/-3% at best. $\endgroup$ Commented Nov 20, 2023 at 21:08
0
$\begingroup$

It was also suggested, in a meeting with a statistician, that one could compare means and use t tests (or Mann–Whitney U tests for small samples where the assumption of normality becomes difficult to assess) to compare dropouts and completers on the dependent variables of interest to assess whether the dropouts were different or not than the completers on those variables.

Example reprex:

library(report)

# Prepare data
data <- mtcars
names(data)[2] <- "group"
names(data)[9] <- "dropout"
data$group <- as.factor(ifelse(data$group == 4, "Control",
                            ifelse(data$group == 6, "Treatment1",
                               "Treatment2")))
data$dropout <- as.factor(data$dropout)

# t test
x <- t.test(data$mpg ~ data$dropout)
report(x)
#> Effect sizes were labelled following Cohen's (1988) recommendations.
#> 
#> The Welch Two Sample t-test testing the difference of data$mpg by data$dropout
#> (mean in group 0 = 17.15, mean in group 1 = 24.39) suggests that the effect is
#> negative, statistically significant, and large (difference = -7.24, 95% CI
#> [-11.28, -3.21], t(18.33) = -3.77, p = 0.001; Cohen's d = -1.41, 95% CI [-2.26,
#> -0.53])

# Could also be checked for a specific group, if desired
data_Treatment2 <- data[data$group == "Treatment2", ]

x <- t.test(data_Treatment2$mpg ~ data_Treatment2$dropout)
report(x)
#> Effect sizes were labelled following Cohen's (1988) recommendations.
#> 
#> The Welch Two Sample t-test testing the difference of data_Treatment2$mpg by
#> data_Treatment2$dropout (mean in group 0 = 15.05, mean in group 1 = 15.40)
#> suggests that the effect is negative, statistically not significant, and very
#> small (difference = -0.35, 95% CI [-2.34, 1.64], t(10.19) = -0.39, p = 0.704;
#> Cohen's d = -0.17, 95% CI [-1.05, 0.71])

Created on 2023-11-20 with reprex v2.0.2

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.