Before/After unpaired study with control villages analysed in R

Question

For the implementation of a project, we have been following household access to services in two groups of villages, one an intervention group (I) and one a control group (C). Access as a Yes/No variable was collected before and after an intervention with an interval of five years. The sample of households in each group of villages can be considered random and self-weighted.

Given we have four variables as below:
I_before n=544 households
I_after n=547 households unpaired as it is difficult to find the same household back after 5 years

---

C_before n=253 households
C_after n=257 households unpaired as well

• What type of analysis is recommended to establish the effect size in % points and
• how to establish whether this change is statistically significant?
• What preconditions of the data to check (if any, like e.g. normality) to ensure the validity of the used test.

How would the command in R look like to do the analysis using the above variable names?

Thanks a lot for any help

Answer 1

You would want to do something like a factorial ANOVA and test whether the interaction between intervention group and time period is significant. If it is, then you know that the effect of time period depends on whether the village was in the intervention group or control. That is the effects you are interested in (to determine whether any observed change across time periods is not due simply to maturation). This method is only valid for estimating the causal effect of the intervention and the time period change if the villages in each group truly have been randomly chosen and are independent (i.e., don't structurally share members).

In R, install the car package. The run

car::Anova(aov(y ~ intervention*period, data = data_i), type = 3, white.adjust = TRUE)

You can follow up by looking at the group means to describe the effect. If the interaction is significant, you can say the different in the after and before means varies between the intervention and control group.

With sample sizes that large, typically the results are robust to moderate violations of normality. For severe violations, consider modeling your outcome with a generalized lienar model (more advanced) or transforming the outcome to make it more normal. Setting white.adjust = TRUE makes the analysis robust to violation of equal variance among groups.