# Different ways to define survey design object under MCAR assumption

I have a stratified random sample, and would like to conduct complete-case analysis, assuming Missing Completely At Random. However, I find that there seem to be two ways to define survey design objects, as illustrated below:

library(survey)
library(tidyr)
library(dplyr)

data(api)

### Approach 1 ###
# a stratified random sample
dstrat <-
svydesign(
id =  ~ 1,
strata =  ~ stype,
weights =  ~ pw,
data = apistrat,
fpc =  ~ fpc
)

# calculate acs.core (has 94 missing values in apistrat )
svymean( ~ acs.core, subset(dstrat,!is.na(acs.core)))
svyby(~ acs.core, ~ awards, subset(dstrat,!is.na(acs.core)), svymean)


### Approach 2 ###
# adjust sampling weights 'pw' assuming MCAR within each stratum

itemresponse_prob <- apistrat %>%
group_by(stype) %>%
summarise(itemresponse_prob =
sum(!is.na(acs.core)) / n())

apistrat_new <- apistrat %>%
left_join(itemresponse_prob, by = "stype") %>%
mutate(itemresponse_prob = ifelse(is.na(acs.core), 0,
itemresponse_prob))

apistrat_new$$pw_new <- ifelse( apistrat_new$$itemresponse_prob == 0, 0,
apistrat_new$$pw*( 1/apistrat_new$$itemresponse_prob ) )

sum(apistrat$$pw) sum(apistrat_new$$pw_new)

dstrat_new <-
svydesign(
id =  ~ 1,
strata =  ~ stype,
weights =  ~ pw_new,
data = apistrat_new[!is.na(apistrat_new\$acs.core),],
fpc =  ~ fpc
)

# calculate acs.core
svymean( ~ acs.core, dstrat_new)
svyby(~ acs.core, ~ awards, dstrat_new, svymean)


Their results are quite different.

1. It seems more reasonable to do Approach 1, because it captures the real sampling process, that is, a group of people were sampled and some of them have missing data. On the other hand, Approach 2 assumes that the remaining sample with complete data is the original sample, which is not actually true. Is this reasoning be correct?
2. If Approach 1 is more suitable, by subsetting survey design and removing individuals with missing data on the variable of interest, is the underlying assumption Missing Completely At Random?
3. Which Approach is better? It seems that Approach 2 is conceptually wrong.

Any thoughts or comments would be highly appreciated! Many thanks for your help! :)

Approach 1 is what would normally be called a complete case analysis; it will be valid under MCAR.

Approach 2 should give better results. It treats the missing as random conditional on stype; it might more often be described as post-stratified on stype

Strictly speaking, it would be better to acknowledge the two different types of data-not-thereness. There's a sampling phase, then there's a subsequent missingness phase. Approach 3 would use the twophase function in the survey package, treat the sampling as sampling with known probabilities for the first phase and treat the missingness as sampling with estimated probabilities based on stype as the second phase.

Here, d2 doesn't use any weights at phase 2, but cal2 weights at phase 2 based on stype. You could use other variables as well

d2<-twophase(id=list(~1,~1), strata=list(~stype,~stype),
method="simple", weights=list(~pw, NULL),
subset=~!is.na(acs.core), data=apistrat)
cal2<-calibrate(d2, formula=~stype,phase=2)
svyby(~ acs.core, ~ awards, d2, svymean)
awards acs.core        se
No      No 28.05286 0.4411495
Yes    Yes 25.17013 1.3890701

svyby(~ acs.core, ~ awards, cal2, svymean)
awards acs.core        se
No      No 28.05286 0.4423269
Yes    Yes 25.17013 1.3891629


In this example, the two-phase approach actually agrees with approach 2. More generally it would not -- eg if you estimated the non-response weights based on variables that weren't sampling strata.

• Hi Prof. Lumley. Thank you very much again for your helpful insights. I don't know if I understand correctly, Approach 2 seems more specific about the missing mechanism (which is MCAR within each sampling stratum); if Approach 2 works better, then Approach 1 seems less specific about the missing mechanism (curious about the underlying assumptions behind subsetting survey design object based on missingness). Would it be reasonable to say that, for complete-case analysis (assuming MCAR), approach 2 would be more specific about the missing mechanism and therefore preferred? Many thanks! Commented Oct 19, 2023 at 13:45
• I tried to apply the two phase to my dataset as you illustrated above, but unfortunately came across an error Error when using calibrate function: (function (classes, fdef, mtable): unable to find an inherited method for function ‘calibrate’ for signature ‘"twophase"’. It may be due to the dataset itself. May I ask if there is a way around this error? Commented Oct 19, 2023 at 13:48
• That sounds like you have another package with a calibrate function (probably an S4 one) that's masking the one in the survey package Commented Oct 19, 2023 at 20:31
• Many thanks for this! Commented Oct 21, 2023 at 19:48