How complex can the substantive model be in multiple imputation with smcfcs

Question

We are interested in using a substantive model compatible fully conditional specification (smcfcs) for multiple imputations of missing data. I believe this approach will lead to more unbiased estimates than for example mice, as our analysis will consist of Cox proportional hazards regressions including interaction terms.

In smcfcs the substantive model has to be compatible with the imputation model (the imputation model needs to be nested in the substantive model). As we are also interested in non-linear relationships of continuous variables and want to includes splines, we are wondering how complex we can make the substantive model (how many variables can we include).

The substantive model will be a Cox proportional hazards regression and there are several rules of thumb in survival analysis for the complexity of a model: e.g. 10 (or maybe even fewer) events per variable.

We were wondering whether:

• Do these same rules of thumb apply to the substantive model in smcfcs?
• Do you base the number of events on the complete cases or the complete data?
• We have quite a large fraction of missing data (35% of cases), would you want to be more conservative in such cases (i.e. more events per variable)?

It would be great if you could point me to any literature that is available on this topic.

Answer 1

The "rule of thumb" of about 15 events per predictor (Section 4.4 of Harrell's course notes) is to prevent overfitting. This attempts to ensure that the within-imputation variance estimates aren't overly optimistic.

In your situation, that rule of thumb holds for each application of the Cox model. As you are performing multiple imputation, all cases with events will be included in each of the Cox models, so the total number of cases with events would inform how flexible a model you could try.

The fraction of missing data, in contrast, might be expected to have its major influence on the between-imputation variance estimates, as the imputed data sets might then differ considerably. Depending on how well the imputation works, a high fraction of missing data might also lead to high within-imputation variance.

With respect to smcfcs, carefully consider whether that will really help in your situation and, if so, how best to apply it. As Bartlett et al put it in their 2015 paper introducing substantive model compatible fully conditional specification (SMC-FCS):

in many settings auxiliary variables V may be available, which although not involved in the substantive model, may be useful for inclusion in imputation models in order to improve efficiency (by virtue of their association with variables being imputed) or to increase the plausibility of the MAR assumption. The notion of compatibility between imputation and substantive models does not then apply, since the two models involve different sets of variables. However, one could include the auxiliary variables V as additional covariates in the model for Y in the SMC-FCS algorithm, following which models for [outcome] Y can be fitted using the imputed datasets which omit V.

So your statement that "the imputation model needs to be nested in the substantive model" isn't correct; it might best be the other way around. In general, it makes sense to use as much relevant information as you have available for imputing the missing data, even if many of the predictors used for imputation won't be used in the substantive model. For the multiple imputation itself you don't need to worry about overfitting, unlike for the substantive model.

Stef van Buuren's book on Flexible Imputation of Missing Data might be the best single, easily accessible reference. Chapter 3 of Harrell's course notes, linked above, also discusses missing data. As an example, the aregImpute() function in his R rms package uses extremely flexible models with bootstrapping for multiple imputation. So even if you go with smcfcs, don't limit your imputation to predictors that you will include in your Cox model.

A potential danger in smcfcs is that, if your substantive model isn't well specified, your use of the same model for imputation could tend to freeze in both imputations and a model that aren't properly specified and thus won't well represent the situation in the underlying population. See Section 4.5.5 of van Buuren's book linked above.

Also, in reading the Bartlett et al 2015 paper, I was struck by how similar the results from standard fully conditional specification (FCS) and SMC-FCS were in the Cox model on actual data with many predictors available. The poor performance reported with standard FCS was for a Cox model with simulated data involving only 2 predictors, which didn't provide a lot of flexibility for imputation.

You might consider the combination of bootstrapping with imputation recently discussed by Bartlett and Hughes as a way to deal with uncongeniality (incompatibility) and model misspecification. That is implemented in the bootImpute package in R, with options for imputation via either FCS and SMC-FCS.