Permuting the formula argument to Hmisc:aregImpute - how to evaluate? I just ran across David Norris' comment about aregImpute and formula order in this post: Permuting the formula argument to Hmisc:aregImpute
My question: how can you tell that the solution is fairly "stable" and that order is not having undue influence?
Thanks!
 A: @user94240, I would take the view that uncertainty about the stability of the model fit under alternative choices of formula permutation is just one kind of 'model uncertainty'. Each possible permutation of the formula arguments corresponds to a (slightly, it is hoped) different model. On this view, you could augment your confidence intervals by repeated sampling from the set of formula permutations. My function was designed for the large-n case, and incorporates sampling (with replacement) through the sample() function. But if there aren't too many variables in the formula, then you might not be forced to sample the permutations, but could instead generate and check all of them.
To put this in more operational terms, you could use densityplot() to inspect the distribution of your parameter estimates. The variance of these estimates would be added to the bootstrap-estimated variance you get for the individual models. (When you code it up, you'll be bootstrapping the bootstrap!)
If your question was less about the mechanics, and more about how to make the judgement that a given amount of model uncertainty is excessive, I don't think much can be offered by way of a general rule. What can be said, however, is that users of your analysis should be told about all sources of uncertainty that you can account for objectively. Let's say you've done an extensive multiple bias modeling[1],[2] exercise, and discovered that the uncertainty revealed by 'bootstrapping' the formula permutation turns out to be a sizable portion of the overall uncertainty. In that case, you might decide you need to replace the off-the-shelf aregImpute approach with a custom-built imputation model that is specifically adapted to your problem, using your domain-specific knowledge about the causes and patterns of missingness.
[1]: Greenland, Sander. “Multiple-Bias Modelling for Analysis of Observational Data (with Discussion).” Journal of the Royal Statistical Society: Series A (Statistics in Society) 168, no. 2 (March 2005): 267–306. doi:10.1111/j.1467-985X.2004.00349.x.
[2]: Lash, Timothy L., Matthew P. Fox, and Aliza K. Fink. Applying Quantitative Bias Analysis to Epidemiologic Data. Statistics for Biology and Health. New York, NY: Springer New York, 2009. http://link.springer.com/10.1007/978-0-387-87959-8.
