leave-one-out sensitivity analysis for regression based population study Using a linear mixed model I observe significant associations, however, the coefficients are very subtle. To show that the model is not based on only a few observations I want to validate the model using a subsamplig approach e.g. leave one out sensitivity analysis.
So I iteratively remove a random  subject from the population and fit the model again. this iteration results in a matrix of coefficients.
What would be a proper way of combining these coefficients per iteration, and how would you compare the test coefficients to the ones from the original fit?
Many thanks for your help 
 A: Here's what I do:


*

*I usually treat this as purely a validation step. I.e. the final model has the set of coefficients I get for the full data set. 

*Combining the coefficients you get from the perturbation experiment you do in your sensitivity analysis yields another model. This so-called ensemble model is a legitimate model as well, it just isn't the same as the "normal" model. 

*I describle (I my case, typicall: plot) the variation of the coefficients I observe during the perturbation experiment and then mark the coefficients of the normal whole-data-set model in that plot. 

*I usually stop at a description of the observed coefficient distribution, because (at least with my data), this distribution again is a combination of the effects of several factors, such as: the cases I'm looking at, stability of the models, ...
This means that any further description (such as deriving confidence intervals) would require yet another modeling step. 



To show that the model is not based on only a few observations

This last point also means that you won't be able to get around the fact of having only a few observations. From my point of view, you can argue that with only few observations, you cannot afford to neglect any information that actually is in the data set (and the sensitivity analysis extracts information you would just throw away otherwise), but in my experience there's no way of cheating around the basic fact of having only limited sample size.  

Some points about the splitting:


*

*As you are looking at mixed models, you are already very much aware of structure in your data and model (which factors are crossed vs. nested, which are random and which are fixed).
The important point is to obey this structure also in the resampling for your validation: in order to test with independent cases, you need to properly separate your dependent/outcome factor.  


*

*The random factors require special care: you need to split independently for all random factors crossed with or nested below your dependent (for random factors nested below the dependent, splitting at the uppermost below the dependent will automatically be independent also for all factors nested further down). 

*Fixed factors are benign: the fact that you model them as fixed means you'll legitimately have knowledge of them for unknown cases and implies that you won't encounter unknown levels of this factor. No special care for resampling is needed (instead, the care was needed when deciding whether the factor should be fixed or random for this model). 


*If you leave out more than one case at a time, you can repeat/iterate your cross validation. This gives you an easy possibility of separating uncertainty due to limited sample size from uncertainty due to instability in the model fitting.  For simple situations with just the outcome and all random factors nested below that, you cannot get this from leave-one-out. If you have also crossed random factors, your leave one out will already be leave-one-this-and-one-that-out. 
