I am running a binary logistic regression analysis in R software using rms
package. Due to the small sample size (n=96), I used all data as training data. Given that I don't have test data, what method do you suggest for evaluating/validating the model?
1 Answer
The author of the rms
package often recommends a bootstrap procedure instead of using an explicit holdout set
The idea is that, when you have a holdout set, you deprive the model of precious training data. Perhaps this is not such a big deal when you have billions of samples, but you do not. However, if you just fit to the training data and go with whatever parameter estimates you get, you have no sense of if you have overfit.
Enter bootstrap.
The idea is to train the model on all of the data. Then evaluate on your metric of choice, say log loss or Brier score.
Then you select a bootstrap sample of your data train on that sample, and apply the trained model to the entire data set. Evaluate this model using the same evaluation metric, and compare the performance of this bootstrap-trained model to the performance of the model trained on all observations. Repeat, repeat, repeat. You now have a sense of by how much you have overfit and if that amount is acceptable.
The function rms::validate
will be your friend for this.