3
$\begingroup$

I am running a binary logistic regression analysis in R software using rms package. Due to the small sample size (n=96), I used all data as training data. Given that I don't have test data, what method do you suggest for evaluating/validating the model?

$\endgroup$

1 Answer 1

5
$\begingroup$

The author of the rms package often recommends a bootstrap procedure instead of using an explicit holdout set

The idea is that, when you have a holdout set, you deprive the model of precious training data. Perhaps this is not such a big deal when you have billions of samples, but you do not. However, if you just fit to the training data and go with whatever parameter estimates you get, you have no sense of if you have overfit.

Enter bootstrap.

The idea is to train the model on all of the data. Then evaluate on your metric of choice, say log loss or Brier score.

Then you select a bootstrap sample of your data train on that sample, and apply the trained model to the entire data set. Evaluate this model using the same evaluation metric, and compare the performance of this bootstrap-trained model to the performance of the model trained on all observations. Repeat, repeat, repeat. You now have a sense of by how much you have overfit and if that amount is acceptable.

The function rms::validate will be your friend for this.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.