How do I cross validate when I don't have a test set? Situation: I have two models, fitted on the same data.
Goal: I want to meaure the out-of-sample performance of the two models.
Problem: I don't have a test set, and the original data is too small to split it up  into a batch for training and a batch for testing. So my only recourse is to simulate new test data... but say I bootstrap new data from my original data set. This is not really an "out-of-sample" data, since it's simulated via bootstrap from the data I fitted on! In fact, there's a non-zero chance that my "simulated test data" ... is exactly the same as my original data!
So what do I do?
 A: Since the dataset is small, you can estimate out-of-sample error/performance via leave-one-out cross validation, and compare LOOCV performances of the two models.
Note that, alongside the benefits, LOOCV has its cons as well: It's computationally more expensive, and it may have larger variance in the performance metric. But, this problem can be mitigated via calculating the performance after assigning all the class labels and calculating the performance for the whole set. Also, we won't have a single model in the end.
A: You could use a Bayesian machine learning model. The distribution of your predictions shows you how accurate the model is. If your model overfits it will have a large variance in the predictions, because every model of the ensemble overfits differently. When you have enough data the distribution is very narrow and when you have infinite data equals the prediction of the non Bayesian model.
Every parameter of your model is now a distribution. Depending on the type of the model they can even be interpretable.
One easy way to obtain a Bayesian model is via Bayesian bootstrapping. You can either do that by training the model multiple times with different weights for each data point or by using a package like bayesian_bootstrap. The weights must be drawn from a Direchlet distribution (which in this case is a multivariate uniform distribution).
