How can I use LOOCV to compare several different methods and measure how well they will generalize outside the N=150 sample?

Question

I have a dataset with around N=150 people described in the dataset and I have a large amount of information about each person. I want my model to be interpretable; that is, I want to be able to explain why my model has got the result it has.

I would like to do some exploratory analysis on a dataset to find a method that will predict outcomes in the dataset. I could try a potentially large numbers of different possible designs until I find a model that fits the dataset.

I understand that simply using k-fold or LOO cross-validation to identify the best-performing design will still over-fit my model to the sample and prevent me from finding the model that best generalizes outside the sample to the population.

What is a better method?

I plan to try:

1. Cut my sample in half to N=75 on each side. This gives me very little data to train on but I don't know how else I can approach the task.
2. Test repeatedly on the training set, and identify a model that works best on my training set.
3. Use LOOCV to identify the best-performing model on the training set.
4. Then test that model's performance on the test set. If I test multiple models on the test set and compare them, I will need to somehow penalize my results as this would bias the test upwards.

Question 1: Is there any way to improve on this plan?

There are a couple of possible ways I think I might be able to improve on this.

a) A slight improvement on the test set method would be that once a model has been selected, I retrain that same model, using the same method, across the entire dataset minus 1, and repeat holding out one across the test set. So, LOOCV except that we hold out only across the test set. Then I use this test to report the overall accuracy of the model.

b) I want to report on the usefulness of the model itself for diagnosis. To do that, I could retrain the model once on the entire dataset, using the same method as before, and use this as a better more representative model for generalization. However I could not use this final model to estimate performance for generalizing to novel samples.

Question 2: Are these plausibly improvements, or are there problems with these proposed improvements?

Answer 1

Ben - welcome to the show!

A few things: A) you want an estimate of the generalization error using the 150 data points you have. B)Using 10 fold cross validation you would be able to estimate that (and you would end up using all of the data you have). It might seem counter intuitive, but it is a kosher. C) You dont want to test repeated ly on a static test set - very unkosher. The model learns the test set, but gives you a underestimate of the generalization error. D) here is a recipe.

• Take the data set and split them randomly into 10 equal chunks. Number them 1 to 10.

• Round 1, keep the chunk 1 as test set, use chunk 2 to fine tune any parameters and use chunks 3 to 10 as training. Compute the error on test.

• Round 2, keep chunk 2 as test set, and chunk 3 as validation, use the rest of the chunks for training. Compute error on chunk 2 (your new test set).

Repeat 10 times (proceeding to round 3 etc as above) and average the 10 errors on the test set. You now ahve a robust estimate of generalization error. You could use a similar approach with leave one out. If you still feel nervous, revisit the first step and again using a different random seed split the data into 10 and repeat. You should see consistent results.

If you are in python world take a look at sklearn it has a class for cross validation.

To your other question - interpretability. Depends on the model you are using. Linear SVMs will ofer some interpretability. CNNs not as much, but you can focus on the components that are most predictive or start with features you think are most intuitive and see if they perform well.

And to your yet other question - which precise model specification should I go with as a result of the sample? Pick the specification with the lowest generalization error and estimate that on the entire data set. Score the new sample.