I have a dataset with around N=150 people described in the dataset and I have a large amount of information about each person. I want my model to be interpretable; that is, I want to be able to explain why my model has got the result it has.
I would like to do some exploratory analysis on a dataset to find a method that will predict outcomes in the dataset. I could try a potentially large numbers of different possible designs until I find a model that fits the dataset.
I understand that simply using k-fold or LOO cross-validation to identify the best-performing design will still over-fit my model to the sample and prevent me from finding the model that best generalizes outside the sample to the population.
What is a better method?
I plan to try:
- Cut my sample in half to N=75 on each side. This gives me very little data to train on but I don't know how else I can approach the task.
- Test repeatedly on the training set, and identify a model that works best on my training set.
- Use LOOCV to identify the best-performing model on the training set.
- Then test that model's performance on the test set. If I test multiple models on the test set and compare them, I will need to somehow penalize my results as this would bias the test upwards.
Question 1: Is there any way to improve on this plan?
There are a couple of possible ways I think I might be able to improve on this.
a) A slight improvement on the test set method would be that once a model has been selected, I retrain that same model, using the same method, across the entire dataset minus 1, and repeat holding out one across the test set. So, LOOCV except that we hold out only across the test set. Then I use this test to report the overall accuracy of the model.
b) I want to report on the usefulness of the model itself for diagnosis. To do that, I could retrain the model once on the entire dataset, using the same method as before, and use this as a better more representative model for generalization. However I could not use this final model to estimate performance for generalizing to novel samples.
Question 2: Are these plausibly improvements, or are there problems with these proposed improvements?