I have a dataset with around N=150 people described in the dataset and I have a large amount of information about each person. I want my model to be interpretable; that is, I want to be able to explain why my model has got the result it has.

I would like to do some exploratory analysis on a dataset to find a method that will predict outcomes in the dataset. I could try a potentially large numbers of different possible designs until I find a model that fits the dataset.

I understand that simply using k-fold or LOO cross-validation to identify the best-performing design will still over-fit my model to the sample and prevent me from finding the model that best generalizes outside the sample to the population.

What is a better method?

I plan to try:

  1. Cut my sample in half to N=75 on each side. This gives me very little data to train on but I don't know how else I can approach the task.
  2. Test repeatedly on the training set, and identify a model that works best on my training set.
  3. Use LOOCV to identify the best-performing model on the training set.
  4. Then test that model's performance on the test set. If I test multiple models on the test set and compare them, I will need to somehow penalize my results as this would bias the test upwards.

Question 1: Is there any way to improve on this plan?

There are a couple of possible ways I think I might be able to improve on this.

a) A slight improvement on the test set method would be that once a model has been selected, I retrain that same model, using the same method, across the entire dataset minus 1, and repeat holding out one across the test set. So, LOOCV except that we hold out only across the test set. Then I use this test to report the overall accuracy of the model.

b) I want to report on the usefulness of the model itself for diagnosis. To do that, I could retrain the model once on the entire dataset, using the same method as before, and use this as a better more representative model for generalization. However I could not use this final model to estimate performance for generalizing to novel samples.

Question 2: Are these plausibly improvements, or are there problems with these proposed improvements?


Ben - welcome to the show!

A few things: A) you want an estimate of the generalization error using the 150 data points you have. B)Using 10 fold cross validation you would be able to estimate that (and you would end up using all of the data you have). It might seem counter intuitive, but it is a kosher. C) You dont want to test repeated ly on a static test set - very unkosher. The model learns the test set, but gives you a underestimate of the generalization error. D) here is a recipe.

  • Take the data set and split them randomly into 10 equal chunks. Number them 1 to 10.
  • Round 1, keep the chunk 1 as test set, use chunk 2 to fine tune any parameters and use chunks 3 to 10 as training. Compute the error on test.

  • Round 2, keep chunk 2 as test set, and chunk 3 as validation, use the rest of the chunks for training. Compute error on chunk 2 (your new test set).

Repeat 10 times (proceeding to round 3 etc as above) and average the 10 errors on the test set. You now ahve a robust estimate of generalization error. You could use a similar approach with leave one out. If you still feel nervous, revisit the first step and again using a different random seed split the data into 10 and repeat. You should see consistent results.

If you are in python world take a look at sklearn it has a class for cross validation.

To your other question - interpretability. Depends on the model you are using. Linear SVMs will ofer some interpretability. CNNs not as much, but you can focus on the components that are most predictive or start with features you think are most intuitive and see if they perform well.

And to your yet other question - which precise model specification should I go with as a result of the sample? Pick the specification with the lowest generalization error and estimate that on the entire data set. Score the new sample.

  • $\begingroup$ Thanks for responding very quickly! That sounds like a great way to estimate the generalizability of a particular model. But I haven't started yet. I don't know what sort of model I am going to use (a linear model; which to include? Do I use PCA or another dimensionality reduction method? I'll need to make these decisions as I go). In your method, it seems like there's no stage when I'll be doing that. Have I missed something? $\endgroup$
    – Ben Smith
    Dec 5 '18 at 0:25
  • $\begingroup$ Make a list of models you would like to try. Focus first on rediction models. So try all models that can predict your dependent variable/label. SVM, regression SGD , logistic regression etc. PCA, SVD or NMF could be used for dimensionality reduction before running the prediction models. Run the horse race(i.e do the cross validation approach on all the prediction models) and then pick the model specification that has the lowest generalization error. I ofen run 30 plus models in this manner and pick the best based on generalization error. $\endgroup$ Dec 5 '18 at 3:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.