Timeline for How can I use LOOCV to compare several different methods and measure how well they will generalize outside the N=150 sample?
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 5, 2018 at 18:03 | vote | accept | Ben Smith | ||
| Dec 5, 2018 at 3:03 | comment | added | pythOnometrist | Make a list of models you would like to try. Focus first on rediction models. So try all models that can predict your dependent variable/label. SVM, regression SGD , logistic regression etc. PCA, SVD or NMF could be used for dimensionality reduction before running the prediction models. Run the horse race(i.e do the cross validation approach on all the prediction models) and then pick the model specification that has the lowest generalization error. I ofen run 30 plus models in this manner and pick the best based on generalization error. | |
| Dec 5, 2018 at 0:25 | comment | added | Ben Smith | Thanks for responding very quickly! That sounds like a great way to estimate the generalizability of a particular model. But I haven't started yet. I don't know what sort of model I am going to use (a linear model; which to include? Do I use PCA or another dimensionality reduction method? I'll need to make these decisions as I go). In your method, it seems like there's no stage when I'll be doing that. Have I missed something? | |
| Dec 5, 2018 at 0:15 | history | edited | pythOnometrist | CC BY-SA 4.0 |
added 239 characters in body
|
| Dec 5, 2018 at 0:08 | history | answered | pythOnometrist | CC BY-SA 4.0 |