Timeline for Is this a proper way to perform cross validation?
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 29, 2020 at 20:15 | comment | added | Ryan Volpi | @user5965026 You are correct and I believe I worded that poorly. K-Fold CV can be applied to estimate the performance of multiple models, or the same model with different hyperparameters, and that data can be used for model selection. The distinction I wanted to make is that K-Fold CV itself is not a model selection algorithm or procedure like grid search. | |
| Jun 29, 2020 at 20:10 | comment | added | user5965026 | @einar I used to read articles on medium and towardsdatascience before I realized that many of them are just not credible and wrong, and those are typically pretty high up on Page Rank, but I'm not sure if this applies to CV. The authors in ESL also said that even many peer-reviewed published papers perform CV wrongly. | |
| Jun 29, 2020 at 20:08 | comment | added | einar | I encountered someone using this approach only a few days ago. Wondering whether there is some high-ranked google hit on CV promoting this nonsense | |
| Jun 29, 2020 at 19:58 | comment | added | user5965026 | @RyanVolpi In your first sentence you said "K-Fold CV is not for model selection." And in the second sentence "he results can be used to select between multiple models...". So doesn't the second part imply that K-Fold CV is in a way used for model selection? So when you use CV for something like finding the optimal regularization parameter, isn't that "model selection?" | |
| Jun 29, 2020 at 19:54 | comment | added | Michael M | That sounds very wrong then! | |
| Jun 29, 2020 at 19:53 | comment | added | Ryan Volpi | This approach is flawed. K-Fold CV is not for model selection, it is for model evaluation. The results can be used to select between multiple models that you evaluated individually, but for a given model, the various folds are just used for evaluating the models out of sample performance. The estimated parameters and metrics of each fold are not used for anything | |
| Jun 29, 2020 at 19:51 | history | edited | gunes |
edited tags
|
|
| Jun 29, 2020 at 19:51 | answer | added | gunes | timeline score: 2 | |
| Jun 29, 2020 at 19:41 | comment | added | user5965026 | @MichaelM Unfortunately, I don't have his presentation slides, so everything I'm stating is from my memory of the presentation. When I read my question and think about what he did, I feel like it doesn't make sense. He basically suggested that he found $k$ sets of $\hat{\beta}$ OLS parameters. For future modeling purposes, he chose the $\hat{\beta}$ that gave the smallest prediction error observed during the k-fold validation. | |
| Jun 29, 2020 at 19:34 | comment | added | Michael M | I have not yet understood the second part. Could you outline the process with an example as simple as possible? | |
| Jun 29, 2020 at 19:25 | history | asked | user5965026 | CC BY-SA 4.0 |