Search Results

Data-driven model selection is part of the training process: any kind of data used in order to obtain the final model is involved in training. … You'll find lots of questions and answers about this under the topics of (data-driven) model optimization and model selection. …

, nor data-driven model selection/tuning/optimization. … but data-driven model selection/tuning/optimization is possible as well, but more complex to construct. …

If the model performs well, you model it again on the complete training data set and this final model you then use to predict new values. … That would be data-driven model selection/optimization. If you want to do data-driven model selection/optimization, you need to validate (= measure performace) the chosen model. …

If you do data-driven model selection, you need to measure the performance of the finally selected model with a test set that is independent of the whole training (incl. data-driven model selection). … However, for data-driven optimization, proper scoring rules are the way to go - whatever other figures of merit you look at for the final validation. …

The independent validation is not supposed to increase the performance: it is supposed to measure the performance of the final model (and detect/monitor the optimistic bias introduced to model selection … during data-driven optimization such as cross validating for the "optimal" penalty). …

A couple of ways you could parse the quote (in statistics), assuming optimization refers to (data-driven) model selection: If you care about prediction, you may be better off with model averaging instead … If your dataset is small enough, you might not have enough data to fit the "true" or "best" model for the problem. What does it even mean to do model-selection well, in that case? …

The big problem with both model selection/optimization and data dredging is that people forget to validate their final model with independent test data. … The inner test data is used for model selection/optimization, and once that is finished, you do the outer testing with data that did in no way contribute to the model. …

Note that if you select a model based on the CV results, this model selection procedure (including the CV) is actually part of your training. … To reiterate: the problem is not the CV, the problem is the data-driven model optimization (selection). …

Data-driven model optimization/selection: from a statistical point of view, this will often require much larger test sample sizes than the final evaluation in order to avoid "skimming testing variance" … All in all, I'd expect that the optimization sets should typically be larger than the final evaluation sets - or that instead of data-driven optimization, reasonable hyperparameters may be fixed using …

I like to think of feature selection and model selection by cross validation as data-driven (i.e. by the inner CV results) optimization of the model. … As soon as you use the inner CV results to select/optimize the model, nested validation is necessary. …

Either optimize or measure model performance for validation purposes. … So data-driven optimization of λ is clearly part of the model training. …

Data-driven feature selection is IHMO best discussed as part of general hyperparameter optimization, and it will often be set up in a way that hugely extends the hyperparameter search space. … For your scenario (accuracy, 500 test cases, 80 features among which to select), you cannot afford data-driven feature selection. …

IMHO, the basic rules for data-driven model optimization applies here as well. … Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. …

Data-driven optimization means that information from the test samples enter your final model as you choose a model that performs well for these (CV) test sets. … Samples that were tested for parameter optimization or model selection are not independent any longer. And you should report the uncertainty on this final …

Like @Firebug, I'd recommend nested cross validation if you go for data-driven model selection (unless you have thousands of independent cases at hand, so that a 20 % split still gives you decent testing … estimate calculated using an optimization (validation) test set. …