Search Results

@DikranMarsupial mentioned nested CV an alternative is repeated CV already during your hyperparameter optimization since that allows you to detect instability in the predictions already during your hyperparameter … DOI: 10.1007/s00216-007-1818-6) you can combine these approaches and do nested repeated CV Can overfitting happen even with using cross-validation for hyperparameter optimisation? Yes. …

Yes, you can repeat the cross validations (both or any of them, as necessary). Repeated cross validation improves the estimate of generalization error in a specific way: it allows you to easily measu …

From a certain point of view, optimizing the meta- or hyperparameter is the opposite approach to ensemble models. They are good for sifferent situations. … In that case, you may want to combine both approaches and build a bagged predictor with the optimized hyperparameter set. …

why are most classification learning curves plotted against number of samples? not only most: all. The reason is simply that the definition of learning curve is that it is the predictive performa …

Is there a correct way / order to do [two kind of hyperparameter optimization]? … Here, they do interact => optimize together You can also optimize sequentially, but that should then become an iterative procedure: optimize one type of hyperparameter optimize the other repeat until …

If this uncertainty tells you that there's no way you can do the data-driven hyperparameter optimization you want to, it's a complete waste to nevertheless try it. … If your sample size is so small that you face trouble during optimization, running a 5- or 10-fold cross validation on a coarser hyperparameter grid with only a/5 or 1/10 of the hyperparameter sets to …

. : x x x ... x x x t t t ... t t t r r r r r b with x = unused, t = inner training, r = red = inner testing = hyperparameter tuning, b = outer testing Then: The random uncertainty of the RMSE depends …

One variable at a time is not going to work unless you know the hyperparameter in question does not interact with other of your hyperparameters. …

latter is IMHO crucial if the result is to be used for optimization: while for a final model stability may be established in several ways and I'm OK with saying that only stable models will be considered, hyperparameter …

Yes it holds for preprocessing as well.

It seems straightforward, that you ONLY want to test the parameters, and the less variance, the better Well, it isn't that straightforward. @MatthewGunn already explained that it typically won't …

First of all, it is crucial to realize that the overfitting described in the Cawley paper arises from selecting the model with apparently best performance where determining performance is subject to …

However, which features to use is just a hyperparameter of your modeling process - you can include it with the optimization of the other hyperparameters. …

An inner e.g. cross validation takes care of the hyperparameter optimization. … The crucial step/assumption here to solve the dilemma of whose hyperparameter set should be chosen is to require the optimization to be stable. …

That is, the hyperparameter tuning is part of "the method for building the model". … Though as you note other people have a different view (without hyperparameter tuning). …