Search Results

Choose $λ=λ^∗$ to minimize the average OOS mean square error. This strategy assumes you have enough independent test cases so the error on your OOS estimate is negligible. You are right: if the …

It seems straightforward, that you ONLY want to test the parameters, and the less variance, the better Well, it isn't that straightforward. @MatthewGunn already explained that it typically won't …

A model that does not change much if a small part of the training data is exchanged against a few other training cases is called stable, and I'd extend this also to hyperparameters. You can check th …

If this uncertainty tells you that there's no way you can do the data-driven hyperparameter optimization you want to, it's a complete waste to nevertheless try it. … If your sample size is so small that you face trouble during optimization, running a 5- or 10-fold cross validation on a coarser hyperparameter grid with only a/5 or 1/10 of the hyperparameter sets to …

Yes it holds for preprocessing as well.

An inner e.g. cross validation takes care of the hyperparameter optimization. … The crucial step/assumption here to solve the dilemma of whose hyperparameter set should be chosen is to require the optimization to be stable. …

You can keep the risk low by evaluating only very few models for fixing the regularization hyperparameter plus going for a low complexity within the plausible choice. …

From a certain point of view, optimizing the meta- or hyperparameter is the opposite approach to ensemble models. They are good for sifferent situations. … In that case, you may want to combine both approaches and build a bagged predictor with the optimized hyperparameter set. …

However, which features to use is just a hyperparameter of your modeling process - you can include it with the optimization of the other hyperparameters. …

Is there a correct way / order to do [two kind of hyperparameter optimization]? … Here, they do interact => optimize together You can also optimize sequentially, but that should then become an iterative procedure: optimize one type of hyperparameter optimize the other repeat until …

The most important downside for searching along single parameters instead of optimizing them all together is that you ignore interactions. It is quite common that e.g. more than one parameter influenc …

latter is IMHO crucial if the result is to be used for optimization: while for a final model stability may be established in several ways and I'm OK with saying that only stable models will be considered, hyperparameter …

One variable at a time is not going to work unless you know the hyperparameter in question does not interact with other of your hyperparameters. …

Think of optimizing the "training algorithm" hyperparameter. …

why are most classification learning curves plotted against number of samples? not only most: all. The reason is simply that the definition of learning curve is that it is the predictive performa …