79 votes
Accepted

Do we have to tune the number of trees in a random forest?

It's common to find code snippets that treat $T$ as a hyper-parameter, and attempt to optimize over it in the same way as any other hyper-parameter. This is just wasting computational power: when all ...
user avatar
  • 80.4k
54 votes
Accepted

Practical hyperparameter optimization: Random vs. grid search

Random search has a probability of 95% of finding a combination of parameters within the 5% optima with only 60 iterations. Also compared to other methods it doesn't bog down in local optima. Check ...
user avatar
  • 15.9k
29 votes
Accepted

How should Feature Selection and Hyperparameter optimization be ordered in the machine learning pipeline?

Like you already observed yourself, your choice of features (feature selection) may have an impact on which hyperparameters for your algorithm are optimal, and which hyperparameters you select for ...
user avatar
27 votes
Accepted

Advantages of Particle Swarm Optimization over Bayesian Optimization for hyperparameter tuning?

As the lead developer of Optunity I'll add my two cents. We have done extensive benchmarks comparing Optunity with the most popular Bayesian solvers (e.g., hyperopt, SMAC, bayesopt) on real-world ...
user avatar
  • 17.6k
26 votes

How to use XGboost.cv with hyperparameters optimization?

This is how I have trained a xgboost classifier with a 5-fold cross-validation to optimize the F1 score using randomized search for hyperparameter optimization. ...
user avatar
  • 491
22 votes

What do we mean by hyperparameters?

I suspect what is meant by hyper-parameter depends on the context, but here goes: I would say that the parameters of a model are those that are directly fitted to the data, and the hyper-parameters ...
user avatar
20 votes
Accepted

How bad is hyperparameter tuning outside cross-validation?

The effects of this bias can be very great. A good demonstration of this is given by the open machine learning competitions that feature in some machine learning conferences. These generally have a ...
user avatar
20 votes

How to build the final model and tune probability threshold after nested cross-validation?

Nested cross validation explained without nesting Here's how I see (nested) cross validation and model building. Note that I'm chemist and like you look from the application side to the model ...
user avatar
20 votes
Accepted

How to tune hyperparameters in a random forest

Number of trees is not a parameter that should be tuned, but just set large enough usually. There is no risk of overfitting in random forest with growing number of trees, as they are trained ...
user avatar
  • 1,019
20 votes
Accepted

Are optimal hyperparameters still optimal for a deeper neural net architecture?

Unfortunately, it doesn't work that way. Hyperparameters cooperate in hard-to-predict ways. For example, a bit extreme to make the point. You have no hidden layers, in other words, you are fitting a ...
user avatar
  • 3,479
18 votes

What's in a name: hyperparameters

The term hyperparameter is pretty vague. I will use it to refer to a parameter that is in a higher level of the hierarchy than the other parameters. For an example, consider a regression model with a ...
user avatar
  • 4,521
18 votes

Is hyperparameter tuning on sample of dataset a bad idea?

Is hyperparameter tuning on sample of dataset a bad idea? A: Yes, because you risk overfitting (the hyperparameters) on that specific test set resulting from your chosen train-test split. Do I ...
user avatar
  • 2,022
17 votes
Accepted

Is hyperparameter tuning on sample of dataset a bad idea?

In addition to Jim's (+1) answer: For some classifiers, the hyper-parameter values are dependent on the number of training examples, for instance for a linear SVM, the primal optimization problem is ...
user avatar
16 votes
Accepted

Nested cross-validation - how is it different from model selection via kfold CV on the training set?

5x2cv as far as I have seen in the literature, always refer to a 5 repetition of a 2-fold. There is no nesting at all. do a 2-fold (50/50 split between train and test), repeat it 4 more times. The ...
user avatar
16 votes

Is decision threshold a hyperparameter in logistic regression?

The decision threshold creates a trade-off between the number of positives that you predict and the number of negatives that you predict -- because, tautologically, increasing the decision threshold ...
user avatar
  • 80.4k
15 votes
Accepted

What exactly is tol (tolerance) used as stopping criteria in sklearn models?

As you noted, tol is the tolerance for the stopping criteria. This tells scikit to stop searching for a minimum (or maximum) ...
user avatar
  • 4,573
15 votes
Accepted

Step-by-step explanation of K-fold cross-validation with grid search to optimise hyperparameters

Here's the "default" nested cross-validation procedure to compare between a fixed set of models (e.g. grid search): Randomly split the dataset into $K$ folds. For $i$ from 1 to $K$: Let ...
user avatar
  • 22.4k
15 votes
Accepted

How simple should a Baseline model be?

Mainly, how to do this is a question of experience. This will also tell you what kind of model is a good candidate for such a baseline. For instance, in time series forecasting, the simplest models, ...
user avatar
14 votes

Practical hyperparameter optimization: Random vs. grid search

Look again at the graphic from the paper (Figure 1). Say that you have two parameters, with 3x3 grid search you check only three different parameter values from each of the parameters (three rows and ...
user avatar
  • 117k
14 votes

Guideline to select the hyperparameters in Deep Learning

A wide variety of methods exist. They can be largely partitioned in random/undirected search methods (like grid search or random search) and direct methods. Be aware, though, that they all require ...
user avatar
  • 17.6k
14 votes

What are some of the disavantage of bayesian hyper parameter optimization?

results are sensitive to parameters of the surrogate model, which are typically fixed at some value; this underestimates uncertainty; or else you have to be fully Bayesian and marginalize over hyper ...
user avatar
  • 80.4k
14 votes
Accepted

Hyperparameter Optimization Using Gaussian Processes

As a result of doing that you will also overfit the validation set (the more so the more you tuned the hyperparameters - if you tried two or three configurations, the effect is less than if you did ...
user avatar
  • 23.5k
14 votes

Is decision threshold a hyperparameter in logistic regression?

But varying the threshold will change the predicted classifications. Does this mean the threshold is a hyperparameter? Yup, it does, sorta. It's a hyperparameter of you decision rule, but not the ...
user avatar
13 votes

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

In regards to the evidence in regards to the claim, I believe the only evidence supporting the claim can be found on figure 4 in their paper. They show the final results under a range of different ...
user avatar
  • 18.3k
13 votes

Hyper parameters tuning: Random search vs Bayesian optimization

I think that the answer here is the same as everywhere in data science: it depends on the data :-) It might happen that one method outperforms another (here https://arimo.com/data-science/2016/...
user avatar
12 votes
Accepted

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

Information is leaked because you're using the validation data to make hyper-parameter choices. Essentially, you're creating a complicated optimization problem: minimize the loss over hyper-parameters ...
user avatar
  • 80.4k
11 votes

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

Adam learns the learning rates itself, on a per-parameter basis. The parameters $\beta_1$ and $\beta_2$ don't directly define the learning rate, just the timescales over which the learned learning ...
user avatar
  • 4,349
10 votes

Guideline to select the hyperparameters in Deep Learning

Look no further! Yoshua Bengio published one of my favorite applied papers, one that I recommend to all new machine learning engineers when they start training neural nets: Practical recommendations ...
user avatar
  • 1,680
10 votes

What's in a name: hyperparameters

A hyperparameter is simply a parameter that impacts, completely or partly, other parameters. They do not directly solve the optimization problem you face, but rather optimize parameters that can solve ...
user avatar
  • 645
10 votes
Accepted

How to obtain optimal hyperparameters after nested cross validation?

Overview As @RockTheStar correctly concluded in the commentaries, the nested cross-validation is used only to access the model performance estimate. Dissociated from that, to find the best ...
user avatar
  • 15.9k

Only top scored, non community-wiki answers of a minimum length are eligible