Skip to main content
92 votes
Accepted

Do we have to tune the number of trees in a random forest?

It's common to find code snippets that treat $T$ as a hyper-parameter, and attempt to optimize over it in the same way as any other hyper-parameter. This is just wasting computational power: when all ...
Sycorax's user avatar
  • 91.6k
63 votes
Accepted

Practical hyperparameter optimization: Random vs. grid search

Random search has a probability of 95% of finding a combination of parameters within the 5% optima with only 60 iterations. Also compared to other methods it doesn't bog down in local optima. Check ...
Firebug's user avatar
  • 19.3k
34 votes
Accepted

How should Feature Selection and Hyperparameter optimization be ordered in the machine learning pipeline?

Like you already observed yourself, your choice of features (feature selection) may have an impact on which hyperparameters for your algorithm are optimal, and which hyperparameters you select for ...
Dennis Soemers's user avatar
26 votes

How to use XGboost.cv with hyperparameters optimization?

This is how I have trained a xgboost classifier with a 5-fold cross-validation to optimize the F1 score using randomized search for hyperparameter optimization. ...
darXider's user avatar
  • 491
24 votes

How to build the final model and tune probability threshold after nested cross-validation?

Nested cross validation explained without nesting Here's how I see (nested) cross validation and model building. Note that I'm chemist and like you look from the application side to the model ...
cbeleites unhappy with SX's user avatar
20 votes
Accepted

Is hyperparameter tuning on sample of dataset a bad idea?

In addition to Jim's (+1) answer: For some classifiers, the hyper-parameter values are dependent on the number of training examples, for instance for a linear SVM, the primal optimization problem is ...
Dikran Marsupial's user avatar
20 votes
Accepted

How to tune hyperparameters in a random forest

Number of trees is not a parameter that should be tuned, but just set large enough usually. There is no risk of overfitting in random forest with growing number of trees, as they are trained ...
PhilippPro's user avatar
  • 1,135
20 votes
Accepted

Are optimal hyperparameters still optimal for a deeper neural net architecture?

Unfortunately, it doesn't work that way. Hyperparameters cooperate in hard-to-predict ways. For example, a bit extreme to make the point. You have no hidden layers, in other words, you are fitting a ...
Gijs's user avatar
  • 3,644
19 votes

Is hyperparameter tuning on sample of dataset a bad idea?

Is hyperparameter tuning on sample of dataset a bad idea? A: Yes, because you risk overfitting (the hyperparameters) on that specific test set resulting from your chosen train-test split. Do I ...
Jim's user avatar
  • 2,172
18 votes

What's in a name: hyperparameters

The term hyperparameter is pretty vague. I will use it to refer to a parameter that is in a higher level of the hierarchy than the other parameters. For an example, consider a regression model with a ...
jaradniemi's user avatar
  • 4,701
18 votes
Accepted

How to obtain optimal hyperparameters after nested cross validation?

Overview As @RockTheStar correctly concluded in the commentaries, the nested cross-validation is used only to access the model performance estimate. Dissociated from that, to find the best ...
Firebug's user avatar
  • 19.3k
18 votes
Accepted

What exactly is tol (tolerance) used as stopping criteria in sklearn models?

As you noted, tol is the tolerance for the stopping criteria. This tells scikit to stop searching for a minimum (or maximum) ...
ilanman's user avatar
  • 4,787
18 votes
Accepted

Step-by-step explanation of K-fold cross-validation with grid search to optimise hyperparameters

Here's the "default" nested cross-validation procedure to compare between a fixed set of models (e.g. grid search): Randomly split the dataset into $K$ folds. For $i$ from 1 to $K$: Let ...
Danica's user avatar
  • 25k
17 votes

Is it a bad practice to learn hyperparameters from the training data set?

It's completely inappropriate to pick hyperparameters from the test set, if it's meant to be a test set, because it makes the results on the test set unreliable. I.e. even if the test set is perfectly ...
Björn's user avatar
  • 32.9k
16 votes

Hyper parameters tuning: Random search vs Bayesian optimization

I think that the answer here is the same as everywhere in data science: it depends on the data :-) It might happen that one method outperforms another (here https://arimo.com/data-science/2016/...
Fabian Werner's user avatar
16 votes

Is decision threshold a hyperparameter in logistic regression?

But varying the threshold will change the predicted classifications. Does this mean the threshold is a hyperparameter? Yup, it does, sorta. It's a hyperparameter of you decision rule, but not the ...
Matthew Drury's user avatar
15 votes

Practical hyperparameter optimization: Random vs. grid search

Look again at the graphic from the paper (Figure 1). Say that you have two parameters, with 3x3 grid search you check only three different parameter values from each of the parameters (three rows and ...
Tim's user avatar
  • 139k
15 votes

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

In regards to the evidence in regards to the claim, I believe the only evidence supporting the claim can be found on figure 4 in their paper. They show the final results under a range of different ...
Cliff AB's user avatar
  • 21.2k
15 votes
Accepted

How simple should a Baseline model be?

Mainly, how to do this is a question of experience. This will also tell you what kind of model is a good candidate for such a baseline. For instance, in time series forecasting, the simplest models, ...
Stephan Kolassa's user avatar
14 votes

What are some of the disavantage of bayesian hyper parameter optimization?

results are sensitive to parameters of the surrogate model, which are typically fixed at some value; this underestimates uncertainty; or else you have to be fully Bayesian and marginalize over hyper ...
Sycorax's user avatar
  • 91.6k
14 votes
Accepted

Hyperparameter Optimization Using Gaussian Processes

As a result of doing that you will also overfit the validation set (the more so the more you tuned the hyperparameters - if you tried two or three configurations, the effect is less than if you did ...
Björn's user avatar
  • 32.9k
13 votes
Accepted

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

Information is leaked because you're using the validation data to make hyper-parameter choices. Essentially, you're creating a complicated optimization problem: minimize the loss over hyper-parameters ...
Sycorax's user avatar
  • 91.6k
11 votes

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

Adam learns the learning rates itself, on a per-parameter basis. The parameters $\beta_1$ and $\beta_2$ don't directly define the learning rate, just the timescales over which the learned learning ...
Hugh Perkins's user avatar
  • 4,797
11 votes

Why don't we just learn the hyper parameters?

"Why don't we just learn the hyper parameters?" It's a great question! I'll try to provide a more general answer. The TL;DR answer is that you can definitely learn hyperparameters, just not from the ...
galoosh33's user avatar
  • 2,302
11 votes
Accepted

Extensive hyperparameter tuning yields nothing, XGBoost classifier

Your datasets sounds at a superficial level reasonable large, so I would normally expect some value from hyperparameter tuning and in small datasets the right amount of regularization can be rather ...
Björn's user avatar
  • 32.9k
10 votes

What's in a name: hyperparameters

A hyperparameter is simply a parameter that impacts, completely or partly, other parameters. They do not directly solve the optimization problem you face, but rather optimize parameters that can solve ...
gaborous's user avatar
  • 757
10 votes

How to get hyper parameters in nested cross validation?

(I'm sure I wrote most of this already in some answer - but can't find it right now. If anyone stumbles across that answer, please link it). I see 2 slightly different approaches here, which I think ...
cbeleites unhappy with SX's user avatar
10 votes

How to speed up hyperparameter optimization?

Here are some general techniques to speed up hyperparameter optimization. If you have a large dataset, use a simple validation set instead of cross validation. This will increase the speed by a ...
user20160's user avatar
  • 32.7k
10 votes

Hyper parameters tuning: Random search vs Bayesian optimization

Bayesian optimization is better, because it makes smarter decisions. You can check this article in order to learn more: Hyperparameter optimization for neural networks. This articles also has info ...
itdxer's user avatar
  • 7,799
10 votes
Accepted

Confused in selecting the number of hidden layers and neurons in an MLP for a binary classification problem

In Deep Learning there are no hard & fast rules to set the number of layers, the number of hidden units per layer and not even the kind of connections between layers: who claims the contrary often ...
DeltaIV's user avatar
  • 18k

Only top scored, non community-wiki answers of a minimum length are eligible