72

It's common to find code snippets that treat $T$ as a hyper-parameter, and attempt to optimize over it in the same way as any other hyper-parameter. This is just wasting computational power: when all other hyper-parameters are fixed, the model’s loss stochastically decreases as the number of trees increases. Intuitive explanation Each tree in a random ...


52

Random search has a probability of 95% of finding a combination of parameters within the 5% optima with only 60 iterations. Also compared to other methods it doesn't bog down in local optima. Check this great blog post at Dato by Alice Zheng, specifically the section Hyperparameter tuning algorithms. I love movies where the underdog wins, and I love machine ...


26

This is how I have trained a xgboost classifier with a 5-fold cross-validation to optimize the F1 score using randomized search for hyperparameter optimization. Note that X and y here should be pandas dataframes. from scipy import stats from xgboost import XGBClassifier from sklearn.model_selection import RandomizedSearchCV, KFold from sklearn.metrics ...


26

As the lead developer of Optunity I'll add my two cents. We have done extensive benchmarks comparing Optunity with the most popular Bayesian solvers (e.g., hyperopt, SMAC, bayesopt) on real-world problems, and the results indicate that PSO is in fact not less efficient in many practical cases. In our benchmark, which consists of tuning SVM classifiers on ...


26

Like you already observed yourself, your choice of features (feature selection) may have an impact on which hyperparameters for your algorithm are optimal, and which hyperparameters you select for your algorithm may have an impact on which choice of features would be optimal. So, yes, if you really really care about squeezing every single percent of ...


25

There are basically four methods: Manual Search: Using knowledge you have about the problem guess parameters and observe the result. Based on that result tweak the parameters. Repeat this process until you find parameters that work well or you run out of time. Grid Search: Using knowledge you have about the problem identify ranges for the hyperparameters. ...


21

I suspect what is meant by hyper-parameter depends on the context, but here goes: I would say that the parameters of a model are those that are directly fitted to the data, and the hyper-parameters are those parameters that are set by the user or which are indirectly fitted to the data. For instance in ridge regression, the parameters are the regression ...


20

The effects of this bias can be very great. A good demonstration of this is given by the open machine learning competitions that feature in some machine learning conferences. These generally have a training set, a validation set and a test set. The competitors don't get to see the labels for either the validation set or the test set (obviously). The ...


20

Unfortunately, it doesn't work that way. Hyperparameters cooperate in hard-to-predict ways. For example, a bit extreme to make the point. You have no hidden layers, in other words, you are fitting a logistic regression. A logistic regression will usually not really overfit. So you use a relatively big learning rate and a lot of epochs, and find that that ...


18

The term hyperparameter is pretty vague. I will use it to refer to a parameter that is in a higher level of the hierarchy than the other parameters. For an example, consider a regression model with a known variance (1 in this case) $$ y \sim N(X\beta,I) $$ and then a prior on the parameters, e.g. $$ \beta \sim N(0,\lambda I) $$ Here $\lambda$ ...


18

Number of trees is not a parameter that should be tuned, but just set large enough usually. There is no risk of overfitting in random forest with growing number of trees, as they are trained independently from each other. See our paper for more information about this: https://arxiv.org/abs/1705.05654 Max depth is a parameter that most of the times should be ...


17

Nested cross validation explained without nesting Here's how I see (nested) cross validation and model building. Note that I'm chemist and like you look from the application side to the model building process (see below). My main point here is from my point of view I don't need a dedicated nested variety of cross validation. I need a validation method (e.g. ...


17

Is hyperparameter tuning on sample of dataset a bad idea? A: Yes, because you risk overfitting (the hyperparameters) on that specific test set resulting from your chosen train-test split. Do I limit my classification accuracy? A: Yes, but common machine learning wisdom is: with your optimal hyperparameters, say $\lambda^*$, refit your model(s) on the ...


17

In addition to Jim's (+1) answer: For some classifiers, the hyper-parameter values are dependent on the number of training examples, for instance for a linear SVM, the primal optimization problem is $\mathrm{min} \frac12\|w\|^2 + C\sum_{i=1}^\ell \xi_i$ subject to $y_i(x_i\cdot w _ b) \geq 1 - \xi_i, \quad \mathrm{and} \quad \xi_i \geq 0 \quad \forall i$ ...


15

5x2cv as far as I have seen in the literature, always refer to a 5 repetition of a 2-fold. There is no nesting at all. do a 2-fold (50/50 split between train and test), repeat it 4 more times. The 5x2cv was popularised by the paper Approximate statistical tests for comparing supervised classification learning algorithms by Dietterich as a way of obtaining ...


15

Here's the "default" nested cross-validation procedure to compare between a fixed set of models (e.g. grid search): Randomly split the dataset into $K$ folds. For $i$ from 1 to $K$: Let test be fold $i$. Let trainval be all the data except that which is in test. Randomly split trainval into $L$ subfolds $(i, 1), (i, 2), \dots, (i, L)$ . So, each subfold $(...


14

A wide variety of methods exist. They can be largely partitioned in random/undirected search methods (like grid search or random search) and direct methods. Be aware, though, that they all require testing a considerable amount of hyperparameter settings unless you get lucky (hundreds at least, depends on the number of parameters). In the class of direct ...


14

results are sensitive to parameters of the surrogate model, which are typically fixed at some value; this underestimates uncertainty; or else you have to be fully Bayesian and marginalize over hyper parameter distributions, which can be expensive and unwieldy. it takes a dozen or so samples to get a good surrogate surface in 2 or 3 dimensions of search space;...


14

As a result of doing that you will also overfit the validation set (the more so the more you tuned the hyperparameters - if you tried two or three configurations, the effect is less than if you did some systematic search e.g. using the Gaussian process approach). The standard solution to this would be to not just have a training and validation set, but a ...


14

The decision threshold creates a trade-off between the number of positives that you predict and the number of negatives that you predict -- because, tautologically, increasing the decision threshold will decrease the number of positives that you predict and increase the number of negatives that you predict. The decision threshold is not a hyper-parameter in ...


13

Look again at the graphic from the paper (Figure 1). Say that you have two parameters, with 3x3 grid search you check only three different parameter values from each of the parameters (three rows and three columns on the plot on the left), while with random search you check nine (!) different parameter values of each of the parameters (nine distinct rows and ...


13

In regards to the evidence in regards to the claim, I believe the only evidence supporting the claim can be found on figure 4 in their paper. They show the final results under a range of different values for $\beta_1$, $\beta_2$ and $\alpha$. Personally, I don't find their argument convincing, in particular because they do not present results across a ...


12

I think that the answer here is the same as everywhere in data science: it depends on the data :-) It might happen that one method outperforms another (here https://arimo.com/data-science/2016/bayesian-optimization-hyperparameter-tuning/ people compare Bayesian hyperparameter optimization and achieve a better result on the San Francisco crime kaggle ...


12

Information is leaked because you're using the validation data to make hyper-parameter choices. Essentially, you're creating a complicated optimization problem: minimize the loss over hyper-parameters $\phi$ as evaluated against the validation data, where these hyper-parameters regularize a neural network model that has parameters $\theta$ trained by use of ...


12

But varying the threshold will change the predicted classifications. Does this mean the threshold is a hyperparameter? Yup, it does, sorta. It's a hyperparameter of you decision rule, but not the underlying regression. If so, why is it (for example) not possible to easily search over a grid of thresholds using scikit-learn's GridSearchCV method (as you ...


11

Adam learns the learning rates itself, on a per-parameter basis. The parameters $\beta_1$ and $\beta_2$ don't directly define the learning rate, just the timescales over which the learned learning rates decay. If they decay really fast, then the learning rates will jump about all over the place. If they decay slowly, it will take ages for the learning rates ...


11

As you noted, tol is the tolerance for the stopping criteria. This tells scikit to stop searching for a minimum (or maximum) once some tolerance is achieved, i.e. once you're close enough. tol will change depending on the objective function being minimized and the algorithm they use to find the minimum, and thus will depend on the model you are fitting. ...


10

Look no further! Yoshua Bengio published one of my favorite applied papers, one that I recommend to all new machine learning engineers when they start training neural nets: Practical recommendations for gradient-based training of deep architectures. To get his perspective on hyperparameter turning: including learning rate, learning rate schedule, early ...


10

A hyperparameter is simply a parameter that impacts, completely or partly, other parameters. They do not directly solve the optimization problem you face, but rather optimize parameters that can solve the problem (hence the hyper, because they are not part of the optimization problem, but rather are "addons"). For what I've seen, but I have no reference, ...


10

"Why don't we just learn the hyper parameters?" It's a great question! I'll try to provide a more general answer. The TL;DR answer is that you can definitely learn hyperparameters, just not from the same data. Read on for a slightly more detailed reply. A hyperparameter typically corresponds to a setting of the learning algorithm, rather than one of its ...


Only top voted, non community-wiki answers of a minimum length are eligible