49

The answer depends on whether you are assuming the symmetric or asymmetric dirichlet distribution (or, more technically, whether the base measure is uniform). Unless something else is specified, most implementations of LDA assume the distribution is symmetric. For the symmetric distribution, a high alpha-value means that each document is likely to contain a ...


41

Random search has a probability of 95% of finding a combination of parameters within the 5% optima with only 60 iterations. Also compared to other methods it doesn't bog down in local optima. Check this great blog post at Dato by Alice Zheng, specifically the section Hyperparameter tuning algorithms. I love movies where the underdog wins, and I love ...


37

It's common to find code snippets that treat $T$ as a hyper-parameter, and attempt to optimize over it in the same way as any other hyper-parameter. This is just wasting computational power: when all other hyper-parameters are fixed, the model’s loss stochastically decreases as the number of trees increases. Intuitive explanation Each tree in a random ...


26

As the lead developer of Optunity I'll add my two cents. We have done extensive benchmarks comparing Optunity with the most popular Bayesian solvers (e.g., hyperopt, SMAC, bayesopt) on real-world problems, and the results indicate that PSO is in fact not less efficient in many practical cases. In our benchmark, which consists of tuning SVM classifiers on ...


24

There are basically four methods: Manual Search: Using knowledge you have about the problem guess parameters and observe the result. Based on that result tweak the parameters. Repeat this process until you find parameters that work well or you run out of time. Grid Search: Using knowledge you have about the problem identify ranges for the hyperparameters. ...


21

I suspect what is meant by hyper-parameter depends on the context, but here goes: I would say that the parameters of a model are those that are directly fitted to the data, and the hyper-parameters are those parameters that are set by the user or which are indirectly fitted to the data. For instance in ridge regression, the parameters are the regression ...


19

Unfortunately, it doesn't work that way. Hyperparameters cooperate in hard-to-predict ways. For example, a bit extreme to make the point. You have no hidden layers, in other words, you are fitting a logistic regression. A logistic regression will usually not really overfit. So you use a relatively big learning rate and a lot of epochs, and find that that ...


18

The term hyperparameter is pretty vague. I will use it to refer to a parameter that is in a higher level of the hierarchy than the other parameters. For an example, consider a regression model with a known variance (1 in this case) $$ y \sim N(X\beta,I) $$ and then a prior on the parameters, e.g. $$ \beta \sim N(0,\lambda I) $$ Here $\lambda$ ...


17

The effects of this bias can be very great. A good demonstration of this is given by the open machine learning competitions that feature in some machine learning conferences. These generally have a training set, a validation set and a test set. The competitors don't get to see the labels for either the validation set or the test set (obviously). The ...


16

This is how I have trained a xgboost classifier with a 5-fold cross-validation to optimize the F1 score using randomized search for hyperparameter optimization. Note that X and y here should be pandas dataframes. from scipy import stats from xgboost import XGBClassifier from sklearn.model_selection import RandomizedSearchCV, KFold from sklearn.metrics ...


15

In addition to Jim's (+1) answer: For some classifiers, the hyper-parameter values are dependent on the number of training examples, for instance for a linear SVM, the primal optimization problem is $\mathrm{min} \frac12\|w\|^2 + C\sum_{i=1}^\ell \xi_i$ subject to $y_i(x_i\cdot w _ b) \geq 1 - \xi_i, \quad \mathrm{and} \quad \xi_i \geq 0 \quad \forall i$ ...


14

A wide variety of methods exist. They can be largely partitioned in random/undirected search methods (like grid search or random search) and direct methods. Be aware, though, that they all require testing a considerable amount of hyperparameter settings unless you get lucky (hundreds at least, depends on the number of parameters). In the class of direct ...


14

Like you already observed yourself, your choice of features (feature selection) may have an impact on which hyperparameters for your algorithm are optimal, and which hyperparameters you select for your algorithm may have an impact on which choice of features would be optimal. So, yes, if you really really care about squeezing every single percent of ...


14

As a result of doing that you will also overfit the validation set (the more so the more you tuned the hyperparameters - if you tried two or three configurations, the effect is less than if you did some systematic search e.g. using the Gaussian process approach). The standard solution to this would be to not just have a training and validation set, but a ...


12

Number of trees is not a parameter that should be tuned, but just set large enough usually. There is no risk of overfitting in random forest with growing number of trees, as they are trained independently from each other. See our paper for more information about this: https://arxiv.org/abs/1705.05654 Max depth is a parameter that most of the times should be ...


12

The decision threshold creates a trade-off between the number of positives that you predict and the number of negatives that you predict -- because, tautologically, increasing the decision threshold will decrease the number of positives that you predict and increase the number of negatives that you predict. The decision threshold is not a hyper-parameter in ...


11

David Blei has a great talk introducing LDA to students of a summer class: http://videolectures.net/mlss09uk_blei_tm/ In the first video he covers extensively the basic idea of topic modelling and how Dirichlet distribution come into play. The plate notation is explained as if all hidden variables are observed to show the dependencies. Basically topics are ...


11

5x2cv as far as I have seen in the literature, always refer to a 5 repetition of a 2-fold. There is no nesting at all. do a 2-fold (50/50 split between train and test), repeat it 4 more times. The 5x2cv was popularized by the paper Approximate statistical tests for comparing supervised classification learning algorithms by Dietterich as a way of obtaining ...


11

Is hyperparameter tuning on sample of dataset a bad idea? A: Yes, because you risk overfitting (the hyperparameters) on that specific test set resulting from your chosen train-test split. Do I limit my classification accuracy? A: Yes, but common machine learning wisdom is: with your optimal hyperparameters, say $\lambda^*$, refit your model(s) on the ...


11

Here's the "default" nested cross-validation procedure to compare between a fixed set of models (e.g. grid search): Randomly split the dataset into $K$ folds. For $i$ from 1 to $K$: Let test be fold $i$. Let trainval be all the data except that which is in test. Randomly split trainval into $L$ subfolds $(i, 1), (i, 2), \dots, (i, L)$ . So, each subfold $(...


11

results are sensitive to parameters of the surrogate model, which are typically fixed at some value; this underestimates uncertainty; or else you have to be fully Bayesian and marginalize over hyper parameter distributions, which can be expensive and unwieldy. it takes a dozen or so samples to get a good surrogate surface in 2 or 3 dimensions of search space;...


11

Information is leaked because you're using the validation data to make hyper-parameter choices. Essentially, you're creating a complicated optimization problem: minimize the loss over hyper-parameters $\phi$ as evaluated against the validation data, where these hyper-parameters regularize a neural network model that has parameters $\theta$ trained by use of ...


10

Look no further! Yoshua Bengio published one of my favorite applied papers, one that I recommend to all new machine learning engineers when they start training neural nets: Practical recommendations for gradient-based training of deep architectures. To get his perspective on hyperparameter turning: including learning rate, learning rate schedule, early ...


10

A hyperparameter is simply a parameter that impacts, completely or partly, other parameters. They do not directly solve the optimization problem you face, but rather optimize parameters that can solve the problem (hence the hyper, because they are not part of the optimization problem, but rather are "addons"). For what I've seen, but I have no reference, ...


10

But varying the threshold will change the predicted classifications. Does this mean the threshold is a hyperparameter? Yup, it does, sorta. It's a hyperparameter of you decision rule, but not the underlying regression. If so, why is it (for example) not possible to easily search over a grid of thresholds using scikit-learn's GridSearchCV method (as you ...


10

the parameter values just simply dependent on the data This is the key part of your question. This is where you are confused. Yes, the parameter values depend on the data. But the data are fixed when we fit a model. In other words, we fit a model conditional on the observations. It does not make sense to compare the complexity of different models that were ...


9

Look again at the graphic from the paper (Figure 1). Say that you have two parameters, with 3x3 grid search you check only three different parameter values from each of the parameters (three rows and three columns on the plot on the left), while with random search you check nine (!) different parameter values of each of the parameters (nine distinct rows and ...


9

I think that the answer here is the same as everywhere in data science: it depends on the data :-) It might happen that one method outperforms another (here https://arimo.com/data-science/2016/bayesian-optimization-hyperparameter-tuning/ people compare Bayesian hyperparameter optimization and achieve a better result on the San Francisco crime kaggle ...


8

A hyperparameter is a parameter for the (prior) distribution of some parameter. So for a simple example, let's say we state that the variance parameter $\tau^2$ in some problem has a uniform prior on $(0,\theta)$. (I personally would be unlikely to do such a thing, but it happens; I might in some very particular circumstance) Then $\tau^2$ is a parameter (...


8

Nested cross validation explained without nesting Here's how I see (nested) cross validation and model building. Note that I'm chemist and like you look from the application side to the model building process (see below). My main point here is from my point of view I don't need a dedicated nested variety of cross validation. I need a validation method (e.g. ...


Only top voted, non community-wiki answers of a minimum length are eligible