Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
43
votes
6answers
21k views

Practical hyperparameter optimization: Random vs. grid search

I'm currently going through Bengio's and Bergsta's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving ...
38
votes
3answers
30k views

Guideline to select the hyperparameters in Deep Learning

I'm looking for a paper that could help in giving a guideline on how to choose the hyperparameters of a deep architecture, like stacked auto-encoders or deep believe networks. There are a lot of ...
31
votes
1answer
6k views

Do we have to tune the number of trees in a random forest?

Software implementations of random forest classifiers have a number of parameters to allow users to fine-tune the algorithm's behavior, including the number of trees in the forest. Is this a parameter ...
24
votes
3answers
8k views

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Bengio, Goodfellow and Courville: Adam is generally regarded as ...
21
votes
2answers
23k views

Natural interpretation for LDA hyperparameters

Can somebody explain what is the natural interpretation for LDA hyperparameters? ALPHA and BETA are parameters of Dirichlet ...
20
votes
4answers
5k views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
19
votes
5answers
2k views

What's in a name: hyperparameters

So in a normal distribution, we have two parameters: mean $\mu$ and variance $\sigma^2$. In the book Pattern Recognition and Machine Learning, there suddenly appears a hyperparameter $\lambda$ in the ...
19
votes
6answers
9k views

Is hyperparameter tuning on sample of dataset a bad idea?

I have a dataset of 140000 examples and 30 features for which I am training several classifiers for a binary classification (SVM, Logistic Regression, Random Forest etc) In many cases hyperparameter ...
18
votes
2answers
2k views

Advantages of Particle Swarm Optimization over Bayesian Optimization for hyperparameter tuning?

There's substantial contemporary research on Bayesian Optimization (1) for tuning ML hyperparameters. The driving motivation here is that a minimal number of data points are required to make informed ...
17
votes
1answer
2k views

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
17
votes
3answers
6k views

How to get hyper parameters in nested cross validation?

I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation: Nested cross validation for model selection ...
13
votes
2answers
3k views

Is decision threshold a hyperparameter in logistic regression?

Predicted classes from (binary) logistic regression are determined by using a threshold on the class membership probabilities generated by the model. As I understand it, typically 0.5 is used by ...
13
votes
3answers
6k views

How should Feature Selection and Hyperparameter optimization be ordered in the machine learning pipeline?

My objective is to classify sensor signals. The concept of my solution so far is : i) Engineering features from raw signal ii) Selecting relevant features with ReliefF and a clustering approach iii) ...
13
votes
3answers
9k views

Hyper parameters tuning: Random search vs Bayesian optimization

So, we know that random search works better than grid search, but a more recent approach is Bayesian optimization (using gaussian processes). I've looked up a comparison between the two, and found ...
13
votes
1answer
3k views

Choosing an appropriate minibatch size for stochastic gradient descent (SGD)

Is there any literature that examines the choice of minibatch size when performing stochastic gradient descent? In my experience, it seems to be an empirical choice, usually found via cross-...
12
votes
1answer
3k views

What do we mean by hyperparameters? [duplicate]

Can anyone give me full details about what we mean by hyperparameters, and what in the Dirichlet distribution are called hyperparameters? A practice example for the estimation of those parameters ...
12
votes
1answer
6k views

Hyperparameter tuning in Gaussian Process Regression

I am trying to tune the hyperparameters of the gaussian process regression algorithm I've implemented. I simply want to maximize the log marginal likelihood given by the formula $$\log(\mathbf{y}|X,\...
11
votes
1answer
3k views

Hyperprior density for hierarchical Gamma-Poisson model

In a hierarchical model of data $y$ where $$y \sim \textrm{Poisson}(\lambda)$$ $$\lambda \sim \textrm{Gamma}(\alpha, \beta)$$ it appears to be typical in practice to chose values ($\alpha, \beta)$ ...
10
votes
3answers
364 views

Why don't we just learn the hyper parameters?

I was implementing a pretty popular paper "EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES" and in the paper, it trains an adversarial objective function J''(θ) = αJ(θ) + (1 − α)J'(θ). It treats α ...
10
votes
1answer
3k views

Relation between learning rate and number of hidden layers?

Is there any rule of thumb between depth of a neural network and learning rate? I have been noticing that the deeper the network is, the lower the learning rate must be. If that's correct, why is ...
9
votes
1answer
911 views

Are optimal hyperparameters still optimal for a deeper neural net architecture?

I have found a set of optimal hyperparameters (e.g. learning rate for gradient descent) using cross validation and bayesian optimisation. While searching for the optimal hyperparameters, my neural net ...
9
votes
2answers
4k views

Nested cross-validation - how is it different from model selection via kfold CV on the training set?

I often see people talking about 5x2 cross-validation as a special case of nested cross validation. I assume the first number (here: 5) refers to the number of folds in the inner loop and the second ...
9
votes
1answer
156 views

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

In François Chollet's Deep Learning with Python it says: As a result, tuning the configuration of the model based on its performance on the validation set can quickly result in overfitting to ...
9
votes
1answer
339 views

In linear regression why does regularisation penalise the parameter values as well?

Currently learning ridge regression and I was a little confused about the penalisation of more complex models (or the definition of a more complex model). From what I understand, model complexity ...
8
votes
2answers
19k views

How to use XGboost.cv with hyperparameters optimization?

I want to optimize hyperparameters of XGboost using crossvalidation. However, it is not clear how to obtain the model from xgb.cv. For instance I call ...
8
votes
1answer
2k views

What are some of the disavantage of bayesian hyper parameter optimization?

I am fairly new to machine learning and statistics but I was wondering why bayesian optimization is not referred more often online when learning machine learning to optimize your algorithm ...
8
votes
2answers
4k views

Understanding early stopping in neural networks and its implications when using cross-validation

I'm a bit troubled and confused by the idea of how the technique early stopping is defined. If you take a look it Wikipedia, it is defined as follows: Split the training data into a training set and ...
8
votes
1answer
2k views

How to obtain optimal hyperparameters after nested cross validation?

In general, if we have a large dataset, we can split it into (1) training, (2) validation, and (3) test. We use validation to identify the best hyperparameters in cross validation (e.g., C in SVM) and ...
8
votes
1answer
365 views

Fully Bayesian hyper-parameter selection in GPML

Is it possible to perform an approximated fully Bayesian (1) selection of hyper-parameters (e.g. covariance scale) with the GPML code, instead of maximizing the marginal likelihood (2) ? I think using ...
7
votes
3answers
9k views

How to tune hyperparameters in a random forest

I don't know how I should tune the hyperparameters: "max depth" and "number of tree" of my model (a random forest). I use Python and I just discovered grid search, but I don't know which range I ...
7
votes
3answers
6k views

Step-by-step explanation of K-fold cross-validation with grid search to optimise hyperparameters

I'm well aware of the advantages of k-fold (and leave-one-out) cross-validation, as well as of the advantages of splitting your training set to create a third holdout 'validation' set, which you use ...
7
votes
2answers
10k views

What exactly is tol (tolerance) used as stopping criteria in sklearn models?

What exactly is the tol (tolerance for stopping criteria) in scikit-learn? What is that quantity which is checked with tol to end the training?
7
votes
1answer
949 views

What exactly is a hyperparameter?

Title says it all. I have seen both "the hyperparameter of the Dirichlet distribution" and "the parameter of the Dirichlet distribution" What are the differences?
7
votes
1answer
123 views

Decision rule as a hyper-parameter in LASSO

I have a question that is related to the following: Is decision threshold a hyperparameter in logistic regression? but would like some clarification. The general consensus is that the decision rule ...
7
votes
2answers
901 views

Understanding the effect of hyperparameters in machine learning experiments

In machine learning every algorithm has a set of hyperparameters which needs to be optimized for best prediction performance. The simplest method for this optimization is called grid search which ...
6
votes
1answer
2k views

Is there any method for choosing the number of layers and neurons?

I'm learning autoencoders applied to image classification. However, I'm in the beginning stage (training the autoencoder for feature extraction). I was testing different topologies by changing the ...
6
votes
1answer
53 views

Finding the best cookie recipe. Hyper-parameter optimization using noisy local comparison

I was watching applied science explore the state space while making cookies in this video https://youtu.be/8YEdHjGMeho. He was setting everything manually and it looks like he was searching a 10 ...
6
votes
2answers
2k views

How exactly does one marginalize over parameters in an N-dimensional likelihood?

I see no equations for the following, so I'm not sure exactly what they are talking about: "For each model, we determine the best fit parameters from the peak of the N-dimensional likelihood surface. ...
5
votes
1answer
338 views

Hyperparameter Optimization Using Gaussian Processes

I have a dataset that is divided into training and validation dataset. I am using Gaussian Processes to perform hyperparameter optimization. So I am using the accuracy recorded on the validation ...
5
votes
2answers
855 views

Bias in classifier model selection

Say I have a set of classifier models, each generated using feature selection inside a repeated k-fold cross-validation. Each classifier model is generated using a different set of regularization ...
5
votes
1answer
2k views

Nested Cross-Validation for Feature Selection and Hyperparameter Optimization

I spent quite a few hours trying to understand nested cross-validation and try and make an implementation myself — I'm really uncertain if I am doing this right, and I am not sure how to test if I am ...
5
votes
2answers
672 views

How to select penalty parameter after cross validation?

Say I have a feature matrix $X$ and a target $y$. I use $k$-fold cross validation to generate $k$ out-of-sample MSE curves as a function of a penalty parameter $\lambda$ $$MSE_i(\lambda) \quad (i=1,\...
5
votes
1answer
2k views

Can I do hyper-parameters optimization before model selection?

For every N model: Split in test and train subsets(Using the same seed for every N model) Randomized Search of parameters with 5 k-folds on train subset Select the best estimator obtained after the ...
5
votes
1answer
7k views

Hyperparameter estimation in Gaussian process

I am trying to optimize the hyperparameters for a Gaussian process. As a starter I choose the squared exponential function for covariance where iI have to optimize 3 parameters $\sigma_f$, $\sigma_n$ ...
5
votes
1answer
2k views

What is a sensible order for parameter tuning in neural networks?

There are so many aspects one could possibly change in a deep neural network that it is generally not feasible to do a grid search over all of them (e.g. activation function, layer type, number of ...
5
votes
1answer
164 views

Why do you need a separate criterion (Sequential model-based global optimization) in hyperparameter tuning?

In the paper 'Algorithms for Hyper-Parameter Optimization' (pdf), where they explain the 'Sequential Model-based Global optimization method (SMBO)', the authors made a comment that, ...
5
votes
1answer
2k views

What are the state-of-the-art methods to determine parameters in CNN, NN, RNN, or any deep learning models

The question is "how do we determine the (hyper)parameters in deep learning models, usch as CNN, RNN?" This is a difficult question that so far I am not aware of a solid solution and I want to bring ...
5
votes
2answers
171 views

What to treat as (hyper-)parameter and why

I have been wondering about the differences between model paramters and model hyperparameters, as well as what their categorization means for a learning problem. Is the distinction between model ...
5
votes
3answers
1k views

Guidelines to improve a convolutional neural network?

I am trying to use a convolutional neural network (implemented with keras) to solve a modified version of the MNIST classification problem (I am trying the background variations as described here). I ...
5
votes
1answer
739 views

Is an SVM's (maximum) likelihood uniquely defined as a function of hyperparameters?

I think that I must be reading this paragraph (below) incorrectly. Note that both types of evidence that we have defined in general depend on the inverse noise level $C$ and the kernel $K(x, x^\...