Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
30
votes
1answer
6k views

Do we have to tune the number of trees in a random forest?

Software implementations of random forest classifiers have a number of parameters to allow users to fine-tune the algorithm's behavior, including the number of trees in the forest. Is this a parameter ...
7
votes
1answer
949 views

What exactly is a hyperparameter?

Title says it all. I have seen both "the hyperparameter of the Dirichlet distribution" and "the parameter of the Dirichlet distribution" What are the differences?
43
votes
6answers
21k views

Practical hyperparameter optimization: Random vs. grid search

I'm currently going through Bengio's and Bergsta's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving ...
17
votes
1answer
2k views

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
19
votes
5answers
2k views

What's in a name: hyperparameters

So in a normal distribution, we have two parameters: mean $\mu$ and variance $\sigma^2$. In the book Pattern Recognition and Machine Learning, there suddenly appears a hyperparameter $\lambda$ in the ...
17
votes
3answers
6k views

How to get hyper parameters in nested cross validation?

I have read the following posts for nested cross validation and still am not 100% sure what I am to do with model selection with nested cross validation: Nested cross validation for model selection ...
1
vote
1answer
472 views

Is epoch optimization in CV with constant mini-batch size even possible?

Assume that you found the optimal hyperparameters of a neural network (e.g. a multi layer feed forward NN) with k-fold cross validation in a grid search. Lets assume you have varied the number of ...
12
votes
1answer
3k views

What do we mean by hyperparameters? [duplicate]

Can anyone give me full details about what we mean by hyperparameters, and what in the Dirichlet distribution are called hyperparameters? A practice example for the estimation of those parameters ...
38
votes
3answers
30k views

Guideline to select the hyperparameters in Deep Learning

I'm looking for a paper that could help in giving a guideline on how to choose the hyperparameters of a deep architecture, like stacked auto-encoders or deep believe networks. There are a lot of ...
19
votes
6answers
9k views

Is hyperparameter tuning on sample of dataset a bad idea?

I have a dataset of 140000 examples and 30 features for which I am training several classifiers for a binary classification (SVM, Logistic Regression, Random Forest etc) In many cases hyperparameter ...
2
votes
1answer
9k views

About SVM cost and gamma parameters tuning

I am using R and e1071 package to tune a C-classification SVM. My question is: regardless of the kernel type (linear, ...
6
votes
1answer
2k views

Is there any method for choosing the number of layers and neurons?

I'm learning autoencoders applied to image classification. However, I'm in the beginning stage (training the autoencoder for feature extraction). I was testing different topologies by changing the ...
5
votes
1answer
2k views

What are the state-of-the-art methods to determine parameters in CNN, NN, RNN, or any deep learning models

The question is "how do we determine the (hyper)parameters in deep learning models, usch as CNN, RNN?" This is a difficult question that so far I am not aware of a solid solution and I want to bring ...
8
votes
2answers
4k views

Understanding early stopping in neural networks and its implications when using cross-validation

I'm a bit troubled and confused by the idea of how the technique early stopping is defined. If you take a look it Wikipedia, it is defined as follows: Split the training data into a training set and ...
7
votes
1answer
123 views

Decision rule as a hyper-parameter in LASSO

I have a question that is related to the following: Is decision threshold a hyperparameter in logistic regression? but would like some clarification. The general consensus is that the decision rule ...
21
votes
2answers
23k views

Natural interpretation for LDA hyperparameters

Can somebody explain what is the natural interpretation for LDA hyperparameters? ALPHA and BETA are parameters of Dirichlet ...
20
votes
4answers
5k views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
8
votes
1answer
2k views

What are some of the disavantage of bayesian hyper parameter optimization?

I am fairly new to machine learning and statistics but I was wondering why bayesian optimization is not referred more often online when learning machine learning to optimize your algorithm ...
13
votes
2answers
3k views

Is decision threshold a hyperparameter in logistic regression?

Predicted classes from (binary) logistic regression are determined by using a threshold on the class membership probabilities generated by the model. As I understand it, typically 0.5 is used by ...
2
votes
1answer
97 views

Why is it valid to use CV to set parameters and hyperparameters but not seeds?

This is a specification or continuation of my prior question: [Is it 'fair' to set a seed in a random forest regression to yield the highest accuracy? Its valid to use cross-validation to ...
1
vote
1answer
47 views

$R^2$ of 1 but RMSE > 0

I am running k-fold cross validation on my training data, and then choosing the best set of hyper parameters, re-training on the training data and testing on a new (unseen) testing data. I am getting ...
1
vote
2answers
107 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
0
votes
1answer
357 views

Tree complexity using gbm

I am using the gbm package and I am using the gbm.step function. In the ?gbm.step I see <...
13
votes
3answers
6k views

How should Feature Selection and Hyperparameter optimization be ordered in the machine learning pipeline?

My objective is to classify sensor signals. The concept of my solution so far is : i) Engineering features from raw signal ii) Selecting relevant features with ReliefF and a clustering approach iii) ...
24
votes
3answers
8k views

What is the reason that the Adam Optimizer is considered robust to the value of its hyper parameters?

I was reading about the Adam optimizer for Deep Learning and came across the following sentence in the new book Deep Learning by Bengio, Goodfellow and Courville: Adam is generally regarded as ...
7
votes
3answers
6k views

Step-by-step explanation of K-fold cross-validation with grid search to optimise hyperparameters

I'm well aware of the advantages of k-fold (and leave-one-out) cross-validation, as well as of the advantages of splitting your training set to create a third holdout 'validation' set, which you use ...
18
votes
2answers
2k views

Advantages of Particle Swarm Optimization over Bayesian Optimization for hyperparameter tuning?

There's substantial contemporary research on Bayesian Optimization (1) for tuning ML hyperparameters. The driving motivation here is that a minimal number of data points are required to make informed ...
3
votes
1answer
3k views

Hyperparameter Tuning - What is possible in terms of accuracy gain?

A question from a newbie: I played around with parameter tuning (grid, random search) in R (caret, xgboost) and my observation is as follows: in terms of accuracy gains I was able to get 3 - 7% but ...
13
votes
3answers
9k views

Hyper parameters tuning: Random search vs Bayesian optimization

So, we know that random search works better than grid search, but a more recent approach is Bayesian optimization (using gaussian processes). I've looked up a comparison between the two, and found ...
5
votes
1answer
4k views

XGBoost feature subsampling

I have a dataset with ~30k samples and 35 features (after feature selection; these seem to be the most important features for this dataset and they have low correlation between each other). After ...
4
votes
2answers
324 views

Random search for the optimal number of input features and optimal number of hidden layers for a MLP?

I've performed a random search in hypothesis space $$\{(c,h)| c \in U[1,256]; h\in U[1,100];c \in \mathrm{Z} \text{ and } h \in \mathrm{Z}\}$$ that defines the parameters of a standard multilayer ...
3
votes
1answer
847 views

HMM with unknown number of hidden states

The Baum-Welch algorithm can estimate the parameters for a given structure of a HMM. However, how to determine the structure, specifically the number of hidden states? Ideas so far: Trial and error, ...
2
votes
1answer
428 views

What's your methodology of tuning neural network hyperparameters?

I'm curious to see what methods all of you use to tune your neural network hyperparameters. Specifically, How do you choose the number of layers, and how many hidden units are in each layer? Do you ...
2
votes
1answer
412 views

Selecting optimal number of input features and optimal number of hidden layers for a MLP?

What is the best way to select parameters for a binary neural network classifier? More specifically I have 265 features ranked according to Mutual Information Criterion. I have to determine the ...
4
votes
0answers
207 views

Prior elicitation with Normal-Gamma or Normal-Inverse-Gamma

I am looking for a way to have experts elicit a prior for a Normal-Inverse-Gamma Bayesian linear regression model. Is there any material suggesting intuitive ways to go about this?
4
votes
1answer
2k views

Time series forecasting using Gaussian Process regression

I used Gaussian Process Regression to predict a time series, what I have is sensor's readings that come every hour ( I have data for about 3 years) I chose the periodic kernel function mentioned here [...
3
votes
3answers
2k views

How to speed up hyperparameter optimization?

For my application, I am classifying sensor signals. I wrote a 'Rapid Prototyping' script to quickly build machine learning models and return their cross validation performance. My pipeline is the ...
3
votes
1answer
1k views

Reasonable hyperparameter range for Latent Dirichlet Allocation?

What are good ranges for the hyperparameters $\alpha$ and $\beta$ (explained well here) in LDA? I appreciate hyperparameter tuning always depends on the use case, data, content of documents etc., but ...
3
votes
1answer
3k views

Trouble minimizing perplexity in LDA

I am running LDA from Mark Steyver's MATLAB Topic Modelling toolkit on a few Apache Java open source projects. I have taken care of stop word removal (for e.g. words such Apache, java keywords are ...
3
votes
0answers
179 views

Analyse sensitivity of hyper-parameters of Machine Learning Models

I want to analyse how sensitive my non neural net machine learning models are to the choice of the different parameters. I am currently using grid search to tune the models. Is there any method that I ...
2
votes
2answers
284 views

Importance of Hyper-Parameter Optimization in Deep Learning models

Setting hyper-parameters in Deep learning models is considered more of intuitive or some form of black art. Hyper-Parameter Optimization (HPO) methods paves a principled approach of finding it. ...
1
vote
2answers
2k views

Likelihood vs. noise kernel hyperparameter in GPML Toolbox

I'm using GPML toolbox by C.E.Rasmussen to solve the basic GP regression problem (presented in the book) with noisy observations. That is to say, estimate the underlying function $f$ of a static noisy ...
9
votes
1answer
156 views

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

In François Chollet's Deep Learning with Python it says: As a result, tuning the configuration of the model based on its performance on the validation set can quickly result in overfitting to ...
3
votes
1answer
398 views

Can different classification methods be compared in the same manner as models during hyper-parameter tuning?

If I would like to choose between different classifiers, e.g. support vector machines (SVM) and boosted trees, based on their generalization performance, can I do this in the same way as I would do ...
3
votes
1answer
804 views

The role of $\gamma$ & $C$ in SVM

I'm using support vector machine method with the Gaussian kernel. Is it true that $\gamma$ and $C$ are hyper parameters of SVM? What is their role exactly?
1
vote
0answers
161 views

Concluding from Learning and validation curves

Background I am fitting a dataset of 1500 observations and 375 features (after one hot encoding of categorical features) dealing with prediction of house prices. I am using a gradient boosting model (...
1
vote
1answer
522 views

Multilayer perceptron for binary classification: threshold learning

In a basic contest, the MLP loss function (cross entropy) uses as value for the label ŷ: +1 if the net output is greater or equal to 0.5 -1 otherwise Where the net output is a value in [0,1] ...
1
vote
0answers
66 views

How to sample weights for weighted kernels?

I'm using a SVM classifier with a weighted RBF kernel. My dataset has 17 features. In the RBF kernel I will use a weight for each feature. Of course the weights must sum to one. For choosing the best ...
1
vote
1answer
150 views

Bayesian inference for parameter estimation [closed]

I am new in theBayesian inference domain, so sorry if my questions seem silly. However, I realy need some help in understanding this concept, since reading the same information each time from ...
0
votes
0answers
24 views

Hyperparameter-free method for Moving Average/ Exponential smoothing?

I want to find hyperparameter-free method for Moving Average/ Exponential smoothing. Is there any related paper or python code? S(t)= alpha * F(t) + (1-alpha) * S(t-1) Any methods can avoid the ...