Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
8
votes
2answers
4k views

Understanding early stopping in neural networks and its implications when using cross-validation

I'm a bit troubled and confused by the idea of how the technique early stopping is defined. If you take a look it Wikipedia, it is defined as follows: Split the training data into a training set and ...
3
votes
3answers
886 views

Optimal grid search for $C$ in SVM

Say we are training a radial SVM with parameters $C$ and $\gamma$. We would typically do this using grid search and CV. My prof. alluded that one does not have to exhaust all of $C$ values in the grid ...
0
votes
0answers
564 views

For Lasso regression, does increasing the tuning parameter $\lambda$ result in more significant coefficients?

In Lasso regression, for a sparse estimate of coefficients $\beta$, we have: $$ \hat{\beta}(\lambda) = \arg \min_b \Bigl\{\frac{1}{2} ||y-Xb||^2_{2} + \lambda||b||_1\Bigl\} $$ Then, ...
3
votes
1answer
847 views

HMM with unknown number of hidden states

The Baum-Welch algorithm can estimate the parameters for a given structure of a HMM. However, how to determine the structure, specifically the number of hidden states? Ideas so far: Trial and error, ...
1
vote
1answer
362 views

(Nested) cross-validation for model selection and optimization?

I try to solve a binary classification problem. I have a set of features to build a model. The simplest model just pics a single feature $f$ and optimizes a cutoff $c_f$ to separate the two classes. ...
13
votes
3answers
9k views

Hyper parameters tuning: Random search vs Bayesian optimization

So, we know that random search works better than grid search, but a more recent approach is Bayesian optimization (using gaussian processes). I've looked up a comparison between the two, and found ...
1
vote
1answer
36 views

Help on the beta coefficients returned by cvglmnet

I have a question about cross-validation and how the beta values are returned by cvglmnet. First, I understand that when using 10-fold cross validaiton for optimal parameter search, you take the ...
1
vote
0answers
121 views

Is there any logic to the many meanings of the value of L2-regularization “λ”?

The conventional definition of the L2-regularization "weight decay" hyperparameter $\lambda$ is generally of the form $$\text{J}(\mathbf{w}\vert \mathbf{X},\mathbf{y})= \text{L}(\mathbf{\widehat{y}}(\...
1
vote
1answer
150 views

Bayesian inference for parameter estimation [closed]

I am new in theBayesian inference domain, so sorry if my questions seem silly. However, I realy need some help in understanding this concept, since reading the same information each time from ...
4
votes
1answer
306 views

Negative values of hyperparameters in Gaussian Process

I am trying to optimize the hyperparameters for a Gaussian process. I am using a squared exponential kernel, where I am optimizing three parameters. $$k_y(x_p,x_q) = \sigma^2_f \exp\left(-\frac{1}{2l^...
-1
votes
4answers
2k views

Should the validation set come from the training set or the testing set?

I'm surprised that this question has not been asked before (Maybe I just couldn't find it), but from where should the validation set come from? Should we split the the total dataset into training, ...
1
vote
1answer
2k views

K-fold cross validation for choosing number of epochs

I'm trying to figure out the optimal number of epochs I need to train a convolutional network for, using 4 fold cross validation. I divided the dataset into 4 parts, trained the model on 3 parts, ...
1
vote
0answers
270 views

Hessian matrix of log marginal likelihood of Gaussian Process

I'm trying to compute the exact second derivatives of log marginal likelihood of Gaussian Process for learning hyperparameters. The log marginal likelihood and its partial derivative are given in 5 ...
8
votes
1answer
2k views

What are some of the disavantage of bayesian hyper parameter optimization?

I am fairly new to machine learning and statistics but I was wondering why bayesian optimization is not referred more often online when learning machine learning to optimize your algorithm ...
0
votes
1answer
456 views

What is Tophat prior?

I constantly run into Tophat prior being mentioned as a prior for certain hyperparameters (prior for kernel length scales for example), but I have never seen its analytical expression or have read ...
2
votes
0answers
79 views

How to find a robust and performant hyper parameter region?

Assume we randomly sampled hyper parameters and for each configuration we evaluated our model on three fixed seeds. What methods are there to obtain a robust and performant hyper parameter ...
2
votes
2answers
284 views

Importance of Hyper-Parameter Optimization in Deep Learning models

Setting hyper-parameters in Deep learning models is considered more of intuitive or some form of black art. Hyper-Parameter Optimization (HPO) methods paves a principled approach of finding it. ...
0
votes
0answers
83 views

Random Search parameters optimization [duplicate]

This is an example, of hyper-parameters search in a hypothetical python code. My question is How can I be sure that RandomSearch really will find the best parameters that lies within the top 5% of the ...
5
votes
1answer
164 views

Why do you need a separate criterion (Sequential model-based global optimization) in hyperparameter tuning?

In the paper 'Algorithms for Hyper-Parameter Optimization' (pdf), where they explain the 'Sequential Model-based Global optimization method (SMBO)', the authors made a comment that, ...
0
votes
1answer
621 views

Feature selection with XGBoost

XGBoost will produce different values for feature importances with different hyperparameters on the same dataset. When using XGBoost as a feature selection algorithm for a different model, should I ...
1
vote
0answers
62 views

Is there any method for learning regularization parameter? [duplicate]

One of the hardest tasks in using machine learning methods is choosing the appropriate hyper-parameters of the model such as regularization parameter. As far as I know, this task is performed by a ...
0
votes
1answer
2k views

SVM kernel parameter and tunning parameter

In the svm function, you can apply three cases to the kernel parameter. "Linear," "radial," and "polynomia." And I try to derive the optimal svm result by ...
1
vote
0answers
33 views

How can we learn the values of the parameters for Levenshtein distance?

I am trying to filter out similar-looking names from a database. Once I have figured out the names, I will merge them into a single entity. To achieve this, I am planning on using edit-distance ...
1
vote
1answer
182 views

Why do we sample from log space when optimizing learning rate + regularization params?

Since I took Karpathy's CS231n I used the method he mentions on the 5th lecture for hyperparameter optimization of neural networks which samples the learning rate and regularization parameters from ...
3
votes
2answers
3k views

How to split dataset for model selection and tuning

I have read as many questions as I could on model selection, cross validation, and hyperparameter tuning and I am still confused on how to partition a dataset for the full training/tuning process. ...
0
votes
1answer
357 views

Tree complexity using gbm

I am using the gbm package and I am using the gbm.step function. In the ?gbm.step I see <...
0
votes
0answers
1k views

How to choose a Support Vector Machine classifier and tune the hyperparameters?

I'm trying to use a Support Vector Machine for classification using Scikit-Learn while understanding how to tune the ...
1
vote
1answer
89 views

How can I model such a distribution consisting of a mix of different distribution types?

Excuse the title, suggestions are welcome but I wasn't able to come up with an easier one. I am trying to find a simple model which "kind of", at least visually, appears to follow a distribution like ...
0
votes
1answer
461 views

How do we use cross-validation to get hyper parameters for soft-margin Gaussian kernel SVM?

I was reading this How to select kernel for SVM? but as I am relatively new to the applied techniques in ML (only familiar with some theory) I may need some explaining on the answers. Specifically, ...
2
votes
1answer
62 views

Finetunng a deep network

Can we perform fine tuning using the pre-trained weight file when the network is slightly changed? I have changed the CNN from classification to regression network and rest of the network is the same....
1
vote
1answer
684 views

Learning of hyperparameters for Gaussian process

Following the paper Practical Bayesian Optimization of Machine Learning Algorithms. Link It's not clear to me as to how the hyper-parameters (different from the ...
2
votes
2answers
58 views

How to factor random variables that involve conditionals on hyper-parameters?

Suppose we have a Bayesian DAG with the following structure: \begin{array}{c} \alpha & & \beta & & \gamma & & \delta\\ & \searrow & \downarrow && \downarrow &...
3
votes
1answer
179 views

What is artistic about hyperparameter tuning?

I recently started competing at kaggle which exposed me with hyper parameter tuning of models such as GBM,xgboost etc.All the sources that I studied to get started with an approach to tuning ...
1
vote
0answers
161 views

Concluding from Learning and validation curves

Background I am fitting a dataset of 1500 observations and 375 features (after one hot encoding of categorical features) dealing with prediction of house prices. I am using a gradient boosting model (...
0
votes
1answer
83 views

How to show $f(p, q|\alpha, \beta, \gamma, \delta) = f(p|\alpha, \beta)f(q|\gamma, \delta)$ if $\alpha, \beta,\gamma,\delta$ are hyper-parameters?

Suppose we have that $f(p|\alpha, \beta)$ is a distribution of a random variable $P$ with hyperparameters $\alpha,\beta$, and that $f(q|\gamma, \delta)$ is a distribution of a random variable $Q$ with ...
0
votes
0answers
399 views

Range of parameters in svm light for polynomial and sigmoid kernel

I'm using svm light: http://svmlight.joachims.org/ I'm doing grid search for parameter tuning. I would test the polynomial and the sigmoid kernel. I would know a possible range values for parameters s ...
0
votes
1answer
44 views

Starting configurations for RNN and MLPs

How do you go about choosing initial hyperparameters (layer size, # of hidden units in RNN and dense layers, etc.) when training RNNs and MLPs? How do you iteratively tweak these settings -- do you ...
2
votes
1answer
428 views

What's your methodology of tuning neural network hyperparameters?

I'm curious to see what methods all of you use to tune your neural network hyperparameters. Specifically, How do you choose the number of layers, and how many hidden units are in each layer? Do you ...
1
vote
0answers
88 views

Finding the optimal hyperparameters

I'm a little bit confused when estimating the hyperparameter in a problem of regression with gaussian process. In "Gaussian process for machine learning" (Rasmussen and Williams, MIT Press, 2006), the ...
0
votes
1answer
179 views

How to validate a model when first exploring model hyperparameter space?

For my class project I am comparing various tree-based ensemble methods such as bagging, boosting, random forest, and AdaBoost against my data set and I can't quite determine my methodology. I know ...
0
votes
1answer
230 views

Using cross-validation both for feature selection and hyperparameters optimization

I would like to use cross validation both to tune the hyperparameters for my supervised learning model, and to perform feature selection. Is it a bad practice to use cross-validation more than once on ...
2
votes
0answers
289 views

Covariance term in Gradient of Gaussian Process marginal likelihood

log marginal likelihood for Gaussian Process as given by Rasmussen's: Gaussian Processes for Machine Learning equation 5.8 is $$\log p(y|X, \theta) = -\frac{1}{2}y^{T} K_y^{-1}y - \frac{1}{2}\log|...
5
votes
1answer
4k views

XGBoost feature subsampling

I have a dataset with ~30k samples and 35 features (after feature selection; these seem to be the most important features for this dataset and they have low correlation between each other). After ...
1
vote
0answers
85 views

RNN optimization for bigger datasets

I am training bi-directional RNN model with attention for answer selection referring this. I am attaching screenshot of tensorflow graphs. My dataset consist of Question and 5 multiple choice ...
1
vote
0answers
71 views

Tuning deep models with dataset subsample

I have a quite big dataset (380k samples), and I am try to do model selection over my validation set (3K samples). Since to run a single model requires days, I have subsample my training set (taking ...
1
vote
1answer
692 views

Random Search for Hyper Parameters

I want to tune my neural network and find the some good lambda and eta values. I can do exhaustive grid search to find the best combo. However Bergstra in this http://www.jmlr.org/papers/volume13/...
4
votes
2answers
677 views

conjugate prior: is ever the best choice?

I'm reading about the conjugate prior of classic probability distributions (e.g. beta distribution for binomial distribution); it's explained just as "algebric trick" to have easier calculation in ...
1
vote
0answers
216 views

R_How to select hyperprior distribution for Beta distribution parameter in R?

I have 2-mixture weibull distrubution. And this distribution haver the portion parameter θ whose value should lie between (0,1). Therefore, I am assuming the prior distribution of θθ to be a beta ...
0
votes
1answer
582 views

what is the difference between grid search tuning and h2o tuning in gbm?

I have to choose proper parameters in gbm() function. Until now, I have used grid search as using train(), trainControl() functions. Recently, I found the h2o package. As using this, I can do choose ...
2
votes
0answers
90 views

Right procedure to picking the value of hyper parameters

I've asked a similar question before about cross-validation here and here. I've received many useful answers and benefit from the discussions on these websites. Here I summarize my original question ...