# Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

385 questions
Filter by
Sorted by
Tagged with
4k views

### Understanding early stopping in neural networks and its implications when using cross-validation

I'm a bit troubled and confused by the idea of how the technique early stopping is defined. If you take a look it Wikipedia, it is defined as follows: Split the training data into a training set and ...
886 views

### Optimal grid search for $C$ in SVM

Say we are training a radial SVM with parameters $C$ and $\gamma$. We would typically do this using grid search and CV. My prof. alluded that one does not have to exhaust all of $C$ values in the grid ...
564 views

### For Lasso regression, does increasing the tuning parameter $\lambda$ result in more significant coefficients?

In Lasso regression, for a sparse estimate of coefficients $\beta$, we have: $$\hat{\beta}(\lambda) = \arg \min_b \Bigl\{\frac{1}{2} ||y-Xb||^2_{2} + \lambda||b||_1\Bigl\}$$ Then, ...
847 views

### HMM with unknown number of hidden states

The Baum-Welch algorithm can estimate the parameters for a given structure of a HMM. However, how to determine the structure, specifically the number of hidden states? Ideas so far: Trial and error, ...
362 views

### (Nested) cross-validation for model selection and optimization?

I try to solve a binary classification problem. I have a set of features to build a model. The simplest model just pics a single feature $f$ and optimizes a cutoff $c_f$ to separate the two classes. ...
9k views

### Hyper parameters tuning: Random search vs Bayesian optimization

So, we know that random search works better than grid search, but a more recent approach is Bayesian optimization (using gaussian processes). I've looked up a comparison between the two, and found ...
36 views

### Help on the beta coefficients returned by cvglmnet

I have a question about cross-validation and how the beta values are returned by cvglmnet. First, I understand that when using 10-fold cross validaiton for optimal parameter search, you take the ...
121 views

2k views

### Should the validation set come from the training set or the testing set?

I'm surprised that this question has not been asked before (Maybe I just couldn't find it), but from where should the validation set come from? Should we split the the total dataset into training, ...
2k views

### K-fold cross validation for choosing number of epochs

I'm trying to figure out the optimal number of epochs I need to train a convolutional network for, using 4 fold cross validation. I divided the dataset into 4 parts, trained the model on 3 parts, ...
270 views

### Hessian matrix of log marginal likelihood of Gaussian Process

I'm trying to compute the exact second derivatives of log marginal likelihood of Gaussian Process for learning hyperparameters. The log marginal likelihood and its partial derivative are given in 5 ...
2k views

### What are some of the disavantage of bayesian hyper parameter optimization?

I am fairly new to machine learning and statistics but I was wondering why bayesian optimization is not referred more often online when learning machine learning to optimize your algorithm ...
456 views

### What is Tophat prior?

I constantly run into Tophat prior being mentioned as a prior for certain hyperparameters (prior for kernel length scales for example), but I have never seen its analytical expression or have read ...
79 views

### How to find a robust and performant hyper parameter region?

Assume we randomly sampled hyper parameters and for each configuration we evaluated our model on three fixed seeds. What methods are there to obtain a robust and performant hyper parameter ...
284 views

### Importance of Hyper-Parameter Optimization in Deep Learning models

Setting hyper-parameters in Deep learning models is considered more of intuitive or some form of black art. Hyper-Parameter Optimization (HPO) methods paves a principled approach of finding it. ...
83 views

### Random Search parameters optimization [duplicate]

This is an example, of hyper-parameters search in a hypothetical python code. My question is How can I be sure that RandomSearch really will find the best parameters that lies within the top 5% of the ...
164 views

### Why do you need a separate criterion (Sequential model-based global optimization) in hyperparameter tuning?

In the paper 'Algorithms for Hyper-Parameter Optimization' (pdf), where they explain the 'Sequential Model-based Global optimization method (SMBO)', the authors made a comment that, ...
621 views

### Feature selection with XGBoost

XGBoost will produce different values for feature importances with different hyperparameters on the same dataset. When using XGBoost as a feature selection algorithm for a different model, should I ...
62 views

### Is there any method for learning regularization parameter? [duplicate]

One of the hardest tasks in using machine learning methods is choosing the appropriate hyper-parameters of the model such as regularization parameter. As far as I know, this task is performed by a ...
2k views

### SVM kernel parameter and tunning parameter

In the svm function, you can apply three cases to the kernel parameter. "Linear," "radial," and "polynomia." And I try to derive the optimal svm result by ...
33 views

### How can we learn the values of the parameters for Levenshtein distance?

I am trying to filter out similar-looking names from a database. Once I have figured out the names, I will merge them into a single entity. To achieve this, I am planning on using edit-distance ...
182 views

### Why do we sample from log space when optimizing learning rate + regularization params?

Since I took Karpathy's CS231n I used the method he mentions on the 5th lecture for hyperparameter optimization of neural networks which samples the learning rate and regularization parameters from ...
3k views

### How to split dataset for model selection and tuning

I have read as many questions as I could on model selection, cross validation, and hyperparameter tuning and I am still confused on how to partition a dataset for the full training/tuning process. ...
357 views

### Tree complexity using gbm

I am using the gbm package and I am using the gbm.step function. In the ?gbm.step I see <...
1k views

### How to choose a Support Vector Machine classifier and tune the hyperparameters?

I'm trying to use a Support Vector Machine for classification using Scikit-Learn while understanding how to tune the ...
89 views

### How can I model such a distribution consisting of a mix of different distribution types?

Excuse the title, suggestions are welcome but I wasn't able to come up with an easier one. I am trying to find a simple model which "kind of", at least visually, appears to follow a distribution like ...
461 views

### How do we use cross-validation to get hyper parameters for soft-margin Gaussian kernel SVM?

I was reading this How to select kernel for SVM? but as I am relatively new to the applied techniques in ML (only familiar with some theory) I may need some explaining on the answers. Specifically, ...
62 views

### Finetunng a deep network

Can we perform fine tuning using the pre-trained weight file when the network is slightly changed? I have changed the CNN from classification to regression network and rest of the network is the same....
684 views

### Learning of hyperparameters for Gaussian process

Following the paper Practical Bayesian Optimization of Machine Learning Algorithms. Link It's not clear to me as to how the hyper-parameters (different from the ...
58 views

### How to factor random variables that involve conditionals on hyper-parameters?

Suppose we have a Bayesian DAG with the following structure: \begin{array}{c} \alpha & & \beta & & \gamma & & \delta\\ & \searrow & \downarrow && \downarrow &...
179 views

### What is artistic about hyperparameter tuning?

I recently started competing at kaggle which exposed me with hyper parameter tuning of models such as GBM,xgboost etc.All the sources that I studied to get started with an approach to tuning ...
161 views

### Concluding from Learning and validation curves

Background I am fitting a dataset of 1500 observations and 375 features (after one hot encoding of categorical features) dealing with prediction of house prices. I am using a gradient boosting model (...
83 views

### How to show $f(p, q|\alpha, \beta, \gamma, \delta) = f(p|\alpha, \beta)f(q|\gamma, \delta)$ if $\alpha, \beta,\gamma,\delta$ are hyper-parameters?

Suppose we have that $f(p|\alpha, \beta)$ is a distribution of a random variable $P$ with hyperparameters $\alpha,\beta$, and that $f(q|\gamma, \delta)$ is a distribution of a random variable $Q$ with ...
399 views

### Range of parameters in svm light for polynomial and sigmoid kernel

I'm using svm light: http://svmlight.joachims.org/ I'm doing grid search for parameter tuning. I would test the polynomial and the sigmoid kernel. I would know a possible range values for parameters s ...
44 views

### Starting configurations for RNN and MLPs

How do you go about choosing initial hyperparameters (layer size, # of hidden units in RNN and dense layers, etc.) when training RNNs and MLPs? How do you iteratively tweak these settings -- do you ...
428 views

### What's your methodology of tuning neural network hyperparameters?

I'm curious to see what methods all of you use to tune your neural network hyperparameters. Specifically, How do you choose the number of layers, and how many hidden units are in each layer? Do you ...
88 views

### Finding the optimal hyperparameters

I'm a little bit confused when estimating the hyperparameter in a problem of regression with gaussian process. In "Gaussian process for machine learning" (Rasmussen and Williams, MIT Press, 2006), the ...
179 views

### How to validate a model when first exploring model hyperparameter space?

For my class project I am comparing various tree-based ensemble methods such as bagging, boosting, random forest, and AdaBoost against my data set and I can't quite determine my methodology. I know ...
230 views

### Using cross-validation both for feature selection and hyperparameters optimization

I would like to use cross validation both to tune the hyperparameters for my supervised learning model, and to perform feature selection. Is it a bad practice to use cross-validation more than once on ...
289 views

### Covariance term in Gradient of Gaussian Process marginal likelihood

log marginal likelihood for Gaussian Process as given by Rasmussen's: Gaussian Processes for Machine Learning equation 5.8 is \log p(y|X, \theta) = -\frac{1}{2}y^{T} K_y^{-1}y - \frac{1}{2}\log|...
4k views

### XGBoost feature subsampling

I have a dataset with ~30k samples and 35 features (after feature selection; these seem to be the most important features for this dataset and they have low correlation between each other). After ...
85 views

### RNN optimization for bigger datasets

I am training bi-directional RNN model with attention for answer selection referring this. I am attaching screenshot of tensorflow graphs. My dataset consist of Question and 5 multiple choice ...
71 views

### Tuning deep models with dataset subsample

I have a quite big dataset (380k samples), and I am try to do model selection over my validation set (3K samples). Since to run a single model requires days, I have subsample my training set (taking ...
692 views

### Random Search for Hyper Parameters

I want to tune my neural network and find the some good lambda and eta values. I can do exhaustive grid search to find the best combo. However Bergstra in this http://www.jmlr.org/papers/volume13/...
677 views

### conjugate prior: is ever the best choice?

I'm reading about the conjugate prior of classic probability distributions (e.g. beta distribution for binomial distribution); it's explained just as "algebric trick" to have easier calculation in ...
216 views

### R_How to select hyperprior distribution for Beta distribution parameter in R?

I have 2-mixture weibull distrubution. And this distribution haver the portion parameter θ whose value should lie between (0,1). Therefore, I am assuming the prior distribution of θθ to be a beta ...