Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

147 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
453 views

General procedures for combined feature selection, model tuning, and model selection?

What is the general procedure for a combined task of model tuning (i.e., hyperparameter selection), feature selection and model selection? I know some basic principles for each task, but when ...
4
votes
0answers
209 views

Prior elicitation with Normal-Gamma or Normal-Inverse-Gamma

I am looking for a way to have experts elicit a prior for a Normal-Inverse-Gamma Bayesian linear regression model. Is there any material suggesting intuitive ways to go about this?
3
votes
0answers
49 views

Parameter dimensionality reduction in a Kalman filter framework

My problem is related to parameter identification with maximum likelihood in a Kalman filter. This framework consists of a multivariate set-up, wherein the unobserved components of the initial ...
3
votes
0answers
33 views

Bayesian Hyperparameter Optimization. What makes it “bayesian”?

I'm using some bayesian hyperparameter optimization. I know how they works . They always calculate the next values of the hyperparameter dependent on the result of former evaluations. But what makes ...
3
votes
0answers
191 views

Analyse sensitivity of hyper-parameters of Machine Learning Models

I want to analyse how sensitive my non neural net machine learning models are to the choice of the different parameters. I am currently using grid search to tune the models. Is there any method that I ...
3
votes
0answers
1k views

How could a hyperparameter grid search be visualised?

Consider a hyperparameter grid search that looks at the training and testing scores of an estimator with respect to multiple parameters like training epochs, number of nodes in layer 1, number of ...
2
votes
1answer
19 views

Model tuning in the presence of incorrect training labels

I have a situation where I have a large amount of labeled data (~40 million records) with a binary outcome variable that has about 50% positive and 50% negative cases. The issue is that I know that ...
2
votes
1answer
34 views

What are some of the most correct/accepted ways to tune and compare different models in an academic context?

Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of ...
2
votes
0answers
14 views

How to optimize hyperparameters in stacked model?

I was wondering whether somebody could explain how to optimize hyperparameters for the base learners and meta algorithm when stacking? In many tutorials they seem to be plucked out of thin air! ...
2
votes
1answer
158 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
2
votes
0answers
54 views

My roc is low while precision and recall are high.Why is it so?

I bulit a naive bayes classifier from 60k vectors of text and each of the text is a 2000 dimension vector(Used bag of words for vectorization). Used simple cross validator to find the best alpha and ...
2
votes
1answer
2k views

Cross validation and train test split

I am having a fundamental doubt about cross validation. I know that cross validation trains the model on dataset keeping aside a part of it for testing the model and each for each iteration the train/...
2
votes
0answers
36 views

Choosing Gaussian PDF basis bandwidth depending on number of bases and range of data

Summary (details below!) I have a basis expansion of $m$ (univariate) Gaussian PDFs to model the density of a sample $X$. The means of these PDFs are spaced equidistantly through the domain of $X$ ...
2
votes
0answers
97 views

Can Bayesian Optimization solve this problem?

Suppose ${\bf{x}} = (x_1,\ldots,x_n)$ and $f({\bf{x}})\propto 1_A({\bf{x}}) \prod_{i=1}^n {x_i}^{\alpha_i-1} e^{-\beta_i x_i}$ , i.e. $f$ is proportional to the product of independent gamma ...
2
votes
0answers
1k views

Gaussian Process Hyperparameter Tuning

I'm planning to use Gaussian Process (GP) to model my case. However, while learning the GP I found out that we have to tuning the hyperparameters to give us the best solution. I have checked several ...
2
votes
0answers
28 views

Change of optimal learning rate with small changes to architecture and data

I am training variants of similar neural networks, which differ slightly in the number of filters or layers. Additionally, the data is sometimes slightly changed using different preprocessing like ...
2
votes
0answers
1k views

Unsupervised anomaly detection - metric for tuning Isolation Forest parameters

I have a project, in which, one of the stages is to find and label anomalous data points, that are likely to be outliers. As a first step, I am using Isolation Forest algorithm, which, after plotting ...
2
votes
0answers
34 views

Hyper-parameters which minimize the variance of transformed multi-variate Guassian variable

Let $k < p$ be positive integers and $g: \mathbb R^k \rightarrow \mathbb R^p$ be a smooth Lipschitz continuous function. Let $y_1,\ldots, y_N \in \mathbb R^p$ and $a = (a_1,\ldots,a_N) \in \mathbb ...
2
votes
0answers
105 views

evolution of C and gamma in SVR with the size of training examples

Do you know how C and gamma evolve while the size of training examples (X) rises ? I have found C and gamma for 10% and 20% of my data and I would like to save time. Can I determine C and gamma for my ...
2
votes
0answers
102 views

Information leaks related to test or validation sets

Lately I've been reading about (indirect) information leaks, related to validation and tests sets in the context of hyperparameter tuning. For example, Prakhar Agarwal, in his answer Does my ...
2
votes
1answer
123 views

Higher Test Scores but Higher Variance?

I am tuning hyper-parameters using 5-fold cross-validated grid search for various multiclass classifiers, and I keep running into the same issue that I can't quite wrap my head around. The hyper-...
2
votes
0answers
115 views

Does bias during hyperparameter tuning matter?

Different validation methods have different bias and variance. For instance, k-fold cross validation with high enough k (e.g. 10) has low bias, but high variance, whereas the bootstrap has lower ...
2
votes
0answers
81 views

How to find a robust and performant hyper parameter region?

Assume we randomly sampled hyper parameters and for each configuration we evaluated our model on three fixed seeds. What methods are there to obtain a robust and performant hyper parameter ...
2
votes
0answers
291 views

Covariance term in Gradient of Gaussian Process marginal likelihood

log marginal likelihood for Gaussian Process as given by Rasmussen's: Gaussian Processes for Machine Learning equation 5.8 is $$\log p(y|X, \theta) = -\frac{1}{2}y^{T} K_y^{-1}y - \frac{1}{2}\log|...
2
votes
0answers
223 views

R_How to select hyperprior distribution for Beta distribution parameter in R?

I have 2-mixture weibull distrubution. And this distribution haver the portion parameter θ whose value should lie between (0,1). Therefore, I am assuming the prior distribution of θθ to be a beta ...
2
votes
0answers
94 views

Right procedure to picking the value of hyper parameters

I've asked a similar question before about cross-validation here and here. I've received many useful answers and benefit from the discussions on these websites. Here I summarize my original question ...
2
votes
0answers
403 views

Maximum Likelihood Estimator - Covariance Squared Exponential Matlab

Following the Rasmussen&Williams Gpml Machine Learning book i'm trying to implement my gaussian process in matlab, avoiding to use other existing toolbox or complex pre-assembled functions, but ...
2
votes
0answers
410 views

Hyperopt with TPE strategy always sampling the same points

I want to do a convergence study of Ridge regression with increasing training size. For this I train the Ridge regressor using hyperopt for different training sizes ...
2
votes
0answers
52 views

Why is there a perceived difference between a Bayesian Hyperparameter and a Machine Learning Hyperparameter?

There are two seperate Wikipedia articles for the term Hyperparameter. One for Bayesian statistics and another for machine learning methods. Why is this so? BOTH of these definitions imply that ...
2
votes
0answers
475 views

Gaussian Process , selecting the hyperparameters

I am using Gaussian Process regression toolbox from the site http://www.gaussianprocess.org/gpml/code/matlab/doc/ I was able to use implement the code in matlab easily, following the guide lines. ...
2
votes
0answers
239 views

Maximizing incomplete likelihood

Given the conditional distribution $p(x|y)$ and the prior of the hidden variables $p(y|\theta)$ with unknown hyper-parameter $\theta$. Now we have observed i.i.d. samples of $x$. Besides the Bayes ...
2
votes
0answers
102 views

How to use multiple datasets in order to measure the performance of a learning system?

I’m working on a project where I need to test a machine learning system which has a lot of hyper-parameters. Further, in order to gauge the performance of system, I’m planning to use several datasets....
1
vote
0answers
14 views

Decrease hyparam 'C' in SVM classifier

In a hypothetical case where I have a small dataset and I break it into train/test. Then I tune the hyperparams doing k-fold on the train set and choose the 'C' hyperparameter that maximizes my AUC on ...
1
vote
0answers
30 views

Order of features for gridsearch and model fitting

Assuming that the same columns (i.e., features) are used for hyperparameter tuning and model fitting, and ensemble models are used for modeling (e.g., Random forest or XGboost), then does the order of ...
1
vote
1answer
23 views

Inference on Dirichlet hyper-parameter

I'm working on a Gibbs sampler for a (somewhat custom version of) Latent Dirichlet Allocation model. In short, I have data that comes from a $K$-dimensional Dirichlet-Multinomial distribution, i.e. $$...
1
vote
1answer
12 views

Find weight distribution in multiple term loss

I have a question if it is possible to find/learn the weight distribution in a multiple term loss where each weight models the importance of each term on the total loss. ...
1
vote
0answers
17 views

Optimal penalty for finding changepoints with the fused lasso, assuming some probabilistic model?

I am interested in detecting changepoints in a signal using the fused lasso (as implemented here for example). I am in particular interested in getting estimates of changepoints which are close to the ...
1
vote
0answers
32 views

Choosing a space function for hyperopt

I was originally doing a grid search for my parameter optimization and with 7 parameters being optimized, it would take ages. So I am choosing to use hyperopt at this point. I am however confused on ...
1
vote
1answer
78 views

EarlyStopping after GridSearchCV

I want to optimize the hyperparams for a CNN-architecture by using GridSearchCV. As hyperparameters to optimize, I would like to use the learning rate, dropout rate, number of neurons in den dense ...
1
vote
0answers
33 views

Does it make sense to combine Early Stopping with k-fold cross validation?

I have a CNN architecture for which I want to optimize the hyperparameters such as learning rate, dropout rate and number of epochs. I am thinking of a combination of k-fold cross validation and ...
1
vote
0answers
44 views

Proper Way to Combine Feature Selection and Hyperparameter Tuning?

Been doing reading on feature selection and hyperparameter tuning but I'm getting lost on how to properly code/set up the experiment. I am doing a classified ML experiment, I have 1200 samples and 400 ...
1
vote
0answers
113 views

Bayesian hyperparameter optimization + cross-validation

I want to use Bayesian optimization to search a space of hyperparameters for a neural network model. My objective function for this optimization is validation set accuracy. In addition, I want to ...
1
vote
0answers
18 views

How to select the most optimal hyperparameter in grid-search cross validation if the process is repeated X (i.e. 3) times?

I have some data (total N = 100,000 rows). I randomly selected 10% from it to become the validation set to help identify the best set of hyperparameters. To do that I am conducting grid search based ...
1
vote
0answers
109 views

Can LogisticRegressionCV be used with StandardScaler?

If we apply StandardScaler to transform the training data before we fit the LogisticRegressionCV model, I think it is incorrect ...
1
vote
0answers
37 views

Fitting Gaussian process with varying sample density

I have some underlying function of parameters $\theta_i$ that I'm trying to minimize. I sample this function using a latin hypercube and then, using some acquisition function, I obtain successive ...
1
vote
0answers
52 views

Log or MSE loss for hyperparameter tuning of probabilistic NN

I am building a predictive model of a dynamical system using a NN whose output neurons enconde the mean and diagonal covariance of a Gaussian distribution. For training, the negative log prediction ...
1
vote
0answers
97 views

AIC based model selection, hyperparameter optimization and in-sample prediction

I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following: I have two input variables (A and B), and a single target variable. All variables ...
1
vote
0answers
59 views

Range of Search Space for the hyperparameters of Support Vector Regresssion (SVR)

I need to know "what should be the practical range of c, gamma and epsilon hyper-parameters during grid search optimization in SVR". The range of dependent variable lies between 1 to 300 with mean ...
1
vote
0answers
104 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
1
vote
0answers
27 views

What are the relevant criteria to compare neural nets with different hyper parameter settings?

I want to compare different hyperparameter settings on the same network and the same task to get an impression of what works good and what works better. I am comparing different initializer, ...