Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
0
votes
1answer
113 views

Hyperparameter tuning using grid search/randomised search

I am conducting hyperparameter tuning for my XGBClassifier model for a multi-class classification problem using scikit-learn ...
0
votes
0answers
9 views

using the test population as an eval_set when doing hyperparameter optimization

I'm looking at this guide for hyperparameters optimization of boosting regressors using hyperopt. I noticed that for each trial, it uses the following code for the ...
1
vote
1answer
147 views

Can I use cross validation on a subset of the training set to select hyperparameters?

I am using R, and I had a dataset with 400000 rows and 800 columns, training a random forest model with only 100 trees on this dataset will take me about 1 and half hour on my laptop. So I went on and ...
0
votes
0answers
10 views

When tuning an SVM what are reasonable bounds for C and gamma when performing a gridsearch?

I am trying to tune the hyperparamters of an RBF-kernel SVM by utilizing a gridsearch strategy. I found different sources stating different ranges (2^-15, ... 2^15 or 10^-3,...10^3) all they have in ...
0
votes
0answers
9 views

Doesn't matter which epoch number I use during grid search?

I want to optimize the learning rate and dropout rate of a CNN through grid search. I wonder which number of epochs should I choose for grid search? Is this a problem when I use a constant number of ...
1
vote
0answers
27 views

Order of features for gridsearch and model fitting

Assuming that the same columns (i.e., features) are used for hyperparameter tuning and model fitting, and ensemble models are used for modeling (e.g., Random forest or XGboost), then does the order of ...
1
vote
0answers
10 views

Inference on Dirichlet hyper-parameter

I'm working on a Gibbs sampler for a (somewhat custom version of) Latent Dirichlet Allocation model. In short, I have data that comes from a $K$-dimensional Dirichlet-Multinomial distribution, i.e. $$...
1
vote
1answer
12 views

Find weight distribution in multiple term loss

I have a question if it is possible to find/learn the weight distribution in a multiple term loss where each weight models the importance of each term on the total loss. ...
2
votes
1answer
144 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
2
votes
1answer
122 views

Higher Test Scores but Higher Variance?

I am tuning hyper-parameters using 5-fold cross-validated grid search for various multiclass classifiers, and I keep running into the same issue that I can't quite wrap my head around. The hyper-...
0
votes
2answers
44 views

Do Parameters optimization more feasible for industrial purpose than academic?

We use hyperparameter optimization to increase the performance of the data miners, however most of the researchers don't use the parameters tuning as it needs a lot of time and effort. In academic ...
2
votes
1answer
17 views

Model tuning in the presence of incorrect training labels

I have a situation where I have a large amount of labeled data (~40 million records) with a binary outcome variable that has about 50% positive and 50% negative cases. The issue is that I know that ...
1
vote
0answers
14 views

Optimal penalty for finding changepoints with the fused lasso, assuming some probabilistic model?

I am interested in detecting changepoints in a signal using the fused lasso (as implemented here for example). I am in particular interested in getting estimates of changepoints which are close to the ...
7
votes
1answer
123 views

Decision rule as a hyper-parameter in LASSO

I have a question that is related to the following: Is decision threshold a hyperparameter in logistic regression? but would like some clarification. The general consensus is that the decision rule ...
0
votes
0answers
9 views

cross validation for hyperparameter selection: is it correct to split data to k-folds in each try differently?

I am doing hyperparameter selection with bayesian optimization. In each iteration I compute the cross validation error, which is objective to be minimized (or times -1 and will be maximized it does ...
0
votes
1answer
35 views

What do we want to maximize in hyperparameter tuning with bayesian optimization?

So I am trying to implement Bayesian optimization for various machine learning methods, all of then consist of hyperparameters which should be tuned (eg. complexity parameter, minimum samples in split ...
10
votes
3answers
366 views

Why don't we just learn the hyper parameters?

I was implementing a pretty popular paper "EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES" and in the paper, it trains an adversarial objective function J''(θ) = αJ(θ) + (1 − α)J'(θ). It treats α ...
1
vote
2answers
28 views

Hyperparameter tunning if the validation set is not big enough

Does it make sense to perform hyperparameter tunning if the validation set is not big enough? I know, because that size of the validation set is not big (or maybe representative) enough since the ...
1
vote
0answers
24 views

Choosing a space function for hyperopt

I was originally doing a grid search for my parameter optimization and with 7 parameters being optimized, it would take ages. So I am choosing to use hyperopt at this point. I am however confused on ...
2
votes
1answer
23 views

Should a model be trained until it is stable to find optimal hyperparameters?

A model may take several days to train until it reaches an equilibrium - say if the change in error between epochs is lower than some threshold $\epsilon$, or accuracy reaches some equilibrium. When ...
9
votes
1answer
911 views

Are optimal hyperparameters still optimal for a deeper neural net architecture?

I have found a set of optimal hyperparameters (e.g. learning rate for gradient descent) using cross validation and bayesian optimisation. While searching for the optimal hyperparameters, my neural net ...
1
vote
1answer
65 views

EarlyStopping after GridSearchCV

I want to optimize the hyperparams for a CNN-architecture by using GridSearchCV. As hyperparameters to optimize, I would like to use the learning rate, dropout rate, number of neurons in den dense ...
0
votes
1answer
238 views

Question about Validation Set for hyperparameter tuning

Okay, I'm still a bit confused as to this Training/Validation/Test Set split. I might be wrong here, but from what I understand, the model is first applied to the Training set, to "learn" from it and ...
0
votes
1answer
22 views

Grid Search Combined with Random Search

Is there a way to combine both grid search and random search together ? Lets say I provided a very big range of hyper parameters, can I use random search to minimize this range, and then I follow it ...
1
vote
0answers
29 views

Does it make sense to combine Early Stopping with k-fold cross validation?

I have a CNN architecture for which I want to optimize the hyperparameters such as learning rate, dropout rate and number of epochs. I am thinking of a combination of k-fold cross validation and ...
0
votes
0answers
27 views

Hyper Parameter Tuning - Selecting Ranges of Values

I am working on tuning a machine learning model and want to perform a grid search / hyperparameter tuning on my model to find the best hyperparameters. The literature I have found it pretty good with ...
1
vote
1answer
29 views

Question about “cv” parameter in sklear model and Kfold()

It may sounds like a silly question but let's take the RidgeCV model from sklearn.linear_model. This one has the parameter "cv". cv : int, cross-validation generator or an iterable, optional ...
3
votes
1answer
25 views

Calculating the possible number of configurations

I am wondering how did they get the $19200$ possible configurations? Like, $5^6 = 15625$, where $6$ is the number of hyper-parameters: ps: just to check I'm doing the righ thing. Is this okay? The ...
1
vote
1answer
440 views

How to perform feature selection and hyperparameter optimization in cross validation?

note: I read a lot of the questions already posted on this topic, but still have some confusion. I want to perform feature selection and model selection for multiple models e.g. Random forest (RF), ...
43
votes
6answers
21k views

Practical hyperparameter optimization: Random vs. grid search

I'm currently going through Bengio's and Bergsta's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving ...
2
votes
0answers
102 views

evolution of C and gamma in SVR with the size of training examples

Do you know how C and gamma evolve while the size of training examples (X) rises ? I have found C and gamma for 10% and 20% of my data and I would like to save time. Can I determine C and gamma for my ...
2
votes
1answer
64 views

Choosing optimal Batch Size : contradicting results?

I'm a grad student in Mathematics, with little background in Machine Learning. I've recently come across the paper "A Disciplined Approach to Neural Network Hyper-Parameters : Part 1" by Leslie Smith, ...
0
votes
1answer
27 views

Should a training set be used for grid search with cross validation?

I'm looking at an example of using grid search in sklearn, and noticed that after doing train-test splits, the author performs grid search using only the training ...
4
votes
1answer
431 views

General procedures for combined feature selection, model tuning, and model selection?

What is the general procedure for a combined task of model tuning (i.e., hyperparameter selection), feature selection and model selection? I know some basic principles for each task, but when ...
0
votes
1answer
11 views

Standardize data before plotting learning curve

I have implemented cross validation function with hyper parameter tuning. Basically, doing the following: Split the data into 80% training, 20% testing apply cross validation with hyper parameter ...
0
votes
1answer
54 views

EarlyStopping in combination with GridSearchCV für hyperparameter tuning?

I want to find the optimal hyperparameter (dropout rate, learning rate, number of epochs) for training an CNN-architecture. Does it make sense to integrate EarlyStopping already in GridSearchCV? Or ...
1
vote
0answers
13 views

Glmnet R - can't modify fdev parameter when lower = 0 [closed]

I want to solve the following optimisation problem $\hat{\beta} = \arg \min_{\beta \geq 0} \| y- A\beta\|_2^2 + \lambda \|\beta\|_1$ For that, I am using glmnet package (cv.glmnet for finding $\...
2
votes
1answer
24 views

How to explain the difference between parameter and hyperparameter in machine learning? [duplicate]

In practice, this difference is obvious, but how to put it in words?
2
votes
1answer
2k views

Cross validation and train test split

I am having a fundamental doubt about cross validation. I know that cross validation trains the model on dataset keeping aside a part of it for testing the model and each for each iteration the train/...
8
votes
1answer
2k views

What are some of the disavantage of bayesian hyper parameter optimization?

I am fairly new to machine learning and statistics but I was wondering why bayesian optimization is not referred more often online when learning machine learning to optimize your algorithm ...
1
vote
1answer
183 views

Why do we sample from log space when optimizing learning rate + regularization params?

Since I took Karpathy's CS231n I used the method he mentions on the 5th lecture for hyperparameter optimization of neural networks which samples the learning rate and regularization parameters from ...
0
votes
0answers
24 views

Imbalanced training data columns in regression training

I have a training data which after cleaning, wrangling has around 17k with more than 40 columns. The categorical columns are 35 and numerical 5. The categorical columns have value either 0 or 1. Now ...
0
votes
0answers
18 views

How to deal with datasets of different sizes using the same network architecture

I have obtained 15 years worth of temporal data that I am using to build a neural network model. I am currently attempting to determine the best network architecture and hyperparameters, so I am using ...
1
vote
0answers
36 views

Proper Way to Combine Feature Selection and Hyperparameter Tuning?

Been doing reading on feature selection and hyperparameter tuning but I'm getting lost on how to properly code/set up the experiment. I am doing a classified ML experiment, I have 1200 samples and 400 ...
1
vote
1answer
27 views

alternating negative and positive value of slope and y-intercept in gradient descent

I'm working with the following code for gradient descent for simple linear regression: ...
0
votes
0answers
15 views

feature selection and hyper-parameter tuning via cross validation

Recently, I've read many articles or books which deal with cross-validation. But I'm a little bit confused. Generally, when we build a machine, we decide hypothesis sets. And then, we train each model ...
0
votes
0answers
169 views

Change hyperparameter of YOLOv3 for face detection

I have tried with some github implementation on YOLOv3 in tensorflow. I trained yolov3 for faces with WIDER face dataset, I haven't changed the original configuration of YOLOv3. After training the ...
0
votes
0answers
29 views

Estimate distribution of aleatoric variable using Bayesian inference

Given a model as follows: $$y = cx + e$$ where y is the model output, x is the model input, c is an unknown variable and e is a Gaussian model error with zero mean: $$e \sim N(0,\sigma)$$ Data is ...
8
votes
2answers
19k views

How to use XGboost.cv with hyperparameters optimization?

I want to optimize hyperparameters of XGboost using crossvalidation. However, it is not clear how to obtain the model from xgb.cv. For instance I call ...
1
vote
1answer
338 views

Parameter Tuning for Random Forest Text Classifier

I train a binary random forest classifier on skikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold crossvalidation on the training data. Is there any ...