Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
1
vote
0answers
21 views

Decision rule as a hyper-parameter in LASSO

I have a question that is related to the following: Is decision threshold a hyperparameter in logistic regression? but would like some clarification. The general consensus is that the decision rule ...
1
vote
2answers
24 views

Hyperparameter tunning if the validation set is not big enough

Does it make sense to perform hyperparameter tunning if the validation set is not big enough? I know, because that size of the validation set is not big (or maybe representative) enough since the ...
1
vote
0answers
18 views

Choosing a space function for hyperopt

I was originally doing a grid search for my parameter optimization and with 7 parameters being optimized, it would take ages. So I am choosing to use hyperopt at this point. I am however confused on ...
2
votes
1answer
21 views

Should a model be trained until it is stable to find optimal hyperparameters?

A model may take several days to train until it reaches an equilibrium - say if the change in error between epochs is lower than some threshold $\epsilon$, or accuracy reaches some equilibrium. When ...
8
votes
1answer
841 views

Are optimal hyperparameters still optimal for a deeper neural net architecture?

I have found a set of optimal hyperparameters (e.g. learning rate for gradient descent) using cross validation and bayesian optimisation. While searching for the optimal hyperparameters, my neural net ...
1
vote
1answer
42 views

EarlyStopping after GridSearchCV

I want to optimize the hyperparams for a CNN-architecture by using GridSearchCV. As hyperparameters to optimize, I would like to use the learning rate, dropout rate, number of neurons in den dense ...
0
votes
1answer
15 views

Grid Search Combined with Random Search

Is there a way to combine both grid search and random search together ? Lets say I provided a very big range of hyper parameters, can I use random search to minimize this range, and then I follow it ...
1
vote
0answers
22 views

Does it make sense to combine Early Stopping with k-fold cross validation?

I have a CNN architecture for which I want to optimize the hyperparameters such as learning rate, dropout rate and number of epochs. I am thinking of a combination of k-fold cross validation and ...
0
votes
0answers
18 views

Hyper Parameter Tuning - Selecting Ranges of Values

I am working on tuning a machine learning model and want to perform a grid search / hyperparameter tuning on my model to find the best hyperparameters. The literature I have found it pretty good with ...
1
vote
1answer
17 views

Question about “cv” parameter in sklear model and Kfold()

It may sounds like a silly question but let's take the RidgeCV model from sklearn.linear_model. This one has the parameter "cv". cv : int, cross-validation generator or an iterable, optional ...
3
votes
1answer
24 views

Calculating the possible number of configurations

I am wondering how did they get the $19200$ possible configurations? Like, $5^6 = 15625$, where $6$ is the number of hyper-parameters: ps: just to check I'm doing the righ thing. Is this okay? The ...
0
votes
1answer
22 views

Should a training set be used for grid search with cross validation?

I'm looking at an example of using grid search in sklearn, and noticed that after doing train-test splits, the author performs grid search using only the training ...
2
votes
1answer
44 views

Choosing optimal Batch Size : contradicting results?

I'm a grad student in Mathematics, with little background in Machine Learning. I've recently come across the paper "A Disciplined Approach to Neural Network Hyper-Parameters : Part 1" by Leslie Smith, ...
0
votes
1answer
10 views

Standardize data before plotting learning curve

I have implemented cross validation function with hyper parameter tuning. Basically, doing the following: Split the data into 80% training, 20% testing apply cross validation with hyper parameter ...
0
votes
1answer
33 views

EarlyStopping in combination with GridSearchCV für hyperparameter tuning?

I want to find the optimal hyperparameter (dropout rate, learning rate, number of epochs) for training an CNN-architecture. Does it make sense to integrate EarlyStopping already in GridSearchCV? Or ...
1
vote
0answers
13 views

Glmnet R - can't modify fdev parameter when lower = 0 [closed]

I want to solve the following optimisation problem $\hat{\beta} = \arg \min_{\beta \geq 0} \| y- A\beta\|_2^2 + \lambda \|\beta\|_1$ For that, I am using glmnet package (cv.glmnet for finding $\...
2
votes
1answer
23 views

How to explain the difference between parameter and hyperparameter in machine learning? [duplicate]

In practice, this difference is obvious, but how to put it in words?
0
votes
0answers
24 views

Imbalanced training data columns in regression training

I have a training data which after cleaning, wrangling has around 17k with more than 40 columns. The categorical columns are 35 and numerical 5. The categorical columns have value either 0 or 1. Now ...
0
votes
0answers
17 views

How to deal with datasets of different sizes using the same network architecture

I have obtained 15 years worth of temporal data that I am using to build a neural network model. I am currently attempting to determine the best network architecture and hyperparameters, so I am using ...
1
vote
0answers
22 views

Proper Way to Combine Feature Selection and Hyperparameter Tuning?

Been doing reading on feature selection and hyperparameter tuning but I'm getting lost on how to properly code/set up the experiment. I am doing a classified ML experiment, I have 1200 samples and 400 ...
1
vote
1answer
20 views

alternating negative and positive value of slope and y-intercept in gradient descent

I'm working with the following code for gradient descent for simple linear regression: ...
0
votes
0answers
15 views

feature selection and hyper-parameter tuning via cross validation

Recently, I've read many articles or books which deal with cross-validation. But I'm a little bit confused. Generally, when we build a machine, we decide hypothesis sets. And then, we train each model ...
0
votes
0answers
92 views

Change hyperparameter of YOLOv3 for face detection

I have tried with some github implementation on YOLOv3 in tensorflow. I trained yolov3 for faces with WIDER face dataset, I haven't changed the original configuration of YOLOv3. After training the ...
0
votes
0answers
29 views

Estimate distribution of aleatoric variable using Bayesian inference

Given a model as follows: $$y = cx + e$$ where y is the model output, x is the model input, c is an unknown variable and e is a Gaussian model error with zero mean: $$e \sim N(0,\sigma)$$ Data is ...
0
votes
1answer
124 views

Error while performing multiclass classification using Gridsearch CV

I am trying to solve a multiclass classification problem using SVC as the base estimator and GridSearchCV to tune my results. Mentioned below is the code and the error being received: ...
0
votes
0answers
56 views

Bayesian hyperparameter optimization + cross-validation

I want to use Bayesian optimization to search a space of hyperparameters for a neural network model. My objective function for this optimization is validation set accuracy. In addition, I want to ...
1
vote
1answer
45 views

$R^2$ of 1 but RMSE > 0

I am running k-fold cross validation on my training data, and then choosing the best set of hyper parameters, re-training on the training data and testing on a new (unseen) testing data. I am getting ...
1
vote
0answers
17 views

How to select the most optimal hyperparameter in grid-search cross validation if the process is repeated X (i.e. 3) times?

I have some data (total N = 100,000 rows). I randomly selected 10% from it to become the validation set to help identify the best set of hyperparameters. To do that I am conducting grid search based ...
0
votes
0answers
14 views

How do I account for the random weights of my LSTM when tuning its hyperparameters?

It seems to a relative novice like myself that effective use of the more thorough hyperparameter tuning functions (GridSearchCV(), RandomizedSearchCV(), etc.) is stymied by the stochastic nature of ...
0
votes
0answers
10 views

Final all data refit of random forest after hyper-parameter optimisation without out-of-bag

Using a random forrest implementation that does not support out-of-bag errors in combination with a bayesian hyper-parameter method, I am creating random validation datasets during the search. As ...
0
votes
0answers
28 views

Repeated K fold cross validation

I want to use repeated Kfold cross validation in my experiment, since I have a small dataset that might be prone to fluctuation in results of a regular cross valudation regime, so I am opting for a ...
1
vote
1answer
41 views

Cross-validation for (hyper)parameter tuning to be performed in validation set or training set?

I am learning about the use of cross-validation with grid-search to choose the best hyperparameter for SVM. The problem I came across is the references and examples of its application do not follow a ...
0
votes
0answers
10 views

Random Number generator for splitting data during hyperparameter tuning of Neural Network

Should random number generator be set to any specific value(for train-validation split) during hyperparameter tuning of Neural Networks with Cross-Validation?
1
vote
2answers
105 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
0
votes
0answers
13 views

Why my binary classification neural network performance oscillates a lot through epochs? [duplicate]

I am training a CNN with Keras, vgg16-like model and i don't understand the results. For example, in epoch 15 i have good results but in 14 and 16 it's horrible (you can see it in the loss). What ...
0
votes
0answers
46 views

Neural Network - Choosing best model - But what about epochs?

I am trying to construct a neural net. I have several questions about the procedure. I can get results from a manual model, by switching parameters with my instincts and running each time. I am using ...
0
votes
1answer
23 views

The idea behind sk-learn's combined grid-search and cross-validated estimators?

I am trying incorporate a formal strategy to find the most optimal set hyper-parameters for a machine learning algorithm. I understand you can either do a grid-search or a k-fold cross validation, ...
0
votes
1answer
26 views

class_weight = 'balanced' if GridSearch on unbalanced data set?

I'm trying to optimize the hyperparameters of an SVM. I have an unbalanced data set with more than two classes. In some classes very many samples are included in others very few. Using GridSearchCV, I ...
0
votes
0answers
11 views

Range of parameters for hyperparameter optmization in fully connected layers

I have designed a variational autoencoder with 2D convolutions in the encoder and decoder. I have trained this autoenocder on 50'000 unlabelled images (64 x 80). Now, I would like to use this ...
2
votes
1answer
33 views

What are some of the most correct/accepted ways to tune and compare different models in an academic context?

Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of ...
1
vote
1answer
140 views

Early stopping together with hyperparameter tuning in neural networks

Similar to this question (hyperparameter tuning in neural networks), I have a neural network with a similar list of parameters as the link above: Learning rate: $[0.001, 0.01, 0.1]$ $L_1$ penalty: $[...
0
votes
0answers
6 views

Significance of hyper parameters in the DHR model in R forecast package

The Dynamic Harmonic Regression model in R requires the input of parameters K, the length of which depends on the number of seasonality in the forecast data. According to https://otexts.com/fpp2/dhr....
0
votes
1answer
20 views

Validation set for hyperparameter tuning of ML time series model

I'm developing an ML-based model to forecast the daily sales of a whole month. This model takes as input a set of precomputed time series features: day_of_week, <...
0
votes
0answers
11 views

Tune hyper parameters using cross validation

I have around 228 samples and 100 features in total. I wanted to do some stability analysis on the data, so I did repeated 10-fold cross validation on the entire dataset and obtained the mean ...
0
votes
1answer
46 views

Choosing the optimal set of initial weights in a neural network

I am developing a neural network for pattern recognition in Matlab. Currently: I divide my dataset into 6 folds (5 folds CV + 1 fold Test) I choose 10 different number of hidden neurons I choose 10 ...
2
votes
1answer
39 views

Optimizing parameters of fully connected layers

I have a conceptual problem choosing the best strategy for training a neural network. So, let me explain my situation. I have trained an autoencoder on a huge (unlabeled) dataset using train and ...
2
votes
0answers
14 views

How to optimize hyperparameters in stacked model?

I was wondering whether somebody could explain how to optimize hyperparameters for the base learners and meta algorithm when stacking? In many tutorials they seem to be plucked out of thin air! ...
0
votes
1answer
23 views

How to correctly find the best hyperparameter combination when training a neural network?

I am not sure whether this is the right place to ask this question, so feel free to redirect me if not. What I'm doing is bench-marking a model (MobileNet v2 100 224) in terms of performance - size ...
0
votes
1answer
169 views

Optimizing parameters for CNN autoencoder based on training and validation loss

I have designed an autoencoder with a encoder and decoder consiting of 2D convolutational layers (the input are 40'000 2D images). I train the autoencoder using adam optimizer. The autoencoders has ...
0
votes
1answer
39 views

Use of Hyper-Parameter Tuning in Deep Learning in Practice

[While this question on Cross-Validated might look similar to my question, I am asking something different.] I have read several books and dozens of blogs on Deep Learning (DL) but it is very rare to ...