Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
1
vote
1answer
50 views

Cross-validation for (hyper)parameter tuning to be performed in validation set or training set?

I am learning about the use of cross-validation with grid-search to choose the best hyperparameter for SVM. The problem I came across is the references and examples of its application do not follow a ...
0
votes
0answers
11 views

Random Number generator for splitting data during hyperparameter tuning of Neural Network

Should random number generator be set to any specific value(for train-validation split) during hyperparameter tuning of Neural Networks with Cross-Validation?
1
vote
2answers
109 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
0
votes
0answers
13 views

Why my binary classification neural network performance oscillates a lot through epochs? [duplicate]

I am training a CNN with Keras, vgg16-like model and i don't understand the results. For example, in epoch 15 i have good results but in 14 and 16 it's horrible (you can see it in the loss). What ...
0
votes
0answers
102 views

Neural Network - Choosing best model - But what about epochs?

I am trying to construct a neural net. I have several questions about the procedure. I can get results from a manual model, by switching parameters with my instincts and running each time. I am using ...
0
votes
1answer
25 views

The idea behind sk-learn's combined grid-search and cross-validated estimators?

I am trying incorporate a formal strategy to find the most optimal set hyper-parameters for a machine learning algorithm. I understand you can either do a grid-search or a k-fold cross validation, ...
0
votes
1answer
27 views

class_weight = 'balanced' if GridSearch on unbalanced data set?

I'm trying to optimize the hyperparameters of an SVM. I have an unbalanced data set with more than two classes. In some classes very many samples are included in others very few. Using GridSearchCV, I ...
0
votes
0answers
13 views

Range of parameters for hyperparameter optmization in fully connected layers

I have designed a variational autoencoder with 2D convolutions in the encoder and decoder. I have trained this autoenocder on 50'000 unlabelled images (64 x 80). Now, I would like to use this ...
2
votes
1answer
34 views

What are some of the most correct/accepted ways to tune and compare different models in an academic context?

Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of ...
1
vote
1answer
247 views

Early stopping together with hyperparameter tuning in neural networks

Similar to this question (hyperparameter tuning in neural networks), I have a neural network with a similar list of parameters as the link above: Learning rate: $[0.001, 0.01, 0.1]$ $L_1$ penalty: $[...
0
votes
0answers
8 views

Significance of hyper parameters in the DHR model in R forecast package

The Dynamic Harmonic Regression model in R requires the input of parameters K, the length of which depends on the number of seasonality in the forecast data. According to https://otexts.com/fpp2/dhr....
0
votes
1answer
23 views

Validation set for hyperparameter tuning of ML time series model

I'm developing an ML-based model to forecast the daily sales of a whole month. This model takes as input a set of precomputed time series features: day_of_week, <...
0
votes
0answers
12 views

Tune hyper parameters using cross validation

I have around 228 samples and 100 features in total. I wanted to do some stability analysis on the data, so I did repeated 10-fold cross validation on the entire dataset and obtained the mean ...
0
votes
1answer
66 views

Choosing the optimal set of initial weights in a neural network

I am developing a neural network for pattern recognition in Matlab. Currently: I divide my dataset into 6 folds (5 folds CV + 1 fold Test) I choose 10 different number of hidden neurons I choose 10 ...
2
votes
1answer
48 views

Optimizing parameters of fully connected layers

I have a conceptual problem choosing the best strategy for training a neural network. So, let me explain my situation. I have trained an autoencoder on a huge (unlabeled) dataset using train and ...
2
votes
0answers
14 views

How to optimize hyperparameters in stacked model?

I was wondering whether somebody could explain how to optimize hyperparameters for the base learners and meta algorithm when stacking? In many tutorials they seem to be plucked out of thin air! ...
0
votes
1answer
27 views

How to correctly find the best hyperparameter combination when training a neural network?

I am not sure whether this is the right place to ask this question, so feel free to redirect me if not. What I'm doing is bench-marking a model (MobileNet v2 100 224) in terms of performance - size ...
0
votes
1answer
267 views

Optimizing parameters for CNN autoencoder based on training and validation loss

I have designed an autoencoder with a encoder and decoder consiting of 2D convolutational layers (the input are 40'000 2D images). I train the autoencoder using adam optimizer. The autoencoders has ...
0
votes
1answer
40 views

Use of Hyper-Parameter Tuning in Deep Learning in Practice

[While this question on Cross-Validated might look similar to my question, I am asking something different.] I have read several books and dozens of blogs on Deep Learning (DL) but it is very rare to ...
0
votes
0answers
74 views

Is this Feature Selection + Hyperparameter Tuning pipeline correct?

I'm currently working on a project which deals with the prediction of absenteeism of workers. The general outline of the project is the following: Get theoretical knowledge of reasons for absenteeism ...
3
votes
0answers
33 views

Bayesian Hyperparameter Optimization. What makes it “bayesian”?

I'm using some bayesian hyperparameter optimization. I know how they works . They always calculate the next values of the hyperparameter dependent on the result of former evaluations. But what makes ...
0
votes
0answers
26 views

Hyperparameter-free method for Moving Average/ Exponential smoothing?

I want to find hyperparameter-free method for Moving Average/ Exponential smoothing. Is there any related paper or python code? S(t)= alpha * F(t) + (1-alpha) * S(t-1) Any methods can avoid the ...
1
vote
3answers
46 views

What happen if I choose the hyperparameters of a classifier based on lowest generalization error?

In this question, the OP asked about a situation that he/she combined training and test datasets into an agumented dataset and then tuned the hyperparameters for best accuracy and then use the ...
0
votes
1answer
25 views

how to implement a one-to-many LSTM network with output of different dimension at each time step

I'm trying to replicate the controller described in "NEURAL ARCHITECTURE SEARCH WITH REINFORCEMENT LEARNING"(link) to find the best set of hyperparameters for a CNN. The controller is a LSTM network, ...
2
votes
1answer
310 views

The best number of nodes in bottleneck layer in Autoencoder

I would like to perform dimensionality reduction using autoencoders (similar to PCA) and I am not sure how many components are optimal i.e. what should be the size of the bottleneck layer. I was ...
9
votes
1answer
341 views

In linear regression why does regularisation penalise the parameter values as well?

Currently learning ridge regression and I was a little confused about the penalisation of more complex models (or the definition of a more complex model). From what I understand, model complexity ...
0
votes
0answers
33 views

One standard error rule multiple hyperparameters

The one standard error rule for selecting the hyperparameter value after a cross-validation search for the LASSO or ridge regression's $\lambda$ is widely known and used. Is there an analog for this ...
0
votes
0answers
12 views

Bias and variance decomposition to estimate prediction

There are various ways that statisticians have come up for bias vs. variance decomposition in terms of prediction estimation. My question is this, how can one leverage or create a loss function based ...
1
vote
1answer
32 views

Probabilistic prediction with specified utility function

This question is related, but not the same as link. I have read a lot of posts here as well as a post from Frank Harell. It is very clear to me that Accuracy is not a great metric to use, probability ...
1
vote
1answer
38 views

Is it possible to find hyperparameters and evaluate final model on test data, while training on all data?

How would I go about tuning my neural network hyperparameters, getting an estimate of its performance on unseen data, while finally training on the entire dataset? The only way I can think of is maybe ...
0
votes
0answers
79 views

DLM regression with parameter restriction

Good afternoon, I am attempting to fit a state space regression model of the form: $Y_{t} = \beta_{1}Y_{t-1} + (1-\beta_{1})[i^* + \beta_{2}X_{t}] + \epsilon_{1,t}$ $i^* = i^*_{t-1} + \epsilon_{2,t}...
0
votes
1answer
209 views

How can we determine the appropriate number of hidden layers, kernels in convolutional neural network (CNN)?

I have checked a lot of questions here and in other websites. What I concluded is that there is no rules for choosing the right number of hyper-parameters in CNN, all what can we do is just trying ...
0
votes
0answers
22 views

Selecting SVM parameters if training data is oversampled/undersampled

I am working on classification for highly imbalanced data. Let's say I have a strategy to oversample/undersample the training data. I plan to use an SVM classifier to perform the classification. Now, ...
2
votes
2answers
189 views

Randomized search on big dataset

I have a dataset of 700,000 rows that Im applying random search on. My parameter grid looks like this: ...
0
votes
0answers
74 views

cross-validation: feature selection and hyperparameter tuning. Is nesting necessary?

I am a little bit confused by the use of feature selection inside a K-fold CV together with hyperparameter tuning. So I have my dataset. I split in training & test as usual, and work on training ...
1
vote
0answers
109 views

Can LogisticRegressionCV be used with StandardScaler?

If we apply StandardScaler to transform the training data before we fit the LogisticRegressionCV model, I think it is incorrect ...
0
votes
1answer
83 views

Is a cross-validation list needed when there are no hyperparameters

Original problem formulation - 12/04/2019: I have a data set of 14 observations with six features each and an output variable of two classes. I applied a logistic regression model with the following ...
1
vote
0answers
37 views

Fitting Gaussian process with varying sample density

I have some underlying function of parameters $\theta_i$ that I'm trying to minimize. I sample this function using a latin hypercube and then, using some acquisition function, I obtain successive ...
0
votes
0answers
24 views

Doing hyperparameter optimization, the smart way

Once our machine learning model is build, it takes quite a while to fine tune the hyper parameters to get good results. Though techniques like cross validation do exist, I wonder if there are some ...
3
votes
1answer
168 views

Does XGBoost have a max-depth hyper-parameter?

According to the explanation in Complete Guide to Parameter Tuning in XGBoost, XGBoost doesn't use max_depth argument as Random Forest or GBM does. It expands the ...
0
votes
0answers
72 views

Still Overfitting SVM with Cross-Validation and Grid Search

I am relatively new to machine learning and am trying to implement an SVM for the first time on a project, but I'm running into some overfitting-related issues. Basically, I created a function called ...
0
votes
1answer
107 views

How to do k-fold cross validation to get optimal specification in a random forest model?

i am an R beginner and i have to do a 5 or 10-fold cross validation in a random forest model. My problem is i have to do the cv manually and not with an package. What i want to do is: 1. Building k-...
3
votes
1answer
152 views

Is it OK to tune the k parameter in PCA?

Principal Component Analysis (PCA) is used to reduce n-dimensional data to k-dimensional data to speed things up in machine learning. After PCA is applied, one can check how much of the variance of ...
0
votes
0answers
130 views

Problem about tuning hyper-parametres

I have tried GridSearchCV and BayesSearchCV for tuning my lightGBM algorithm (for binary classification). I have used 10 iterations and I have indicated scoring ="roc_auc" In the first iteration, I ...
3
votes
1answer
343 views

what exactly happens during each epoch in neural network training

1.Across different epochs, which of the following is/are updated? initial weights (initial ConvNet filter matrices, initial fully connected weights) hyper parameters: number of ConvNet filters, size ...
5
votes
2answers
172 views

What to treat as (hyper-)parameter and why

I have been wondering about the differences between model paramters and model hyperparameters, as well as what their categorization means for a learning problem. Is the distinction between model ...
2
votes
1answer
158 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
3
votes
1answer
51 views

How does cross-validation work exactly?

I'm having a hard time figuring out how exactly cross validation works in practice: To do K-fold cross validation on a data set, you divide your data into K sets. Then for each fold $i$, $1 \leq i \...
1
vote
1answer
56 views

Should overfitting or underfitting be concerned during hyperparameters tuning

I have built an ANN model using Keras. The problem I'm solving is a regression problem and now I'm trying to tune the hyperparameters. I've found better approaches than using a Grid Search - Hyperopt /...
0
votes
1answer
15 views

Neural architecture search dataset

I have recently started looking more into autoML, where we have a "Controller" system, who outputs architectures and hyperparameters, and is given a reward based on the performance of a system trained ...

1 2 3 4 5 8