Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
0
votes
1answer
34 views

Use of Hyper-Parameter Tuning in Deep Learning in Practice

[While this question on Cross-Validated might look similar to my question, I am asking something different.] I have read several books and dozens of blogs on Deep Learning (DL) but it is very rare to ...
0
votes
0answers
38 views

Is this Feature Selection + Hyperparameter Tuning pipeline correct?

I'm currently working on a project which deals with the prediction of absenteeism of workers. The general outline of the project is the following: Get theoretical knowledge of reasons for absenteeism ...
3
votes
0answers
27 views

Bayesian Hyperparameter Optimization. What makes it “bayesian”?

I'm using some bayesian hyperparameter optimization. I know how they works . They always calculate the next values of the hyperparameter dependent on the result of former evaluations. But what makes ...
0
votes
0answers
11 views

Hyperparameter-free method for Moving Average/ Exponential smoothing?

I want to find hyperparameter-free method for Moving Average/ Exponential smoothing. Is there any related paper or python code? S(t)= alpha * F(t) + (1-alpha) * S(t-1) Any methods can avoid the ...
1
vote
3answers
32 views

What happen if I choose the hyperparameters of a classifier based on lowest generalization error?

In this question, the OP asked about a situation that he/she combined training and test datasets into an agumented dataset and then tuned the hyperparameters for best accuracy and then use the ...
0
votes
1answer
15 views

how to implement a one-to-many LSTM network with output of different dimension at each time step

I'm trying to replicate the controller described in "NEURAL ARCHITECTURE SEARCH WITH REINFORCEMENT LEARNING"(link) to find the best set of hyperparameters for a CNN. The controller is a LSTM network, ...
1
vote
1answer
40 views

The best number of nodes in bottleneck layer in Autoencoder

I would like to perform dimensionality reduction using autoencoders (similar to PCA) and I am not sure how many components are optimal i.e. what should be the size of the bottleneck layer. I was ...
8
votes
1answer
302 views

In linear regression why does regularisation penalise the parameter values as well?

Currently learning ridge regression and I was a little confused about the penalisation of more complex models (or the definition of a more complex model). From what I understand, model complexity ...
0
votes
0answers
20 views

One standard error rule multiple hyperparameters

The one standard error rule for selecting the hyperparameter value after a cross-validation search for the LASSO or ridge regression's $\lambda$ is widely known and used. Is there an analog for this ...
0
votes
0answers
8 views

Bias and variance decomposition to estimate prediction

There are various ways that statisticians have come up for bias vs. variance decomposition in terms of prediction estimation. My question is this, how can one leverage or create a loss function based ...
1
vote
1answer
22 views

Probabilistic prediction with specified utility function

This question is related, but not the same as link. I have read a lot of posts here as well as a post from Frank Harell. It is very clear to me that Accuracy is not a great metric to use, probability ...
1
vote
1answer
33 views

Is it possible to find hyperparameters and evaluate final model on test data, while training on all data?

How would I go about tuning my neural network hyperparameters, getting an estimate of its performance on unseen data, while finally training on the entire dataset? The only way I can think of is maybe ...
0
votes
0answers
64 views

DLM regression with parameter restriction

Good afternoon, I am attempting to fit a state space regression model of the form: $Y_{t} = \beta_{1}Y_{t-1} + (1-\beta_{1})[i^* + \beta_{2}X_{t}] + \epsilon_{1,t}$ $i^* = i^*_{t-1} + \epsilon_{2,t}...
0
votes
1answer
91 views

How can we determine the appropriate number of hidden layers, kernels in convolutional neural network (CNN)?

I have checked a lot of questions here and in other websites. What I concluded is that there is no rules for choosing the right number of hyper-parameters in CNN, all what can we do is just trying ...
0
votes
0answers
19 views

Selecting SVM parameters if training data is oversampled/undersampled

I am working on classification for highly imbalanced data. Let's say I have a strategy to oversample/undersample the training data. I plan to use an SVM classifier to perform the classification. Now, ...
2
votes
2answers
46 views

Randomized search on big dataset

I have a dataset of 700,000 rows that Im applying random search on. My parameter grid looks like this: ...
0
votes
0answers
34 views

cross-validation: feature selection and hyperparameter tuning. Is nesting necessary?

I am a little bit confused by the use of feature selection inside a K-fold CV together with hyperparameter tuning. So I have my dataset. I split in training & test as usual, and work on training ...
1
vote
0answers
70 views

Can LogisticRegressionCV be used with StandardScaler?

If we apply StandardScaler to transform the training data before we fit the LogisticRegressionCV model, I think it is incorrect ...
0
votes
1answer
74 views

Is a cross-validation list needed when there are no hyperparameters

Original problem formulation - 12/04/2019: I have a data set of 14 observations with six features each and an output variable of two classes. I applied a logistic regression model with the following ...
1
vote
0answers
27 views

Fitting Gaussian process with varying sample density

I have some underlying function of parameters $\theta_i$ that I'm trying to minimize. I sample this function using a latin hypercube and then, using some acquisition function, I obtain successive ...
0
votes
0answers
19 views

Doing hyperparameter optimization, the smart way

Once our machine learning model is build, it takes quite a while to fine tune the hyper parameters to get good results. Though techniques like cross validation do exist, I wonder if there are some ...
3
votes
1answer
66 views

Does XGBoost have a max-depth hyper-parameter?

According to the explanation in Complete Guide to Parameter Tuning in XGBoost, XGBoost doesn't use max_depth argument as Random Forest or GBM does. It expands the ...
0
votes
0answers
49 views

Still Overfitting SVM with Cross-Validation and Grid Search

I am relatively new to machine learning and am trying to implement an SVM for the first time on a project, but I'm running into some overfitting-related issues. Basically, I created a function called ...
0
votes
1answer
42 views

How to do k-fold cross validation to get optimal specification in a random forest model?

i am an R beginner and i have to do a 5 or 10-fold cross validation in a random forest model. My problem is i have to do the cv manually and not with an package. What i want to do is: 1. Building k-...
3
votes
1answer
55 views

Is it OK to tune the k parameter in PCA?

Principal Component Analysis (PCA) is used to reduce n-dimensional data to k-dimensional data to speed things up in machine learning. After PCA is applied, one can check how much of the variance of ...
0
votes
0answers
74 views

Problem about tuning hyper-parametres

I have tried GridSearchCV and BayesSearchCV for tuning my lightGBM algorithm (for binary classification). I have used 10 iterations and I have indicated scoring ="roc_auc" In the first iteration, I ...
3
votes
1answer
103 views

what exactly happens during each epoch in neural network training

1.Across different epochs, which of the following is/are updated? initial weights (initial ConvNet filter matrices, initial fully connected weights) hyper parameters: number of ConvNet filters, size ...
5
votes
2answers
154 views

What to treat as (hyper-)parameter and why

I have been wondering about the differences between model paramters and model hyperparameters, as well as what their categorization means for a learning problem. Is the distinction between model ...
1
vote
1answer
53 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
2
votes
1answer
36 views

How does cross-validation work exactly?

I'm having a hard time figuring out how exactly cross validation works in practice: To do K-fold cross validation on a data set, you divide your data into K sets. Then for each fold $i$, $1 \leq i \...
1
vote
1answer
41 views

Should overfitting or underfitting be concerned during hyperparameters tuning

I have built an ANN model using Keras. The problem I'm solving is a regression problem and now I'm trying to tune the hyperparameters. I've found better approaches than using a Grid Search - Hyperopt /...
0
votes
1answer
13 views

Neural architecture search dataset

I have recently started looking more into autoML, where we have a "Controller" system, who outputs architectures and hyperparameters, and is given a reward based on the performance of a system trained ...
0
votes
0answers
19 views

Hyperparameter optimization from experiments with random parameters

I'm trying to get a feel for how to tune fastText parameters to get the best precision, particularly P@1, for categorization of a text corpus. I've experimented with random settings for six of its ...
1
vote
0answers
26 views

Log or MSE loss for hyperparameter tuning of probabilistic NN

I am building a predictive model of a dynamical system using a NN whose output neurons enconde the mean and diagonal covariance of a Gaussian distribution. For training, the negative log prediction ...
0
votes
0answers
170 views

Tuning glmnet hyperparameters in MLR

I want to estimate LASSO using glmnet in MLR with spatial cross-validation to tune lambda. Questions: In makeParamSet, do I specify ...
3
votes
0answers
65 views

Analyse sensitivity of hyper-parameters of Machine Learning Models

I want to analyse how sensitive my non neural net machine learning models are to the choice of the different parameters. I am currently using grid search to tune the models. Is there any method that I ...
0
votes
1answer
42 views

Retuning hyperparameters of the baseline when comparing it with a new model

I have a baseline model which has certain hyperparameters to tune (it's actually a neural network, but I don't know if it's important in this context). I want to compare it with my own extension of ...
2
votes
1answer
54 views

Tuning distance threshold in online clustering

In online clustering there are approaches where a threshold $r$ on the distance to the nearest cluster is used to determine whether a new data point should be associated to an existing cluster or ...
0
votes
0answers
48 views

How is dev set used to tune hyperparameters?

I'm new to the deep learning domain and still did not understand clearly enough the idea behind the dev set. I read that dev set is usually used to tune hyperparameters and to compare the performance ...
13
votes
2answers
2k views

Is decision threshold a hyperparameter in logistic regression?

Predicted classes from (binary) logistic regression are determined by using a threshold on the class membership probabilities generated by the model. As I understand it, typically 0.5 is used by ...
0
votes
0answers
29 views

Standard Error Estimated for Repeated K-fold Cross Validation? (Tuning Parameter Selection)

When using k-fold CV to select a tuning parameter, a "one-standard error rule" can be applied (Friedman, Hastie, and Tibshirani (2010), Breiman et al (1984)). For a given parameter $\theta$ the CV-...
0
votes
0answers
236 views

LinearSVC hyperparameters Optimization using HyperOpt on python

i am try to optimize a LinearSVC hyperparameter C by using HyperOpt library on python and i don't know which range to put to the C. I am using the loguniform distribution implemented in the HyperOpt ...
3
votes
3answers
55 views

Must all supervised algorithms have (complexity) parameters?

I have noticed that all commonly used supervised algorithms (decision tree, logistic regression, random forest, ...) have some (hyper)parameters to tune (otherwise the model may underfit or overfit ...
0
votes
0answers
40 views

Nested k-fold cross validation: How to choose hyperparameter for a SVM

I am currently trying to understand how exactly to use nested k-fold cross validation for hyperparameter tuning / model selection. There is one aspect I really cannot get my head around. I found ...
0
votes
1answer
62 views

Hyperparameter tuning using grid search/randomised search

I am conducting hyperparameter tuning for my XGBClassifier model for a multi-class classification problem using scikit-learn ...
0
votes
0answers
57 views

Confused about hyperparameter selection for elastic net regularization using glmnet

I am following the glmnet tutorial here and confused about the statement: We see that lasso (alpha=1) does about the best here. We also see that the range of ...
1
vote
1answer
47 views

Fitting model on whole dataset, more or less epochs ? (w.r.t validation accuracy) [duplicate]

When tuning my neural networks hyperparameters I use 20% of the data set as validation data. With the holdout set I observe the validation accuracy and validation loss. In my case the model starts ...
9
votes
1answer
128 views

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

In François Chollet's Deep Learning with Python it says: As a result, tuning the configuration of the model based on its performance on the validation set can quickly result in overfitting to ...
0
votes
1answer
15 views

At what stage should you do hyper-parameter optimisation as part of the KDD process

I am comparing multiple regression machine learning algorithms (MLA) for a project. I have been reading Geron's excellent book 'Hands-On Machine Learning with Scikit-Learn & Tensorflow'. He ...
0
votes
1answer
145 views

Question about Validation Set for hyperparameter tuning

Okay, I'm still a bit confused as to this Training/Validation/Test Set split. I might be wrong here, but from what I understand, the model is first applied to the Training set, to "learn" from it and ...