Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

1
vote
1answer
8 views

The best number of nodes in bottleneck layer in Autoencoder

I would like to perform dimensionality reduction using autoencoders (similar to PCA) and I am not sure how many components are optimal i.e. what should be the size of the bottleneck layer. I was ...
8
votes
1answer
286 views

In linear regression why does regularisation penalise the parameter values as well?

Currently learning ridge regression and I was a little confused about the penalisation of more complex models (or the definition of a more complex model). From what I understand, model complexity ...
0
votes
0answers
16 views

One standard error rule multiple hyperparameters

The one standard error rule for selecting the hyperparameter value after a cross-validation search for the LASSO or ridge regression's $\lambda$ is widely known and used. Is there an analog for this ...
0
votes
0answers
7 views

Bias and variance decomposition to estimate prediction

There are various ways that statisticians have come up for bias vs. variance decomposition in terms of prediction estimation. My question is this, how can one leverage or create a loss function based ...
1
vote
1answer
17 views

Probabilistic prediction with specified utility function

This question is related, but not the same as link. I have read a lot of posts here as well as a post from Frank Harell. It is very clear to me that Accuracy is not a great metric to use, probability ...
1
vote
1answer
33 views

Is it possible to find hyperparameters and evaluate final model on test data, while training on all data?

How would I go about tuning my neural network hyperparameters, getting an estimate of its performance on unseen data, while finally training on the entire dataset? The only way I can think of is maybe ...
0
votes
0answers
60 views

DLM regression with parameter restriction

Good afternoon, I am attempting to fit a state space regression model of the form: $Y_{t} = \beta_{1}Y_{t-1} + (1-\beta_{1})[i^* + \beta_{2}X_{t}] + \epsilon_{1,t}$ $i^* = i^*_{t-1} + \epsilon_{2,t}...
0
votes
1answer
32 views

How can we determine the appropriate number of hidden layers, kernels in convolutional neural network (CNN)?

I have checked a lot of questions here and in other websites. What I concluded is that there is no rules for choosing the right number of hyper-parameters in CNN, all what can we do is just trying ...
0
votes
0answers
16 views

Selecting SVM parameters if training data is oversampled/undersampled

I am working on classification for highly imbalanced data. Let's say I have a strategy to oversample/undersample the training data. I plan to use an SVM classifier to perform the classification. Now, ...
2
votes
2answers
39 views

Randomized search on big dataset

I have a dataset of 700,000 rows that Im applying random search on. My parameter grid looks like this: ...
0
votes
0answers
14 views

cross-validation: feature selection and hyperparameter tuning. Is nesting necessary?

I am a little bit confused by the use of feature selection inside a K-fold CV together with hyperparameter tuning. So I have my dataset. I split in training & test as usual, and work on training ...
0
votes
0answers
39 views

Can LogisticRegressionCV be used with StandardScaler?

If we apply StandardScaler to transform the training data before we fit the LogisticRegressionCV model, I think it is incorrect ...
0
votes
1answer
70 views

Is a cross-validation list needed when there are no hyperparameters

Original problem formulation - 12/04/2019: I have a data set of 14 observations with six features each and an output variable of two classes. I applied a logistic regression model with the following ...
1
vote
0answers
24 views

Fitting Gaussian process with varying sample density

I have some underlying function of parameters $\theta_i$ that I'm trying to minimize. I sample this function using a latin hypercube and then, using some acquisition function, I obtain successive ...
0
votes
0answers
18 views

Doing hyperparameter optimization, the smart way

Once our machine learning model is build, it takes quite a while to fine tune the hyper parameters to get good results. Though techniques like cross validation do exist, I wonder if there are some ...
3
votes
1answer
55 views

Does XGBoost have a max-depth hyper-parameter?

According to the explanation in Complete Guide to Parameter Tuning in XGBoost, XGBoost doesn't use max_depth argument as Random Forest or GBM does. It expands the ...
0
votes
0answers
38 views

Still Overfitting SVM with Cross-Validation and Grid Search

I am relatively new to machine learning and am trying to implement an SVM for the first time on a project, but I'm running into some overfitting-related issues. Basically, I created a function called ...
0
votes
1answer
34 views

How to do k-fold cross validation to get optimal specification in a random forest model?

i am an R beginner and i have to do a 5 or 10-fold cross validation in a random forest model. My problem is i have to do the cv manually and not with an package. What i want to do is: 1. Building k-...
3
votes
1answer
50 views

Is it OK to tune the k parameter in PCA?

Principal Component Analysis (PCA) is used to reduce n-dimensional data to k-dimensional data to speed things up in machine learning. After PCA is applied, one can check how much of the variance of ...
0
votes
0answers
56 views

Problem about tuning hyper-parametres

I have tried GridSearchCV and BayesSearchCV for tuning my lightGBM algorithm (for binary classification). I have used 10 iterations and I have indicated scoring ="roc_auc" In the first iteration, I ...
3
votes
1answer
68 views

what exactly happens during each epoch in neural network training

1.Across different epochs, which of the following is/are updated? initial weights (initial ConvNet filter matrices, initial fully connected weights) hyper parameters: number of ConvNet filters, size ...
5
votes
2answers
148 views

What to treat as (hyper-)parameter and why

I have been wondering about the differences between model paramters and model hyperparameters, as well as what their categorization means for a learning problem. Is the distinction between model ...
1
vote
1answer
40 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
2
votes
1answer
27 views

How does cross-validation work exactly?

I'm having a hard time figuring out how exactly cross validation works in practice: To do K-fold cross validation on a data set, you divide your data into K sets. Then for each fold $i$, $1 \leq i \...
1
vote
1answer
36 views

Should overfitting or underfitting be concerned during hyperparameters tuning

I have built an ANN model using Keras. The problem I'm solving is a regression problem and now I'm trying to tune the hyperparameters. I've found better approaches than using a Grid Search - Hyperopt /...
0
votes
1answer
10 views

Neural architecture search dataset

I have recently started looking more into autoML, where we have a "Controller" system, who outputs architectures and hyperparameters, and is given a reward based on the performance of a system trained ...
0
votes
0answers
19 views

Hyperparameter optimization from experiments with random parameters

I'm trying to get a feel for how to tune fastText parameters to get the best precision, particularly P@1, for categorization of a text corpus. I've experimented with random settings for six of its ...
1
vote
0answers
17 views

Log or MSE loss for hyperparameter tuning of probabilistic NN

I am building a predictive model of a dynamical system using a NN whose output neurons enconde the mean and diagonal covariance of a Gaussian distribution. For training, the negative log prediction ...
0
votes
0answers
108 views

Tuning glmnet hyperparameters in MLR

I want to estimate LASSO using glmnet in MLR with spatial cross-validation to tune lambda. Questions: In makeParamSet, do I specify ...
2
votes
0answers
41 views

Analyse sensitivity of hyper-parameters of Machine Learning Models

I want to analyse how sensitive my non neural net machine learning models are to the choice of the different parameters. I am currently using grid search to tune the models. Is there any method that I ...
0
votes
1answer
36 views

Retuning hyperparameters of the baseline when comparing it with a new model

I have a baseline model which has certain hyperparameters to tune (it's actually a neural network, but I don't know if it's important in this context). I want to compare it with my own extension of ...
2
votes
1answer
41 views

Tuning distance threshold in online clustering

In online clustering there are approaches where a threshold $r$ on the distance to the nearest cluster is used to determine whether a new data point should be associated to an existing cluster or ...
0
votes
0answers
40 views

How is dev set used to tune hyperparameters?

I'm new to the deep learning domain and still did not understand clearly enough the idea behind the dev set. I read that dev set is usually used to tune hyperparameters and to compare the performance ...
12
votes
2answers
1k views

Is decision threshold a hyperparameter in logistic regression?

Predicted classes from (binary) logistic regression are determined by using a threshold on the class membership probabilities generated by the model. As I understand it, typically 0.5 is used by ...
0
votes
0answers
27 views

Standard Error Estimated for Repeated K-fold Cross Validation? (Tuning Parameter Selection)

When using k-fold CV to select a tuning parameter, a "one-standard error rule" can be applied (Friedman, Hastie, and Tibshirani (2010), Breiman et al (1984)). For a given parameter $\theta$ the CV-...
0
votes
0answers
145 views

LinearSVC hyperparameters Optimization using HyperOpt on python

i am try to optimize a LinearSVC hyperparameter C by using HyperOpt library on python and i don't know which range to put to the C. I am using the loguniform distribution implemented in the HyperOpt ...
3
votes
3answers
54 views

Must all supervised algorithms have (complexity) parameters?

I have noticed that all commonly used supervised algorithms (decision tree, logistic regression, random forest, ...) have some (hyper)parameters to tune (otherwise the model may underfit or overfit ...
0
votes
0answers
32 views

Nested k-fold cross validation: How to choose hyperparameter for a SVM

I am currently trying to understand how exactly to use nested k-fold cross validation for hyperparameter tuning / model selection. There is one aspect I really cannot get my head around. I found ...
0
votes
1answer
45 views

Hyperparameter tuning using grid search/randomised search

I am conducting hyperparameter tuning for my XGBClassifier model for a multi-class classification problem using scikit-learn ...
0
votes
0answers
40 views

Confused about hyperparameter selection for elastic net regularization using glmnet

I am following the glmnet tutorial here and confused about the statement: We see that lasso (alpha=1) does about the best here. We also see that the range of ...
1
vote
1answer
46 views

Fitting model on whole dataset, more or less epochs ? (w.r.t validation accuracy) [duplicate]

When tuning my neural networks hyperparameters I use 20% of the data set as validation data. With the holdout set I observe the validation accuracy and validation loss. In my case the model starts ...
9
votes
1answer
126 views

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

In François Chollet's Deep Learning with Python it says: As a result, tuning the configuration of the model based on its performance on the validation set can quickly result in overfitting to ...
0
votes
1answer
15 views

At what stage should you do hyper-parameter optimisation as part of the KDD process

I am comparing multiple regression machine learning algorithms (MLA) for a project. I have been reading Geron's excellent book 'Hands-On Machine Learning with Scikit-Learn & Tensorflow'. He ...
0
votes
1answer
81 views

Question about Validation Set for hyperparameter tuning

Okay, I'm still a bit confused as to this Training/Validation/Test Set split. I might be wrong here, but from what I understand, the model is first applied to the Training set, to "learn" from it and ...
0
votes
0answers
21 views

Can existing hyperparameters be used when new features are added to data?

Lets say I have a 1D CNN and a dataset on which I have run bayesian optimiztion and I have the best hyperparameters (decided by lowest loss). Now if I decide to add new features to the data, keep the ...
5
votes
1answer
289 views

Hyperparameter Optimization Using Gaussian Processes

I have a dataset that is divided into training and validation dataset. I am using Gaussian Processes to perform hyperparameter optimization. So I am using the accuracy recorded on the validation ...
2
votes
0answers
46 views

My roc is low while precision and recall are high.Why is it so?

I bulit a naive bayes classifier from 60k vectors of text and each of the text is a 2000 dimension vector(Used bag of words for vectorization). Used simple cross validator to find the best alpha and ...
0
votes
1answer
541 views

Cross validation and train test split

I am having a fundamental doubt about cross validation. I know that cross validation trains the model on dataset keeping aside a part of it for testing the model and each for each iteration the train/...
2
votes
0answers
30 views

Choosing Gaussian PDF basis bandwidth depending on number of bases and range of data

Summary (details below!) I have a basis expansion of $m$ (univariate) Gaussian PDFs to model the density of a sample $X$. The means of these PDFs are spaced equidistantly through the domain of $X$ ...
1
vote
0answers
56 views

AIC based model selection, hyperparameter optimization and in-sample prediction

I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following: I have two input variables (A and B), and a single target variable. All variables ...