Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

0
votes
0answers
37 views

Problem about tuning hyper-parametres

I have tried GridSearchCV and BayesSearchCV for tuning my lightGBM algorithm (for binary classification). I have used 10 iterations and I have indicated scoring ="roc_auc" In the first iteration, I ...
0
votes
0answers
16 views

Reporting performance after hyperparameter tuning

I have a really small dataset ~500 and I am wondering how to report performance. I need to perform hyperparameter tuning, but I was wondering whether the approach is okay. Approach 1: Can I for ...
3
votes
1answer
50 views

what exactly happens during each epoch in neural network training

1.Across different epochs, which of the following is/are updated? initial weights (initial ConvNet filter matrices, initial fully connected weights) hyper parameters: number of ConvNet filters, size ...
2
votes
0answers
43 views

What to treat as (hyper-)parameter and why

I have been wondering about the differences between model paramters and model hyperparameters, as well as what their categorization means for a learning problem. Is the distinction between model ...
1
vote
1answer
33 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
1
vote
1answer
22 views

How does cross-validation work exactly?

I'm having a hard time figuring out how exactly cross validation works in practice: To do K-fold cross validation on a data set, you divide your data into K sets. Then for each fold $i$, $1 \leq i \...
1
vote
1answer
32 views

Should overfitting or underfitting be concerned during hyperparameters tuning

I have built an ANN model using Keras. The problem I'm solving is a regression problem and now I'm trying to tune the hyperparameters. I've found better approaches than using a Grid Search - Hyperopt /...
0
votes
1answer
10 views

Neural architecture search dataset

I have recently started looking more into autoML, where we have a "Controller" system, who outputs architectures and hyperparameters, and is given a reward based on the performance of a system trained ...
0
votes
0answers
18 views

Hyperparameter optimization from experiments with random parameters

I'm trying to get a feel for how to tune fastText parameters to get the best precision, particularly P@1, for categorization of a text corpus. I've experimented with random settings for six of its ...
1
vote
0answers
12 views

Log or MSE loss for hyperparameter tuning of probabilistic NN

I am building a predictive model of a dynamical system using a NN whose output neurons enconde the mean and diagonal covariance of a Gaussian distribution. For training, the negative log prediction ...
0
votes
0answers
45 views

Tuning glmnet hyperparameters in MLR

I want to estimate LASSO using glmnet in MLR with spatial cross-validation to tune lambda. Questions: In makeParamSet, do I specify ...
2
votes
0answers
27 views

Analyse sensitivity of hyper-parameters of Machine Learning Models

I want to analyse how sensitive my non neural net machine learning models are to the choice of the different parameters. I am currently using grid search to tune the models. Is there any method that I ...
0
votes
1answer
26 views

Retuning hyperparameters of the baseline when comparing it with a new model

I have a baseline model which has certain hyperparameters to tune (it's actually a neural network, but I don't know if it's important in this context). I want to compare it with my own extension of ...
1
vote
1answer
26 views

Tuning distance threshold in online clustering

In online clustering there are approaches where a threshold $r$ on the distance to the nearest cluster is used to determine whether a new data point should be associated to an existing cluster or ...
0
votes
0answers
26 views

How is dev set used to tune hyperparameters?

I'm new to the deep learning domain and still did not understand clearly enough the idea behind the dev set. I read that dev set is usually used to tune hyperparameters and to compare the performance ...
12
votes
2answers
1k views

Is decision threshold a hyperparameter in logistic regression?

Predicted classes from (binary) logistic regression are determined by using a threshold on the class membership probabilities generated by the model. As I understand it, typically 0.5 is used by ...
0
votes
0answers
23 views

Standard Error Estimated for Repeated K-fold Cross Validation? (Tuning Parameter Selection)

When using k-fold CV to select a tuning parameter, a "one-standard error rule" can be applied (Friedman, Hastie, and Tibshirani (2010), Breiman et al (1984)). For a given parameter $\theta$ the CV-...
0
votes
0answers
77 views

LinearSVC hyperparameters Optimization using HyperOpt on python

i am try to optimize a LinearSVC hyperparameter C by using HyperOpt library on python and i don't know which range to put to the C. I am using the loguniform distribution implemented in the HyperOpt ...
2
votes
3answers
47 views

Must all supervised algorithms have (complexity) parameters?

I have noticed that all commonly used supervised algorithms (decision tree, logistic regression, random forest, ...) have some (hyper)parameters to tune (otherwise the model may underfit or overfit ...
0
votes
0answers
20 views

Nested k-fold cross validation: How to choose hyperparameter for a SVM

I am currently trying to understand how exactly to use nested k-fold cross validation for hyperparameter tuning / model selection. There is one aspect I really cannot get my head around. I found ...
0
votes
1answer
39 views

Hyperparameter tuning using grid search/randomised search

I am conducting hyperparameter tuning for my XGBClassifier model for a multi-class classification problem using scikit-learn ...
0
votes
0answers
31 views

Confused about hyperparameter selection for elastic net regularization using glmnet

I am following the glmnet tutorial here and confused about the statement: We see that lasso (alpha=1) does about the best here. We also see that the range of ...
1
vote
1answer
35 views

Fitting model on whole dataset, more or less epochs ? (w.r.t validation accuracy) [duplicate]

When tuning my neural networks hyperparameters I use 20% of the data set as validation data. With the holdout set I observe the validation accuracy and validation loss. In my case the model starts ...
8
votes
1answer
114 views

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

In François Chollet's Deep Learning with Python it says: As a result, tuning the configuration of the model based on its performance on the validation set can quickly result in overfitting to ...
0
votes
1answer
14 views

At what stage should you do hyper-parameter optimisation as part of the KDD process

I am comparing multiple regression machine learning algorithms (MLA) for a project. I have been reading Geron's excellent book 'Hands-On Machine Learning with Scikit-Learn & Tensorflow'. He ...
0
votes
1answer
38 views

Question about Validation Set for hyperparameter tuning

Okay, I'm still a bit confused as to this Training/Validation/Test Set split. I might be wrong here, but from what I understand, the model is first applied to the Training set, to "learn" from it and ...
0
votes
0answers
21 views

Can existing hyperparameters be used when new features are added to data?

Lets say I have a 1D CNN and a dataset on which I have run bayesian optimiztion and I have the best hyperparameters (decided by lowest loss). Now if I decide to add new features to the data, keep the ...
5
votes
1answer
245 views

Hyperparameter Optimization Using Gaussian Processes

I have a dataset that is divided into training and validation dataset. I am using Gaussian Processes to perform hyperparameter optimization. So I am using the accuracy recorded on the validation ...
2
votes
0answers
40 views

My roc is low while precision and recall are high.Why is it so?

I bulit a naive bayes classifier from 60k vectors of text and each of the text is a 2000 dimension vector(Used bag of words for vectorization). Used simple cross validator to find the best alpha and ...
0
votes
1answer
281 views

Cross validation and train test split

I am having a fundamental doubt about cross validation. I know that cross validation trains the model on dataset keeping aside a part of it for testing the model and each for each iteration the train/...
1
vote
0answers
26 views

Choosing Gaussian PDF basis bandwidth depending on number of bases and range of data

Summary (details below!) I have a basis expansion of $m$ (univariate) Gaussian PDFs to model the density of a sample $X$. The means of these PDFs are spaced equidistantly through the domain of $X$ ...
1
vote
0answers
50 views

AIC based model selection, hyperparameter optimization and in-sample prediction

I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following: I have two input variables (A and B), and a single target variable. All variables ...
1
vote
0answers
27 views

Range of Search Space for the hyperparameters of Support Vector Regresssion (SVR)

I need to know "what should be the practical range of c, gamma and epsilon hyper-parameters during grid search optimization in SVR". The range of dependent variable lies between 1 to 300 with mean ...
0
votes
0answers
29 views

Hyperparameters in Gaussian process

Without going to far into the details, I am using a Gaussian Process for the prediction of the posterior given by: $$p(\mathbf{T}\vert\mathbf{X},\boldsymbol{\theta}) = \int p(\mathbf{T}\vert f)p(f\...
0
votes
1answer
113 views

RFE number of features with hyperparameter fine tuning within cros-validation

I would like to use cross-validation to select the number of optimal features to select (n_features_to_select) in the recursive feature elimination algorithm (RFE) and the optimal hyperparameter of an ...
0
votes
1answer
54 views

What are two most important hyper parameters in multiplayer Perceptron?

I know there are different hyperparameters for mlpclassifier, however, if I were to choose two most important one, what would they be for a ...
0
votes
0answers
95 views

Support Vector Machine Kernel Choice and hyper-parameter tuning for high class imbalance data

Thanks for your help in advance. My question is this: Given the below information, is there some kernel preference and particular hyperparameter that is preferable to use when dealing with high class ...
0
votes
0answers
17 views

Number of Bootstrap Samples for Random Forest in Python scikit-learn [duplicate]

We knew that the performance of RF gets better with higher number of bootstrap samples (see here). Any ideas what is the number of bootstrap samples for RF in scikit-learn? Is there anyway we can ...
1
vote
2answers
99 views

Using the standard deviation in Cross Validation

I'm running a Grid Search to find the optimal parameters for xgboost via sklearn. I can see that the grid search picks the set of parameters with lowest mean MSE. The problem is that upon inspecting ...
0
votes
1answer
28 views

Overview of sklearn hyper-parameters

Is there an (un)official overview of sklearn hyper-parameters to tune for each model? I find myself often having to google extensively before getting an exhaustive list for any given model. ...
0
votes
0answers
14 views

What hyperparameters should be sampled (together) for neural networks?

I'm using a neural network for a multi-target regression task and would like to perform hyper-parameter optimization. The network has one hidden layer and uses MSE loss on the output. I have large ...
2
votes
0answers
82 views

Can Bayesian Optimization solve this problem?

Suppose ${\bf{x}} = (x_1,\ldots,x_n)$ and $f({\bf{x}})\propto 1_A({\bf{x}}) \prod_{i=1}^n {x_i}^{\alpha_i-1} e^{-\beta_i x_i}$ , i.e. $f$ is proportional to the product of independent gamma ...
0
votes
0answers
9 views

Evaluate a method taking into account when it fails

I am performing a grid-search (looking all/many variations) of a hyperparameter X to find the optimum value of that parameter. Where X is a symmetric matrix, where I set different values in each ...
0
votes
0answers
35 views

Nested cross-validation: hyperparameter tuning, training and saving the best model for future data prediction?

Is it sensible to do the following: Given: Data: X of size n x d Labels: Y of size n x 1 Goal: Save the best model after hyperparameter tuning for future data, ...
3
votes
1answer
497 views

What is a sensible order for parameter tuning in neural networks?

There are so many aspects one could possibly change in a deep neural network that it is generally not feasible to do a grid search over all of them (e.g. activation function, layer type, number of ...
1
vote
0answers
39 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
0
votes
0answers
24 views

How to train a neural network after the hyperparameters have been found? What is the stopping criteria?

Say for example i use validation score for early stopping to prevent the network from overfitting and find the best hyperparameters using the CV techniques. Now that i have found the best ...
8
votes
3answers
202 views

Why don't we just learn the hyper parameters?

Maybe this is a stupid question. I was implementing a pretty popular paper "EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES" and in the paper, it trains an adversarial objective function J''(θ) = αJ(...
0
votes
0answers
18 views

Should I adjust the hyperparameters to make this neural network more efficient?

I am using the CPU of a rhel7 server with 8 core, 64g memory with nothing really running on it besides my artificial neural network model. This network has about 7 million rows, 4 categorical ...