Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Make your voice heard. Take the 2019 Developer Survey now

Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

0
votes
0answers
5 views

LinearSVC hyperparameters Optimization using HyperOpt on python

i am try to optimize a LinearSVC hyperparameter C by using HyperOpt library on python and i don't know which range to put to the C. I am using the loguniform distribution implemented in the HyperOpt ...
2
votes
3answers
40 views

Must all supervised algorithms have (complexity) parameters?

I have noticed that all commonly used supervised algorithms (decision tree, logistic regression, random forest, ...) have some (hyper)parameters to tune (otherwise the model may underfit or overfit ...
0
votes
0answers
7 views

Nested k-fold cross validation: How to choose hyperparameter for a SVM

I am currently trying to understand how exactly to use nested k-fold cross validation for hyperparameter tuning / model selection. There is one aspect I really cannot get my head around. I found ...
0
votes
1answer
28 views

Hyperparameter tuning using grid search/randomised search

I am conducting hyperparameter tuning for my XGBClassifier model for a multi-class classification problem using scikit-learn ...
0
votes
0answers
15 views

Confused about hyperparameter selection for elastic net regularization using glmnet

I am following the glmnet tutorial here and confused about the statement: We see that lasso (alpha=1) does about the best here. We also see that the range of ...
1
vote
1answer
30 views

Fitting model on whole dataset, more or less epochs ? (w.r.t validation accuracy)

When tuning my neural networks hyperparameters I use 20% of the data set as validation data. With the holdout set I observe the validation accuracy and validation loss. In my case the model starts ...
7
votes
1answer
105 views

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

In François Chollet's Deep Learning with Python it says: As a result, tuning the configuration of the model based on its performance on the validation set can quickly result in overfitting to ...
0
votes
1answer
14 views

At what stage should you do hyper-parameter optimisation as part of the KDD process

I am comparing multiple regression machine learning algorithms (MLA) for a project. I have been reading Geron's excellent book 'Hands-On Machine Learning with Scikit-Learn & Tensorflow'. He ...
0
votes
1answer
19 views

Question about Validation Set for hyperparameter tuning

Okay, I'm still a bit confused as to this Training/Validation/Test Set split. I might be wrong here, but from what I understand, the model is first applied to the Training set, to "learn" from it and ...
0
votes
0answers
21 views

Can existing hyperparameters be used when new features are added to data?

Lets say I have a 1D CNN and a dataset on which I have run bayesian optimiztion and I have the best hyperparameters (decided by lowest loss). Now if I decide to add new features to the data, keep the ...
5
votes
1answer
207 views

Hyperparameter Optimization Using Gaussian Processes

I have a dataset that is divided into training and validation dataset. I am using Gaussian Processes to perform hyperparameter optimization. So I am using the accuracy recorded on the validation ...
2
votes
0answers
37 views

My roc is low while precision and recall are high.Why is it so?

I bulit a naive bayes classifier from 60k vectors of text and each of the text is a 2000 dimension vector(Used bag of words for vectorization). Used simple cross validator to find the best alpha and ...
0
votes
1answer
62 views

Cross validation and train test split

I am having a fundamental doubt about cross validation. I know that cross validation trains the model on dataset keeping aside a part of it for testing the model and each for each iteration the train/...
1
vote
0answers
24 views

Choosing Gaussian PDF basis bandwidth depending on number of bases and range of data

Summary (details below!) I have a basis expansion of $m$ (univariate) Gaussian PDFs to model the density of a sample $X$. The means of these PDFs are spaced equidistantly through the domain of $X$ ...
1
vote
0answers
43 views

AIC based model selection, hyperparameter optimization and in-sample prediction

I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following: I have two input variables (A and B), and a single target variable. All variables ...
1
vote
0answers
12 views

Range of Search Space for the hyperparameters of Support Vector Regresssion (SVR)

I need to know "what should be the practical range of c, gamma and epsilon hyper-parameters during grid search optimization in SVR". The range of dependent variable lies between 1 to 300 with mean ...
0
votes
0answers
25 views

Hyperparameters in Gaussian process

Without going to far into the details, I am using a Gaussian Process for the prediction of the posterior given by: $$p(\mathbf{T}\vert\mathbf{X},\boldsymbol{\theta}) = \int p(\mathbf{T}\vert f)p(f\...
0
votes
1answer
55 views

RFE number of features with hyperparameter fine tuning within cros-validation

I would like to use cross-validation to select the number of optimal features to select (n_features_to_select) in the recursive feature elimination algorithm (RFE) and the optimal hyperparameter of an ...
0
votes
1answer
38 views

What are two most important hyper parameters in multiplayer Perceptron?

I know there are different hyperparameters for mlpclassifier, however, if I were to choose two most important one, what would they be for a ...
0
votes
0answers
30 views

Results of cross validation don't consistent, what it mean

I have a data set which has about 100 samples. Each sample has 9 features ($x_1, ..., x_9$) and one targets ($y$). I tried ridge regression on this data set using sklearn. In the regression, the cost ...
0
votes
0answers
64 views

Support Vector Machine Kernel Choice and hyper-parameter tuning for high class imbalance data

Thanks for your help in advance. My question is this: Given the below information, is there some kernel preference and particular hyperparameter that is preferable to use when dealing with high class ...
0
votes
0answers
17 views

Number of Bootstrap Samples for Random Forest in Python scikit-learn [duplicate]

We knew that the performance of RF gets better with higher number of bootstrap samples (see here). Any ideas what is the number of bootstrap samples for RF in scikit-learn? Is there anyway we can ...
1
vote
2answers
72 views

Using the standard deviation in Cross Validation

I'm running a Grid Search to find the optimal parameters for xgboost via sklearn. I can see that the grid search picks the set of parameters with lowest mean MSE. The problem is that upon inspecting ...
0
votes
1answer
27 views

Overview of sklearn hyper-parameters

Is there an (un)official overview of sklearn hyper-parameters to tune for each model? I find myself often having to google extensively before getting an exhaustive list for any given model. ...
0
votes
0answers
8 views

What hyperparameters should be sampled (together) for neural networks?

I'm using a neural network for a multi-target regression task and would like to perform hyper-parameter optimization. The network has one hidden layer and uses MSE loss on the output. I have large ...
2
votes
0answers
75 views

Can Bayesian Optimization solve this problem?

Suppose ${\bf{x}} = (x_1,\ldots,x_n)$ and $f({\bf{x}})\propto 1_A({\bf{x}}) \prod_{i=1}^n {x_i}^{\alpha_i-1} e^{-\beta_i x_i}$ , i.e. $f$ is proportional to the product of independent gamma ...
0
votes
0answers
7 views

Evaluate a method taking into account when it fails

I am performing a grid-search (looking all/many variations) of a hyperparameter X to find the optimum value of that parameter. Where X is a symmetric matrix, where I set different values in each ...
0
votes
0answers
16 views

Denoising Autoencoder Parameter Search

I have ran a hyperparameter search for a denoising autoencoder and the results suggest I should make the sizes of my hidden layers as large as possible (within the range of values I allowed it to ...
0
votes
0answers
16 views

Nested cross-validation: hyperparameter tuning, training and saving the best model for future data prediction?

Is it sensible to do the following: Given: Data: X of size n x d Labels: Y of size n x 1 Goal: Save the best model after hyperparameter tuning for future data, ...
3
votes
1answer
213 views

What is a sensible order for parameter tuning in neural networks?

There are so many aspects one could possibly change in a deep neural network that it is generally not feasible to do a grid search over all of them (e.g. activation function, layer type, number of ...
1
vote
0answers
27 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
0
votes
0answers
21 views

How to train a neural network after the hyperparameters have been found? What is the stopping criteria?

Say for example i use validation score for early stopping to prevent the network from overfitting and find the best hyperparameters using the CV techniques. Now that i have found the best ...
8
votes
3answers
169 views

Why don't we just learn the hyper parameters?

Maybe this is a stupid question. I was implementing a pretty popular paper "EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES" and in the paper, it trains an adversarial objective function J''(θ) = αJ(...
0
votes
0answers
17 views

Should I adjust the hyperparameters to make this neural network more efficient?

I am using the CPU of a rhel7 server with 8 core, 64g memory with nothing really running on it besides my artificial neural network model. This network has about 7 million rows, 4 categorical ...
0
votes
0answers
33 views

Best practice for choosing hyperparameter in cross-validation

I have splitted my whole data into pairs of train set(80% of all data) and validation set(20%).In the end of process of training and validation I have to choose hyperparameter basing on train and ...
1
vote
0answers
26 views

What are the relevant criteria to compare neural nets with different hyper parameter settings?

I want to compare different hyperparameter settings on the same network and the same task to get an impression of what works good and what works better. I am comparing different initializer, ...
0
votes
0answers
142 views

What are the key hyperparameters to tune in CatBoost?

I've used XGBoost for a long time but I'm new to CatBoost. If I wanted to run a sklearn RandomizedSearchCV, what are CatBoost's hyperparameters worthwhile including for a binary classification problem?...
2
votes
1answer
41 views

Distorted hyperpriors when sampling from the prior only

I am currently testing some multilevel models in pymc3 and found that the hyperpriors get distorted when I run the level only to generate the prior. The hyperpriors I am using are generating ...
0
votes
0answers
21 views

When tuning a model with a train/test split, is it “less bad” to resample a train/test split, or use the same split repeatedly?

Sometimes, cross-validation or bootstrapping can be computationally prohibitive. If you need to estimate out-of-sample performance (e.g. accuracy), the next-best alternative I'm aware of is to use a ...
0
votes
0answers
7 views

How to fit many datasets at once with a cross-validated pipeline?

Basically, I need to find a pipeline that will not only be the best on one dataset, but to be the best in average across an ensemble of dataset. How would you solve this problem using a cross-...
2
votes
1answer
36 views

Hyperparameter value while computing the test log-likelihood

I have a very basic machine learning question. My likelihood function includes a parameter $\alpha$ which I set to a fixed value and do not learn from the model, which makes it a "hyperparameter". ...
1
vote
0answers
169 views

Gaussian Process Hyperparameter Tuning

I'm planning to use Gaussian Process (GP) to model my case. However, while learning the GP I found out that we have to tuning the hyperparameters to give us the best solution. I have checked several ...
2
votes
0answers
15 views

Change of optimal learning rate with small changes to architecture and data

I am training variants of similar neural networks, which differ slightly in the number of filters or layers. Additionally, the data is sometimes slightly changed using different preprocessing like ...
1
vote
0answers
311 views

Unsupervised anomaly detection - metric for tuning Isolation Forest parameters

I have a project, in which, one of the stages is to find and label anomalous data points, that are likely to be outliers. As a first step, I am using Isolation Forest algorithm, which, after plotting ...
0
votes
1answer
41 views

Hyperparameters tuning on resampled validation set?

Let's suppose we have trained a neural network on some data set, rather than estimating the hyper-parameters using a basic validation error would not it make more sense to: Generate N validation sets ...
0
votes
1answer
161 views

Intuitive explanation of stratified cross validation and nested cross validation

According to the approach outlined here. I should split the dataset into training set and an independent test set using stratified cross-validation. This is the ...
4
votes
1answer
164 views

What is the difference between model selection and hyperparameter tuning?

In the context of supervised learning, in most statistics based texts and papers, one reads about model selection. For example Hastie, Tibshirani and Friedman in ESL define it as: Model Selection: ...
0
votes
0answers
13 views

Bayesian Inversion - choice of likelihood function and whether to invert for standard deviation

Good evening, There are my main questions before a brief explanation of my work: 1. Should I be inverting for multiple standard deviations (for different portions of the data, or even at each data ...
0
votes
0answers
27 views

autoencoder parameters effect matlab

i'm new in autoencoders and matlab i'm applying this tutorial https://www.mathworks.com/help/nnet/ug/construct-deep-network-using-autoencoders.html after extrating the features i'm trying to check ...