Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
0
votes
0answers
431 views

LinearSVC hyperparameters Optimization using HyperOpt on python

i am try to optimize a LinearSVC hyperparameter C by using HyperOpt library on python and i don't know which range to put to the C. I am using the loguniform distribution implemented in the HyperOpt ...
3
votes
3answers
57 views

Must all supervised algorithms have (complexity) parameters?

I have noticed that all commonly used supervised algorithms (decision tree, logistic regression, random forest, ...) have some (hyper)parameters to tune (otherwise the model may underfit or overfit ...
0
votes
1answer
112 views

Hyperparameter tuning using grid search/randomised search

I am conducting hyperparameter tuning for my XGBClassifier model for a multi-class classification problem using scikit-learn ...
1
vote
1answer
67 views

Fitting model on whole dataset, more or less epochs ? (w.r.t validation accuracy) [duplicate]

When tuning my neural networks hyperparameters I use 20% of the data set as validation data. With the holdout set I observe the validation accuracy and validation loss. In my case the model starts ...
9
votes
1answer
151 views

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

In François Chollet's Deep Learning with Python it says: As a result, tuning the configuration of the model based on its performance on the validation set can quickly result in overfitting to ...
0
votes
1answer
21 views

At what stage should you do hyper-parameter optimisation as part of the KDD process

I am comparing multiple regression machine learning algorithms (MLA) for a project. I have been reading Geron's excellent book 'Hands-On Machine Learning with Scikit-Learn & Tensorflow'. He ...
0
votes
1answer
232 views

Question about Validation Set for hyperparameter tuning

Okay, I'm still a bit confused as to this Training/Validation/Test Set split. I might be wrong here, but from what I understand, the model is first applied to the Training set, to "learn" from it and ...
5
votes
1answer
337 views

Hyperparameter Optimization Using Gaussian Processes

I have a dataset that is divided into training and validation dataset. I am using Gaussian Processes to perform hyperparameter optimization. So I am using the accuracy recorded on the validation ...
2
votes
0answers
53 views

My roc is low while precision and recall are high.Why is it so?

I bulit a naive bayes classifier from 60k vectors of text and each of the text is a 2000 dimension vector(Used bag of words for vectorization). Used simple cross validator to find the best alpha and ...
2
votes
1answer
2k views

Cross validation and train test split

I am having a fundamental doubt about cross validation. I know that cross validation trains the model on dataset keeping aside a part of it for testing the model and each for each iteration the train/...
2
votes
0answers
36 views

Choosing Gaussian PDF basis bandwidth depending on number of bases and range of data

Summary (details below!) I have a basis expansion of $m$ (univariate) Gaussian PDFs to model the density of a sample $X$. The means of these PDFs are spaced equidistantly through the domain of $X$ ...
1
vote
0answers
95 views

AIC based model selection, hyperparameter optimization and in-sample prediction

I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following: I have two input variables (A and B), and a single target variable. All variables ...
1
vote
0answers
56 views

Range of Search Space for the hyperparameters of Support Vector Regresssion (SVR)

I need to know "what should be the practical range of c, gamma and epsilon hyper-parameters during grid search optimization in SVR". The range of dependent variable lies between 1 to 300 with mean ...
0
votes
1answer
499 views

RFE number of features with hyperparameter fine tuning within cros-validation

I would like to use cross-validation to select the number of optimal features to select (n_features_to_select) in the recursive feature elimination algorithm (RFE) and the optimal hyperparameter of an ...
0
votes
1answer
265 views

What are two most important hyper parameters in multiplayer Perceptron?

I know there are different hyperparameters for mlpclassifier, however, if I were to choose two most important one, what would they be for a ...
1
vote
2answers
314 views

Using the standard deviation in Cross Validation

I'm running a Grid Search to find the optimal parameters for xgboost via sklearn. I can see that the grid search picks the set of parameters with lowest mean MSE. The problem is that upon inspecting ...
0
votes
1answer
30 views

Overview of sklearn hyper-parameters

Is there an (un)official overview of sklearn hyper-parameters to tune for each model? I find myself often having to google extensively before getting an exhaustive list for any given model. ...
2
votes
0answers
96 views

Can Bayesian Optimization solve this problem?

Suppose ${\bf{x}} = (x_1,\ldots,x_n)$ and $f({\bf{x}})\propto 1_A({\bf{x}}) \prod_{i=1}^n {x_i}^{\alpha_i-1} e^{-\beta_i x_i}$ , i.e. $f$ is proportional to the product of independent gamma ...
5
votes
1answer
2k views

What is a sensible order for parameter tuning in neural networks?

There are so many aspects one could possibly change in a deep neural network that it is generally not feasible to do a grid search over all of them (e.g. activation function, layer type, number of ...
1
vote
0answers
102 views

What metrics to look at when experimenting with neural network hyperparameters?

So with other machine learning techniques I generally only look at the validation error when deciding on certain hyperparameters. I've been reading up on neural networks and it seems that hand tuning ...
10
votes
3answers
360 views

Why don't we just learn the hyper parameters?

I was implementing a pretty popular paper "EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES" and in the paper, it trains an adversarial objective function J''(θ) = αJ(θ) + (1 − α)J'(θ). It treats α ...
1
vote
0answers
27 views

What are the relevant criteria to compare neural nets with different hyper parameter settings?

I want to compare different hyperparameter settings on the same network and the same task to get an impression of what works good and what works better. I am comparing different initializer, ...
-1
votes
1answer
735 views

What are the key hyperparameters to tune in CatBoost?

I've used XGBoost for a long time but I'm new to CatBoost. If I wanted to run a sklearn RandomizedSearchCV, what are CatBoost's hyperparameters worthwhile including for a binary classification problem?...
2
votes
1answer
95 views

Distorted hyperpriors when sampling from the prior only

I am currently testing some multilevel models in pymc3 and found that the hyperpriors get distorted when I run the level only to generate the prior. The hyperpriors I am using are generating ...
0
votes
0answers
11 views

How to fit many datasets at once with a cross-validated pipeline?

Basically, I need to find a pipeline that will not only be the best on one dataset, but to be the best in average across an ensemble of dataset. How would you solve this problem using a cross-...
2
votes
1answer
90 views

Hyperparameter value while computing the test log-likelihood

I have a very basic machine learning question. My likelihood function includes a parameter $\alpha$ which I set to a fixed value and do not learn from the model, which makes it a "hyperparameter". ...
2
votes
0answers
1k views

Gaussian Process Hyperparameter Tuning

I'm planning to use Gaussian Process (GP) to model my case. However, while learning the GP I found out that we have to tuning the hyperparameters to give us the best solution. I have checked several ...
2
votes
0answers
28 views

Change of optimal learning rate with small changes to architecture and data

I am training variants of similar neural networks, which differ slightly in the number of filters or layers. Additionally, the data is sometimes slightly changed using different preprocessing like ...
2
votes
0answers
1k views

Unsupervised anomaly detection - metric for tuning Isolation Forest parameters

I have a project, in which, one of the stages is to find and label anomalous data points, that are likely to be outliers. As a first step, I am using Isolation Forest algorithm, which, after plotting ...
0
votes
1answer
120 views

Hyperparameters tuning on resampled validation set?

Let's suppose we have trained a neural network on some data set, rather than estimating the hyper-parameters using a basic validation error would not it make more sense to: Generate N validation sets ...
0
votes
1answer
587 views

Intuitive explanation of stratified cross validation and nested cross validation

According to the approach outlined here. I should split the dataset into training set and an independent test set using stratified cross-validation. This is the ...
4
votes
1answer
479 views

What is the difference between model selection and hyperparameter tuning?

In the context of supervised learning, in most statistics based texts and papers, one reads about model selection. For example Hastie, Tibshirani and Friedman in ESL define it as: Model Selection: ...
2
votes
0answers
34 views

Hyper-parameters which minimize the variance of transformed multi-variate Guassian variable

Let $k < p$ be positive integers and $g: \mathbb R^k \rightarrow \mathbb R^p$ be a smooth Lipschitz continuous function. Let $y_1,\ldots, y_N \in \mathbb R^p$ and $a = (a_1,\ldots,a_N) \in \mathbb ...
3
votes
1answer
277 views

Neural Network Tuning with Bayesian Optimisation - Number of layers and size of layers

I'd like to use Bayesian Optimization to tune the hyper parameters of a feed-forward neural network. Among these hyper parameters, there is the number of hidden layers in the network, as well as the ...
0
votes
1answer
75 views

best practices for baysian optimization of hyper parameters DNN [closed]

Many people suggest fine-tuning a network using Bayesian optimization ( or grid search or what every other black box optimization method you like ) so I tried it for my self. I am not sure about the ...
2
votes
0answers
100 views

evolution of C and gamma in SVR with the size of training examples

Do you know how C and gamma evolve while the size of training examples (X) rises ? I have found C and gamma for 10% and 20% of my data and I would like to save time. Can I determine C and gamma for my ...
1
vote
1answer
380 views

2018 Update - Different Algorithms for Hyperparameter Optimization

Common question: What are the different options (in common languages like R or Python) available for optimizing hyperparameters? I am primarily interested in implementations in R that can work with ...
1
vote
1answer
428 views

How to perform feature selection and hyperparameter optimization in cross validation?

note: I read a lot of the questions already posted on this topic, but still have some confusion. I want to perform feature selection and model selection for multiple models e.g. Random forest (RF), ...
1
vote
0answers
70 views

Intuition behind low optimal # variables per split (mtry / max_features) during random forest tuning?

I was wondering if anyone might be able to provide some insight regarding how to interpret the results of some random forest hyperparameter tuning I am performing. The training set consists of: 1000 ...
1
vote
2answers
5k views

Confused in selecting the number of hidden layers and neurons in an MLP for a binary classification problem

I'm working on a disease classification dataset which has 25 features including the class attribute. It is a binary classification problem. The dataset has total 300 training instances. I trained a ...
1
vote
0answers
35 views

Is there a test error metric that takes into account variance of the data?

In standard machine learning applications, models are often evaluated based on metrics like RMSE or Deviance. I was wondering if there is a similar metric tailored towards meta-analyses that takes ...
3
votes
1answer
1k views

Reasonable hyperparameter range for Latent Dirichlet Allocation?

What are good ranges for the hyperparameters $\alpha$ and $\beta$ (explained well here) in LDA? I appreciate hyperparameter tuning always depends on the use case, data, content of documents etc., but ...
2
votes
2answers
600 views

ROC curve and parameter selection

I am employing the concept of ROC curve to select one class SVM classifier's parameters as follows: I have a dataset which include a normal class and an abnormal class. I train the One class SVM on ...
30
votes
1answer
6k views

Do we have to tune the number of trees in a random forest?

Software implementations of random forest classifiers have a number of parameters to allow users to fine-tune the algorithm's behavior, including the number of trees in the forest. Is this a parameter ...
0
votes
1answer
147 views

Preprocessing+hyperparameter selection: nested or nested nested cross validation?

I've studied many questions and answers on the theme of nested cross-validation. I understand why we need it and how I can, after that part, find the optimal hyperparameters and any other things I'm ...
0
votes
1answer
289 views

How to tune bandwidth in machine learning kernel model?

Gaussian kernel $k(x,y) = \exp(-\lVert x-y \rVert^2/\sigma^2)$ has a hyperparameter $\sigma$. I know grid search cross validation, but this would require a lot of computation since computational ...
1
vote
0answers
84 views

selecting iterations in random and grid search

I am trying to optimizing a random forest for binary classification. I am tuning cutoff, ntrees, mtyr and maxnodes using MLR library in R. For searching these parameters i have to define a tuning ...
2
votes
0answers
102 views

Information leaks related to test or validation sets

Lately I've been reading about (indirect) information leaks, related to validation and tests sets in the context of hyperparameter tuning. For example, Prakhar Agarwal, in his answer Does my ...
0
votes
0answers
188 views

Random Forests heuristic guideline for the ratio of max_leaf_nodes to sample size

In Random Forests, is there a reference or heuristic guideline for the ratio of max_leaf_nodes to n (sample size)? Assume a small time series binary classification problem modeled using random ...
-1
votes
2answers
93 views

Dataset partitioning issue of cross-validation

Every time the cross-validation is run, the dataset is partitioned into k even groups randomly. That means every time the result of cross-validation can be different, so which one should we take to ...