# Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

385 questions
Filter by
Sorted by
Tagged with
93 views

### Dataset partitioning issue of cross-validation

Every time the cross-validation is run, the dataset is partitioned into k even groups randomly. That means every time the result of cross-validation can be different, so which one should we take to ...
9k views

### How to tune hyperparameters in a random forest

I don't know how I should tune the hyperparameters: "max depth" and "number of tree" of my model (a random forest). I use Python and I just discovered grid search, but I don't know which range I ...
39 views

### Further spliting dataset

I need to to feature selection, parameter tuning and model evaluation. My question is if it is valid to split dataset: to train-test set Then further train set to train-validation1 set Then further ...
184 views

### How to formulate an adaptive Levenberg-Marquardt (LM) gradient descent

I am currently programming a sparse bundle adjustment (SBA) for my own problem definition of coordinate calibration. Applying the LM algorithm (slide 10) for parameter optimization, I try to ...
26 views

### How to learn optimal unknown constants (hyperparameters) in a function

I have a function that takes 2 known values as input, as well as 8 unknown constants. Lets call it $f(X, k) = y$ where each index $i$ of $X$ is a vector of the 2 input values $k$ is a vector of ...
97 views

### Why is it valid to use CV to set parameters and hyperparameters but not seeds?

This is a specification or continuation of my prior question: [Is it 'fair' to set a seed in a random forest regression to yield the highest accuracy? Its valid to use cross-validation to ...
50 views

### Score on the test set is higher than on the training set, wrong approach?

I have a relatively small, imbalanced data set (~3k datapoints, 12 classes). I want to tune the parameters of a RandomForestClassifier and eventually test the model....
339 views

### Is testing on test set after hyper parameter tuning (with crossvalidation) necessary?

My problem: I have some rule based algorithms and various machine learning algorithms (random forest, boosting, ...) I want to compare for a specific use case. Since I want to optimize the hyper ...
121 views

### Higher Test Scores but Higher Variance?

I am tuning hyper-parameters using 5-fold cross-validated grid search for various multiclass classifiers, and I keep running into the same issue that I can't quite wrap my head around. The hyper-...
597 views

### Hyperparameter tuning on the whole data set reasonable?

It may be a weird question because I don't fully understand hyperparameter-tuning yet. Currently I'm using gridSearchCV of ...
31 views

### Is there any method to update hyperparameters following the addition of data in a dataset?

I have a small dataset that I'm currently using python GridSearchCV to search hyperparameters. From time to time I increase this dataset and have to train again all of my regressors(I'm using sklearn ...
555 views

### Bayesian optimization neural network

When tuning my neural net with Bayesian optimization I want to determine the optimal number of hidden layers and the corresponding number of neurons in each hidden layer. The problem is that with an ...
2k views

### Random forest parameters

I'm trying to make decisions regarding Random forest parameters for classification. My dataset contains 26 features and 6300 instances. How can I decide the values of (the number of trees, number of ...
43 views

### When multiple c,g hyper-parameter values yield the same accuracy, how can I choose which c,g is better?

[ Not a duplicate: First of all, my data set is balanced, so accuracy is a reasonable measure of performance. I.e. any value over 50% means that the result is better than random. ] [ Not a duplicate:...
637 views

### Hyper Parameter Tuning for Unbalanced Data

My question regards the hyperparameter tuning for ML algorithms. It has more to do with the theoretical aspects than the actual coding part. Suppose there is dataset which is extremely imbalanced as ...
2k views

### Why a large gamma in the RBF kernel of SVM leads to a wiggly decision boundary and causes over-fitting?

The hyperparameter $\gamma$ of the Gaussian/rbf kernel controls the tradeoff between error due to bias and variance in your model. If you have a very large value of gamma, then even if your two ...
472 views

### Is epoch optimization in CV with constant mini-batch size even possible?

Assume that you found the optimal hyperparameters of a neural network (e.g. a multi layer feed forward NN) with k-fold cross validation in a grid search. Lets assume you have varied the number of ...
58 views

### How to choose the model hyperparameter after cross-validation when the model fit indices are really similar?

I cross-validated a model using the classification accuracy using leave-one-out-cross-validation (proportion of correctly classified cases). Below is a matrix of accuracies typical to what I see. The ...
44 views

### Finding bandwith with cross validation method

I am trying out some methods for finding the optimal bandwith for a kernel density estimation in R. Now I stumbled across a post on R-bloggers If I compute the Silvermann's rule of thumb bandwith for ...
50 views

### Is bounds of parameter hyperparameter?

Suppose I set [Smin, Smax] to be the limits of slope of a regression, are they hyperparameter? If not, what are they called?
150 views

### Adequacy of cross-validation in a image classification problem

The problem I am facing is remote sensing image classification. My initial idea was to perform feature selection, classifier selection and hyperparameter tuning inside cross-validation inner loop, ...
522 views

### Multilayer perceptron for binary classification: threshold learning

In a basic contest, the MLP loss function (cross entropy) uses as value for the label ŷ: +1 if the net output is greater or equal to 0.5 -1 otherwise Where the net output is a value in [0,1] ...
44 views

### Choosing an ideal minibatch size for a large sample [duplicate]

I know that bigger batch size gives more accurate results, but I'm not sure which batch size is ideal given the following cases: Training on 65000 examples and validating on 13000 examples Training ...
199 views

### Tuning of Hyperparameter [duplicate]

Are there any advanced packages that allows automated tuning of hyperparameters for neural network and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, ...
143 views

### How to choose hyperparameters for comparing different algorithms?

Assume the main task is to classify the data $X$ and it is done based on an optimization setting: $$\min_W f(X,W)+\lambda g(X,W)$$ in which $W$ is the classifier parameter matrix. Now consider that ...
111 views

### How can I compare two hyperparameter optimization algorithms?

I am comparing two hyperparameter optimization algorithms: "Bayesian optimization" (henceforth called B) and "Quasirandom sampling" (Q). I use them to choose parameters for training a complicated ...
113 views

### Does bias during hyperparameter tuning matter?

Different validation methods have different bias and variance. For instance, k-fold cross validation with high enough k (e.g. 10) has low bias, but high variance, whereas the bootstrap has lower ...
38 views

### Choosing parameter ranges for classification problems

How do I choose good values of my param_grid in GridSearchCV()? I see many different methods, but wonder if there is some theory ...
550 views

### Seed in a grid search

When conducting a grid search over a range of parameters of a predictive model which is itself subject to randomness (such as a random forest with bagged features), should you set a seed for the ...
230 views

### Neural network hyperparameters for classification with noisy features

I'm working on a multi-class classification problem using a neural network, where my features are rather noisy, with some very similar inputs may belong to different classes for different training ...
105 views

### Statistical significance of changing a hyperparameter

Suppose we have a machine learning model (e.g. an SVM) and we've run a search over multiple hyperparameters (e.g. kernel, C) to gauge performance. How would we gauge whether changing one parameter (...
660 views

### Random forest tuning mtry - tuneRF versus caret

Multiple posts (below) show that tuneRF and caret (or manually tuning) produce quite different recommendations for mtry. Why are these different? Which approach is more reliable for controlling ...
125 views

### Tuning parameters using inflection points

Assume a Machine Learning model has one real hyper-parameter. In many cases the hyper-parameter is chosen to be value at which the curve of a validation measure (MSE,etc) with respect the hyper-...
114 views

### Bayesian optimizations of hyper-parameters - magical curve based on two data points?

I am trying to understand how Bayesian optimizations of hyper-parameters works in practice. Question 1: How shape of this curve (solid blue line in the middle) is calculated given Gaussian kernel. ...
593 views

### Is there a formula for a recommended batch size depending on the size of the training dataset?

I'm still training my neural network for gender/age classification, and I'm currently experimenting with batch sizes along with everything else. As I've gathered, too small a batch size will lower ...
72 views

### Is it possible to replace Grid Search with ternary search?

So me and my friend decided to learn about tuning hyper parameter of xgboost using Grid Search CV for a regression problem. We decided to tune parameter such as maximal depth, column sampling and so ...
567 views

### Multilayer perceptron: Hyperparameters vs Parameters and Cross validation (nested or not)

I'm a bit confused about the k-fold cross validation (inner and external) done for the model performance evaluation. I've read that when you are trying to validate your model, you need to do it ...
392 views

### Problem with building Optimizer using Random Forest as Surrogate Model

I am trying to use Random Forest as Surrogate Model instead of Gaussian Process for my Bayesian Optimizer Framework and already studied the concept of it through SMAC and mlrMBO Papers. I use the ...
338 views

### Parameter Tuning for Random Forest Text Classifier

I train a binary random forest classifier on skikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold crossvalidation on the training data. Is there any ...
106 views

### Reshuffle before k-fold cross-validation split when doing grid search

I want to find the best hyperparameters of a neural network by a grid search. Let's say I have: activation (ReLU or sigmoid) batch size (32, 64, 128 or 256) so my space of hyper-parameters has 8 ...
49 views

### Cross validation with hyper-parameters re-runs

So, when performing CV we target two goals: Build the best possible model Estimate model performance I have read about nested CV and also about Tibshirani-Tibshirani method. In either case, is clear ...
431 views

### General procedures for combined feature selection, model tuning, and model selection?

What is the general procedure for a combined task of model tuning (i.e., hyperparameter selection), feature selection and model selection? I know some basic principles for each task, but when ...
3k views

### Batch normalization: How to update gamma and beta during backpropagation training step?

The backpropagation step of batch normalization computes the derivative of gamma (let's call it dgamma) and the derivative of <...
411 views

### 2 hidden layers are more powerful than 1

When searching for information on choosing the number of hidden layers in a neural network, I have come across the following table mutiple times, including in this answer: | Number of Hidden Layers ...
19 views

### Clustering with parameterized metrics [closed]

Let's assume that we have some data $x_1,x_2,\dots x_n$. Moreover, we have a parameterized distance metric $\delta(x,x';\theta)$ for some $\theta\in \Theta$. That's it, I have no labels, no clue ...
720 views

### multivariate gaussian processing | bayesian optimization with multiple features (parameter optimization)

Best My goal is very simple, I would like to optimize 3 parameters in my algorithm. And I would like to do this via, multivariate gaussian processes. But the problem which I've is, I do understand ...
128 views

### Adding training and validation set to increase accuracy after finding optimal parameters

General rule of thumb in splitting data in Machine Learning is in 3 parts training set, validation set and ...
465 views

### Training Neural Net with examples it misclassified

So I have a net which is working pretty well(93%+ on the validation set which is the state of the art[https://yoniker.github.io/]) on some problem. I want to squeeze even more performance out of it, ...