Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
-1
votes
2answers
93 views

Dataset partitioning issue of cross-validation

Every time the cross-validation is run, the dataset is partitioned into k even groups randomly. That means every time the result of cross-validation can be different, so which one should we take to ...
7
votes
3answers
9k views

How to tune hyperparameters in a random forest

I don't know how I should tune the hyperparameters: "max depth" and "number of tree" of my model (a random forest). I use Python and I just discovered grid search, but I don't know which range I ...
1
vote
0answers
39 views

Further spliting dataset

I need to to feature selection, parameter tuning and model evaluation. My question is if it is valid to split dataset: to train-test set Then further train set to train-validation1 set Then further ...
0
votes
1answer
184 views

How to formulate an adaptive Levenberg-Marquardt (LM) gradient descent

I am currently programming a sparse bundle adjustment (SBA) for my own problem definition of coordinate calibration. Applying the LM algorithm (slide 10) for parameter optimization, I try to ...
1
vote
0answers
26 views

How to learn optimal unknown constants (hyperparameters) in a function

I have a function that takes 2 known values as input, as well as 8 unknown constants. Lets call it $f(X, k) = y$ where each index $i$ of $X$ is a vector of the 2 input values $k$ is a vector of ...
2
votes
1answer
97 views

Why is it valid to use CV to set parameters and hyperparameters but not seeds?

This is a specification or continuation of my prior question: [Is it 'fair' to set a seed in a random forest regression to yield the highest accuracy? Its valid to use cross-validation to ...
0
votes
0answers
50 views

Score on the test set is higher than on the training set, wrong approach?

I have a relatively small, imbalanced data set (~3k datapoints, 12 classes). I want to tune the parameters of a RandomForestClassifier and eventually test the model....
2
votes
1answer
339 views

Is testing on test set after hyper parameter tuning (with crossvalidation) necessary?

My problem: I have some rule based algorithms and various machine learning algorithms (random forest, boosting, ...) I want to compare for a specific use case. Since I want to optimize the hyper ...
2
votes
1answer
121 views

Higher Test Scores but Higher Variance?

I am tuning hyper-parameters using 5-fold cross-validated grid search for various multiclass classifiers, and I keep running into the same issue that I can't quite wrap my head around. The hyper-...
0
votes
1answer
597 views

Hyperparameter tuning on the whole data set reasonable?

It may be a weird question because I don't fully understand hyperparameter-tuning yet. Currently I'm using gridSearchCV of ...
1
vote
0answers
31 views

Is there any method to update hyperparameters following the addition of data in a dataset?

I have a small dataset that I'm currently using python GridSearchCV to search hyperparameters. From time to time I increase this dataset and have to train again all of my regressors(I'm using sklearn ...
1
vote
1answer
555 views

Bayesian optimization neural network

When tuning my neural net with Bayesian optimization I want to determine the optimal number of hidden layers and the corresponding number of neurons in each hidden layer. The problem is that with an ...
4
votes
1answer
2k views

Random forest parameters

I'm trying to make decisions regarding Random forest parameters for classification. My dataset contains 26 features and 6300 instances. How can I decide the values of (the number of trees, number of ...
0
votes
0answers
43 views

When multiple c,g hyper-parameter values yield the same accuracy, how can I choose which c,g is better?

[ Not a duplicate: First of all, my data set is balanced, so accuracy is a reasonable measure of performance. I.e. any value over 50% means that the result is better than random. ] [ Not a duplicate:...
3
votes
1answer
637 views

Hyper Parameter Tuning for Unbalanced Data

My question regards the hyperparameter tuning for ML algorithms. It has more to do with the theoretical aspects than the actual coding part. Suppose there is dataset which is extremely imbalanced as ...
4
votes
1answer
2k views

Why a large gamma in the RBF kernel of SVM leads to a wiggly decision boundary and causes over-fitting?

The hyperparameter $\gamma$ of the Gaussian/rbf kernel controls the tradeoff between error due to bias and variance in your model. If you have a very large value of gamma, then even if your two ...
1
vote
1answer
472 views

Is epoch optimization in CV with constant mini-batch size even possible?

Assume that you found the optimal hyperparameters of a neural network (e.g. a multi layer feed forward NN) with k-fold cross validation in a grid search. Lets assume you have varied the number of ...
3
votes
1answer
58 views

How to choose the model hyperparameter after cross-validation when the model fit indices are really similar?

I cross-validated a model using the classification accuracy using leave-one-out-cross-validation (proportion of correctly classified cases). Below is a matrix of accuracies typical to what I see. The ...
0
votes
1answer
44 views

Finding bandwith with cross validation method

I am trying out some methods for finding the optimal bandwith for a kernel density estimation in R. Now I stumbled across a post on R-bloggers If I compute the Silvermann's rule of thumb bandwith for ...
1
vote
2answers
50 views

Is bounds of parameter hyperparameter?

Suppose I set [Smin, Smax] to be the limits of slope of a regression, are they hyperparameter? If not, what are they called?
1
vote
0answers
150 views

Adequacy of cross-validation in a image classification problem

The problem I am facing is remote sensing image classification. My initial idea was to perform feature selection, classifier selection and hyperparameter tuning inside cross-validation inner loop, ...
1
vote
1answer
522 views

Multilayer perceptron for binary classification: threshold learning

In a basic contest, the MLP loss function (cross entropy) uses as value for the label ŷ: +1 if the net output is greater or equal to 0.5 -1 otherwise Where the net output is a value in [0,1] ...
2
votes
0answers
44 views

Choosing an ideal minibatch size for a large sample [duplicate]

I know that bigger batch size gives more accurate results, but I'm not sure which batch size is ideal given the following cases: Training on 65000 examples and validating on 13000 examples Training ...
2
votes
1answer
199 views

Tuning of Hyperparameter [duplicate]

Are there any advanced packages that allows automated tuning of hyperparameters for neural network and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, ...
0
votes
0answers
143 views

How to choose hyperparameters for comparing different algorithms?

Assume the main task is to classify the data $X$ and it is done based on an optimization setting: $$\min_W f(X,W)+\lambda g(X,W)$$ in which $W$ is the classifier parameter matrix. Now consider that ...
1
vote
0answers
111 views

How can I compare two hyperparameter optimization algorithms?

I am comparing two hyperparameter optimization algorithms: "Bayesian optimization" (henceforth called B) and "Quasirandom sampling" (Q). I use them to choose parameters for training a complicated ...
2
votes
0answers
113 views

Does bias during hyperparameter tuning matter?

Different validation methods have different bias and variance. For instance, k-fold cross validation with high enough k (e.g. 10) has low bias, but high variance, whereas the bootstrap has lower ...
1
vote
1answer
38 views

Choosing parameter ranges for classification problems

How do I choose good values of my param_grid in GridSearchCV()? I see many different methods, but wonder if there is some theory ...
4
votes
4answers
550 views

Seed in a grid search

When conducting a grid search over a range of parameters of a predictive model which is itself subject to randomness (such as a random forest with bagged features), should you set a seed for the ...
0
votes
1answer
230 views

Neural network hyperparameters for classification with noisy features

I'm working on a multi-class classification problem using a neural network, where my features are rather noisy, with some very similar inputs may belong to different classes for different training ...
0
votes
1answer
105 views

Statistical significance of changing a hyperparameter

Suppose we have a machine learning model (e.g. an SVM) and we've run a search over multiple hyperparameters (e.g. kernel, C) to gauge performance. How would we gauge whether changing one parameter (...
1
vote
0answers
660 views

Random forest tuning mtry - tuneRF versus caret

Multiple posts (below) show that tuneRF and caret (or manually tuning) produce quite different recommendations for mtry. Why are these different? Which approach is more reliable for controlling ...
0
votes
1answer
125 views

Tuning parameters using inflection points

Assume a Machine Learning model has one real hyper-parameter. In many cases the hyper-parameter is chosen to be value at which the curve of a validation measure (MSE,etc) with respect the hyper-...
4
votes
1answer
114 views

Bayesian optimizations of hyper-parameters - magical curve based on two data points?

I am trying to understand how Bayesian optimizations of hyper-parameters works in practice. Question 1: How shape of this curve (solid blue line in the middle) is calculated given Gaussian kernel. ...
1
vote
2answers
593 views

Is there a formula for a recommended batch size depending on the size of the training dataset?

I'm still training my neural network for gender/age classification, and I'm currently experimenting with batch sizes along with everything else. As I've gathered, too small a batch size will lower ...
0
votes
1answer
72 views

Is it possible to replace Grid Search with ternary search?

So me and my friend decided to learn about tuning hyper parameter of xgboost using Grid Search CV for a regression problem. We decided to tune parameter such as maximal depth, column sampling and so ...
3
votes
1answer
567 views

Multilayer perceptron: Hyperparameters vs Parameters and Cross validation (nested or not)

I'm a bit confused about the k-fold cross validation (inner and external) done for the model performance evaluation. I've read that when you are trying to validate your model, you need to do it ...
2
votes
1answer
392 views

Problem with building Optimizer using Random Forest as Surrogate Model

I am trying to use Random Forest as Surrogate Model instead of Gaussian Process for my Bayesian Optimizer Framework and already studied the concept of it through SMAC and mlrMBO Papers. I use the ...
1
vote
1answer
338 views

Parameter Tuning for Random Forest Text Classifier

I train a binary random forest classifier on skikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold crossvalidation on the training data. Is there any ...
2
votes
1answer
106 views

Reshuffle before k-fold cross-validation split when doing grid search

I want to find the best hyperparameters of a neural network by a grid search. Let's say I have: activation (ReLU or sigmoid) batch size (32, 64, 128 or 256) so my space of hyper-parameters has 8 ...
1
vote
0answers
49 views

Cross validation with hyper-parameters re-runs

So, when performing CV we target two goals: Build the best possible model Estimate model performance I have read about nested CV and also about Tibshirani-Tibshirani method. In either case, is clear ...
4
votes
1answer
431 views

General procedures for combined feature selection, model tuning, and model selection?

What is the general procedure for a combined task of model tuning (i.e., hyperparameter selection), feature selection and model selection? I know some basic principles for each task, but when ...
4
votes
1answer
3k views

Batch normalization: How to update gamma and beta during backpropagation training step?

The backpropagation step of batch normalization computes the derivative of gamma (let's call it dgamma) and the derivative of <...
4
votes
1answer
411 views

2 hidden layers are more powerful than 1

When searching for information on choosing the number of hidden layers in a neural network, I have come across the following table mutiple times, including in this answer: | Number of Hidden Layers ...
0
votes
1answer
19 views

Clustering with parameterized metrics [closed]

Let's assume that we have some data $x_1,x_2,\dots x_n$. Moreover, we have a parameterized distance metric $\delta(x,x';\theta)$ for some $\theta\in \Theta$. That's it, I have no labels, no clue ...
4
votes
1answer
720 views

multivariate gaussian processing | bayesian optimization with multiple features (parameter optimization)

Best My goal is very simple, I would like to optimize 3 parameters in my algorithm. And I would like to do this via, multivariate gaussian processes. But the problem which I've is, I do understand ...
2
votes
1answer
128 views

Adding training and validation set to increase accuracy after finding optimal parameters

General rule of thumb in splitting data in Machine Learning is in 3 parts training set, validation set and ...
3
votes
1answer
465 views

Training Neural Net with examples it misclassified

So I have a net which is working pretty well(93%+ on the validation set which is the state of the art[https://yoniker.github.io/]) on some problem. I want to squeeze even more performance out of it, ...
0
votes
1answer
143 views

What is the right way for hyper parameter tuning when do partial fit on chunk data?

I have a huge data file, so i can not read it in memory. I read it chunk by chunk, then fit it by using partial_fit( like as : SGDClassifier). So how can i do hyper parameter tuning for my model ? I ...
2
votes
1answer
1k views

Overfitting during model selection - AutoML vs Grid search

I've recently picked up attention for AutoML algorithms; Meta-algorithms that intelligently search the space of machine learning models to find the "pipeline" (preprocessing/feature selection method/...