Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
1
vote
0answers
12 views

How to interpret the Precision and Recall curve in-sample vs out-of-sample

I have an imbalanced binary classification problem. After all the preprocessing (scaling, feature selection), I am going through an hyperparameter optmimisation using GridSearchCV to find the best ...
0
votes
0answers
10 views

XGB model (or any other ML model) objective function vs scoring metrics and log transformations of the target label

I spent some time googling and could not find a proper answer for my question, maybe I have some terms confused but here is the question: When fitting a XGB model (or any ML model like Keras ANN or ...
0
votes
0answers
20 views

After Deep Learning Hyperparam tuning, what adjustments should be made when dataset size is scaled up?

I'm dealing with a fully connected NN, and I'm wondering if there are any rules of thumb for adjusting hyperparameters for changes to dataset size. For example, if I increase number of obs by 20%, ...
2
votes
1answer
21 views

How to use Hyperopt or perform hyperparamter tuning on a GAN?

I am creating a Generative Adversarial Network, and I want to run Hyperopt on the GAN. However, I am confused about how to do this in Python in Keras because the GAN is essentially 2 models, with the ...
0
votes
0answers
9 views

How does RandAugment theoretically improves generalization/robustness of the model?

Recently I have been experimenting with autoaugmentation methods. I have started from RandAugment ( https://arxiv.org/pdf/1909.13719.pdf). In the paper they also show that magnitude of transformation ...
0
votes
0answers
11 views

How to tune hyperparameter with imbalanced data

I am doing an hyperparameter tuning through GridSearchCV for a binary classification. ...
0
votes
0answers
8 views

Learning Rate Update in fasttext

Does anyone know what the learning rate update hyperparameter in fasttext does? The fasttext page says this ...
2
votes
0answers
21 views

High dimensional hyperparameter tune

Many already known optimization techniques rely on past data (Bayesian optimization for instance) and perform really well for a bunch of hyperparameters. Is there, however, a good tuner/tuning method ...
3
votes
1answer
70 views

How to decide whether to optimize model hyperparameters on a development set or by cross-validation?

Suppose I have a machine learning model. I've seen 2 ways to optimize its hyperparameters: (i) Cross-Validation and (ii) Development Set. Are they parallel to each other? If so, what are the ...
0
votes
0answers
10 views

Multimetric hyperparameter selection

I am solving a supervised learning classification problem, and since I'm using xgboost I've done an optimal hyperparameter search by randomly choosing hyperparameters in a K-fold CV setting. My ...
0
votes
0answers
14 views

How can I determine a sufficient number of inducing variables for stochastic variational Gaussian process regression?

I'm playing around with stochastic variational Gaussian process regression. I'm following one of the GPflow tutorials. One issue of practical interest to me is determining the number of inducing ...
0
votes
0answers
8 views

Is there a principled method for preventing overfitting of a model to the validation set?

Overfitting almost always implicitly refers to overfitting onto the training set, which could occur, for instance, when a model is trained for too long, where we see a dip in the validation accuracy ...
0
votes
0answers
32 views

Bayesian hyperparameter optimization, is it worth it?

On Goodfellow-et-al-2016,Deep Learning book the following sentence can be found: "Currently, we cannot unambiguously recommend Bayesian hyperparameter optimization as an established tool for ...
0
votes
0answers
46 views

Hierarchical softmax vs softmax in hyper-parameter search

I am training an NLP model using fasttext where fasttext allows you to use either hierarchical softmax or softmax. It is my understanding that hierarchichal softmax is orders of magnitude faster ...
2
votes
1answer
11 views

Examples of the performance of a machine learning model to be more sensitive to a small subset of hyperparameters than others?

Someone told me that the performance of a machine learning model tends to be more sensitive to changes in a small subset of hyperparameters than other. Is this true? Can someone give me an example of ...
0
votes
0answers
19 views

Advantage of implementing grid search over random search algorithm

When people compare grid search with random search for hyperparameter tuning, they tend to talk about the advantage of random search algorithm over the grid search algorithm. However, I am wondering ...
0
votes
0answers
12 views

Tuning with a lower early stopping to find relationships between hyperparameters

I have a neural network with the common hyperparameters to optimize. For example, the regularization term (lambda) and the number of hidden units, as well as the number of early stopping rounds. In ...
0
votes
0answers
11 views

How do you account for batch size as a hyper-parameter when doing cross validation on a set

Suppose I am training a simple artificial neural network model on a data set of size 1000 and I want to find good hyper-parameters for the model, including a mini-batch parameter, i.e batch size of 8, ...
1
vote
1answer
24 views

Is the validation set in CV considered “out-of-sample”?

Let's say I want to tune hyperparameter lambda in a Lasso regressor. I have a train and validation set for hyperparameter tuning using 10-fold CV. Can the MSE calculated on the 10% validation fold ...
0
votes
0answers
8 views

relationship between complexity of hyperparameter tuning algorithm and the performance of a machine learning model?

Is there any paper/references that talks about the relationship between complexity of hyperparameter tuning algorithm and the overall performance of a machine learning model (e.g. the performance of a ...
0
votes
0answers
14 views

What is the difference between hyperparameter tuning and neural architecture search?

I am not very clear about the difference between hyperparameter tuning and neural architecture search (NAS). Recently year there has been a lot of these so-called NAS algorithms, such as DARTS. ...
0
votes
0answers
41 views

Hyperparameter Tuning with Keras / Tensorflow for multivariate time series regression

The model I am training a dense feed-forward NN using the Keras API on Tensorflow. Each sample of the training set defines $\mathbf{X_t}$ and $\mathbf{Y_t}$ of an observed time period $t\in T$. Input ...
1
vote
1answer
41 views

Is hyperparameter optimization needed?

I'm trying to compare a model (random forest) trained on two sets of features. My goal is to compare the performance of the model when I use one set of features vs the other. I only have about 120 ...
0
votes
0answers
16 views

Hyperparameter optimization on already good values

Which methods are useful and reliable for hyperparameter optimization when the search space is big, the objective function is noisy (some kind of black-box), non-convex, and the model already has very ...
0
votes
0answers
11 views

Does random search incur less computational cost than the grid search? [duplicate]

I was reading this note that compares grid search with random search algorithm for hyperparameter tuning: https://www.cs.cornell.edu/courses/cs4787/2020sp/lectures/Lecture14.pdf In the note the author ...
0
votes
0answers
18 views

What is the computational cost of random search algorithm? [duplicate]

I know that, for grid search algorithm, the computational cost of the grid search algorithm would be $O(v^p)$ optimizing $p$ parameters each of which can take on $v$ values, then the computational ...
1
vote
0answers
23 views

How does grid search suffers from the curse of dimensionality while the random search doesn't? [duplicate]

Someone told me grid search suffers greatly from the curse of dimensionality while the random search algorithm doesn't. Does this mean that it is computationally more costly to tune the parameters ...
2
votes
1answer
52 views

Grid search vs. random search for hyperparameter tuning

I am interested in comparing the computational cost of grid search vs. random search algorithm for hyperparameter tuning. Is random search computationally less expensive then grid search? Is there an ...
0
votes
1answer
27 views

Comparing classifier performance when using slightly different datasets

Let's say I'm trying to predict whether tomorrow's temperature is higher than today's based on historical data (2 time series A and B). I've chosen XGBoost for the task. For model selection (...
0
votes
1answer
60 views

Optimize number of hidden layers and neurons with RandomizedSearchCV (scikit-learn) -> No unnecessary trainings?

I want to optimize the number of hidden layers and the number of units in each hidden layer. For this I used RandomizedSearchCV from sklearn in this way: ...
1
vote
0answers
10 views

Default Model Parameters Outperform Optimized Parameters on Test Set - Possible Overfitting?

I have performed hyperparameter tuning with cross validation for a classification algorithm on my training data and have tested model performance by making predictions on my holdout set. I have ...
0
votes
0answers
13 views

Explore hyperparameters for one model or try different models first?

When first approaching a regression problem, would you rather try a model you know well and explore its hyper parameters via grid-search + cross-validation, or would you instead explore different ...
0
votes
1answer
79 views

Random search dimensionality dependence

Random search has a probability of 95% of finding a combination of parameters within the 5% optima with only 60 iterations. Also compared to other methods it doesn't bog down in local optima. This ...
0
votes
0answers
8 views

What can I do against oscillating validation accuracy in CNN?

I'm training a convolutional neural network and am seeing the issue that my validation accuracy is oscillating a lot. I'm seeing some oscillating on the training accuracy too but by far not as much. ...
1
vote
0answers
19 views

Bayesian optimization experiment to confirm learning rate schedules in DNNs

Common learning rate schedules usually decrease the learning based on some criteria or some predefined schedule which intuitively makes sense. Has it been confirmed with bayesian hyper parameter ...
0
votes
0answers
47 views

XGBoost Mean Average Precision eval_metric for Classification

I am testing XGBoostClassifier for a binary classification problem. I have tried a few base models, done some simple parameter tuning, and performed feature selection using sklearn's ...
2
votes
0answers
20 views

Uplift modeling for Train, validation, test data sets

I am wondering when I should tune hyper parameter when we build uplift model. In a normal machine learning context, data will be split into train, validation, test. And, we train the model with train ...
0
votes
1answer
46 views

Range of values for hyperparameters of the KNN

Algorithm : Classification by k-nearest neighbors with Euclidean distance (neighbors.KNeighborsClassifier). Determine the important hyperparameters (2 maximum) that can significantly influence ...
0
votes
0answers
12 views

What is the way to compare SVM output?

How can I rigorously compare the results from SVM? I have a feature matrix that contains ~1000 features and the label is either 1 or 0. The features can be grouped into 4 categories, let's say they ...
0
votes
1answer
67 views

Questions regarding implementation of cross-validation

I have a paper coming up and I would like to clear some questions regarding cross-validation, because I could not find this information explicitly stated anywhere in literature or it differs. I ...
0
votes
1answer
40 views

Bayesian optimization with xgb.cv and xgb.XGBClassifier - Mismatch between AUC scores

I'm doing bayesian hyperparameter optimization with bayes_opt and maximizing the AUC. I'm noticing a big discrepancy between the cross-validation scores that I ...
0
votes
1answer
15 views

Hyperparameter tuning vs weight tweaking in Cross-Validation: should I consider 2 different validation sets?

Let's say I have 1000 Samples and want to build an ANN. Then I split my dataset into train set (800) and test set (200). After that, I do the following Cross-validate my train set with different ...
1
vote
0answers
82 views

Should I perform nested CV with Grid Search to make my ensemble model robust?

I'm doing classification of 8 types of hand gestures with stacking models. For that I initially split the data into training and test sets. Then I used GridSerachCV ...
0
votes
0answers
52 views

Treating “probability thresholds” (classification problems) as a hyperparameter

I found this link over here : https://topepo.github.io/caret/using-your-own-model-in-train.html (section 13.8) My understanding is that the probability threshold over here is essentially being ...
1
vote
0answers
34 views

When to stop the chain of priors in Bayesian hierarchical models?

From Wkipedia's article on hyperprior: In Bayesian statistics, a hyperprior is a prior distribution on a hyperparameter, that is, on a parameter of a prior distribution. There will be some ...
1
vote
0answers
23 views

How can one use Grid Search without overfitting the model?

I checked several questions, like Overfitting during model selection - AutoML vs Grid search and Hyperparameter tuning using grid search/randomised search, but I don't think any of them answer my ...
1
vote
1answer
20 views

What to do after knowing the model is overfitted?

So I was trying to run a model using scikit-learn. In order to tune the hyperparameters, I used RandomizedSearchCV, just like this: ...
1
vote
1answer
39 views

I have a neural network, I have a validation set, now how do I start tuning?

I don't really understand how people begin/start tuning their network and there seems to be a lot of conflicting information. One online answer I saw said: Start with the learning rate, tune that, ...
6
votes
1answer
74 views

Is there a hard distinction between hyperparameter vs parameter in machine learning?

I was watching Andrew Ng's lecture on the difference between parameter vs hyperparameter, https://www.youtube.com/watch?v=VTE2KlfoO3Q&ab_channel=Deeplearning.ai, and a question came to me. Is ...
3
votes
1answer
66 views

Repeated Nested Cross validation

I'm aware that nested cross-validation is used for hyperparameter tuning and model selection and that repeated k-fold cross-validation is used to improve the estimated performance of the model. My ...

1
2 3 4 5
10