Questions tagged [hyperparameter]
A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.
654
questions
0
votes
0
answers
20
views
Is the following a deep Gaussian Process and how to proceed with the regression?
I have a problem in 2 dimensions. The signal model is a two-dimensional signal in which each row (indexed with subscript $_b$) is a GP with hyperparameters of the covariance as $\theta_b$:
If the two ...
- 245
1
vote
0
answers
14
views
Assessing Random Search Cross Validation: Tuning in ElasticNet with Large Feature Sets
I'm working on estimating an ElasticNet model for a large dataframe with over 100,000 variables, resulting in a well overidentified scenario. To tune my model, I've set up a grid of hyperparameters (...
- 361
0
votes
0
answers
18
views
Correlation between two Gaussian Processes
I have a space-time series, so it is in 2D. So, the signal model $\mathbf{S}$ is a matrix. If I fix the space, the time series at that point in space is a complex GP:
$$ \mathbf{S}[x, :] \sim \...
- 245
0
votes
1
answer
40
views
Manual selection of parameters and features and bad results by gridsearch
For a very small dataset that I have, when I set the parameters with the help of gridsearch, the test and training results are not acceptable at all and have a huge difference. I have to manually ...
- 43
0
votes
0
answers
35
views
Can I use the Mean Squared Prediction Error to select the prior SD in a CausalImpact model?
I'm using the CausalImpact package (in R), and (as I expect is typical) the findings are very sensitive to the prior being used.
I have an OK understanding, I think, of what the prior is doing in this ...
- 1
0
votes
0
answers
30
views
Avoiding Information Leakage in Backtesting with CPCV-Tuned Hyperparameters
I'm using Combinatorial Purged Cross-Validation to tune hyperparameters for a binary classification model applied in a month-end trading strategy. I have 6 months of data and used CPCV with 15 splits ...
- 1
0
votes
0
answers
15
views
Hyperparameters tuning and Backward feature selection : which one first?
If i have a lot of features and i want to train a light gbm, on the side i want to do the hyperparameter optimisation and on the other side i want to do backward feature selection to reduce the number ...
- 1
0
votes
0
answers
20
views
How to evaluate performance and optimize hyperparameters for clustering algorithms on a dataset with continuous labels?
I'm working on a clustering problem where my dataset's labels are continuous numerical values, not discrete categories. I'm using t-SNE and UMAP to reduce the dimensionality of my dataset's features ...
1
vote
0
answers
117
views
Hyperparameter tuning for small datasets
I have about 10 small imbalanced datasets (some of them only have about 150 samples). I want to try a bunch of balancing techniques on some models. For that, I'm using the repeated stratified cross-...
0
votes
0
answers
15
views
How to develop shared bottom tower serving different tasks
I have two model classes both pyramid architecture.
Let's say first task is predicting user will buy something with architecture [feature_embedding_128, dense_1048, dense_512, dense_128, dense_1]
...
- 314
0
votes
0
answers
11
views
Hyperparameter optimization for CNN
I have a database of defect images on materials, like holes, cuts, and so on. There is not so much information inside the images, I am aware of it.
I am using a CNN, in particular a ResNet50. I know ...
- 151
3
votes
1
answer
62
views
Binary decision boundary requiring 2 hidden layers in neural network with limited neurons
I just started learning about neural networks and was wondering what a neural network with 2 hidden layers is able to express over a neural network with just 1 hidden layer (where number of neurons ...
- 131
0
votes
0
answers
14
views
Determine if the best output of a large number of model fits is due to overtraining
I have a machine learning model (e.g. Gradient Boost Regressor). I use hyperparameter tunning (e.g. random search) in a Cross Validation loop (e.g. 5 folds) in order to recover the optimal parameters ...
- 1
0
votes
0
answers
48
views
SMOTE and Sequential Feature Selection Order
Good morning,
I am doing the following procedure:
Split a Train a Test Dataset
...
0
votes
0
answers
67
views
Advice on Gaussian Process Classifier optimisation best practises? [duplicate]
Hyperparameter Range Determination: My main challenge is in setting effective ranges for hyperparameters such as length_scale, noise_level, and sigma_0. Currently, for length_scale, I've used the ...
2
votes
0
answers
42
views
Should threshold dependent metrics be used as optimization objectives during hyperparameter tuning for classification models?
What metric should we optimize for during hyperparameter tuning?
From what I gather from Frank Harrell's article and other related questions here (Reduce Classification Probability Threshold), the ...
- 153
3
votes
0
answers
69
views
Literature on GP with kernels having no closed form
I have to use GP regression on a complex time series, and the kernel function is not known in closed form. I have found a numerical approximation with the Gauss-Laugerre quadrature. It takes the ...
- 245
0
votes
0
answers
12
views
Why is hyper-parameter tuning decreasing training set performance
Setup: I am using hyperopt for xgboost hyper-parameter tuning. In each trial, the corresponding configuration is evaluated with time-series cross-validation (CV). The CV validation performance the ...
- 33
0
votes
0
answers
27
views
best practices on optimizing feature transforms for a model
I have a regression model that
Transforms some time series features using a different halflife for each feature
Uses the transformed features along with some other features to create a prediction
...
- 101
0
votes
0
answers
13
views
A surrogate function for validation error in order to perform hyper-parameter optimization?
Greed search CV or few other approaches may be computationally expensive in hyper-parameter tuning. Is it possible to come up with a surrogate model or a purposed model for a validation error in order ...
- 81
0
votes
0
answers
28
views
Model fitting with Chinese Restaurant Process
I am trying cluster a trajectory, consisting of (state, action) sequences, by assigning them to the most likely model that generated them using Chinese Restaurant Process. Basically my goal is to ...
- 51
0
votes
1
answer
56
views
Is r2 train= 0.977 and r2 test = 0.941 overfitting for a machine laerning regresssion model?
I'm tuning the hyperparameters for different Machine Learning models with Optuna package. For ExtraTreesRegressor I got r2 train = 0.938, r2 test = 0.922 and RMSE test = 0.907. In this case the ...
- 163
3
votes
0
answers
245
views
Bayesian Optimization: number of iterations as function of search space dimensionality?
I am performing Bayesian Optimization to select a hyperparameter configuration for my supervised learning model. I understand that with each additional hyperparameter that I choose to optimize, the ...
- 1,256
0
votes
0
answers
39
views
Why does my random forest regression model have a 'noisy' fit to training data and a terrible fit to testing data though there is no extrapolation?
I have multiple datasets from an experimental setup and I'm trying to predict one of the system variables using information from the others. I concatenated 80% of the datasets into training data and ...
- 1
0
votes
0
answers
50
views
Getting difficulties which hyperparameters I have to select (5-Fold CV with optuna to find optimal GB regression model's hyperparameters)
I'm currently searching best hyperparameters for a regression model of Gradient Boost, using 5-Fold Cross-Validation and OPTUNA.
However, I'm uncertain about which model's hyperparameters to choose, ...
- 1
0
votes
0
answers
153
views
Is Hyperparameter Tuning for Maximized Recall a Bad Thing?
I have a somewhat theoretical question: I work in an area that requires a number of anomaly detection solutions. When we approach these problems, we cross-validate and for each fold, we oversample ...
- 51
0
votes
0
answers
46
views
Generalized cross-validation (GCV) with nonzero prior mean
I came across this concept of GCV optimization (new to me) for tuning hyperparameters in a model, as an alternative to maximizing the maginal likelihood (MML) of the output, which is what I am used to ...
0
votes
0
answers
37
views
Tuning hyperparameters after multiple runs
I wrote a classifier that uses LGBMClassifier. This is the code:
...
- 101
1
vote
1
answer
124
views
Elbow method for tuning DBSCAN when the minimum number of points per cluster is one
The elbow method calls for setting the number of nearest neighbors (let's call it $k$) to the minimum number of points for a cluster (let's call it $m$), but what do you do when $m\leftarrow1$? Is the ...
- 13
0
votes
0
answers
16
views
Where does cross-validation fit into a model selection workflow with inference?
Say we have some model, $f(x) = \hat{Y}$, such as linear regression, that estimates an output dependent value for some set of input data.
We want to do inference on the coefficients of the model to ...
0
votes
1
answer
883
views
Which hyperparameters should I choose to tune for different ML models? [closed]
I'm applying hyperparameter tuning with techniques such as, randomized search and grid search in python. I could find important hyperparameters and tuned them for Random Forest Regressor algorithm, ...
- 163
0
votes
0
answers
65
views
Tuning Random Forest results in max_features parameter taking a value of 1. Why?
I did a bayesian optimization tuning for parameters of random forest. With 200 iterations, it seems like 70% of the times, very low values (read 1 or 2) of max_features seems to produce better (...
- 71
0
votes
0
answers
77
views
Low CV-RMSE and negative $R^2$ (comparative)
I am trying to predict a numeric variable using XGBoost with optuna for hyperparameter optimization. I defined two objective functions for optuna, one optimized for very small datasets (5 to 17 ...
- 1
5
votes
1
answer
147
views
Doing empirical Bayes with improper prior - marginals that do not exist?
I am considering a Bayesian linear model for which the prior is not proper.
The model is as usual $y = X \theta + w$ where $w \sim N(0, \sigma^2)$, and $\theta, \sigma^2$ are unknown.
The distribution ...
2
votes
0
answers
216
views
How to choose the potential hyperparameters for GridSearchCV on RandomForestClassifier? Will default always be the best?
I'm fairly new to machine learning, and I know similar questions have been asked but I can't find an answer that satisfies my curiosity. I'm working on a Random Forest Classifier model in python, and ...
2
votes
0
answers
114
views
Why would a model combining two pre-trained models not even achieve the performance of the best sub-model?
I have two different CNNs trained on the same dataset. One performs a bit better than the other but I believe each can provide different and useful information.
I use ...
- 172
1
vote
0
answers
45
views
Splitting strategy for performing hyperparameter tuning, algorithm comparison and model validation in one experiment
Let's say that for a supervised machine learning experiment I am using a fixed learning algorithm (e.g. Random Forest), and I want to achieve the following:
Choose optimal hyper-parameters for the ...
- 11
2
votes
0
answers
95
views
A hierarchical model with conjugate hyperprior
I have a modeling problem that I am trying to formulate in a Bayesian manner to do inference.
Basically, I have a prior where the variance is unknown, and we want to treat it as uncertain (though with ...
1
vote
1
answer
192
views
Optimizing a threshold value on a dependent metric using a classifier trained to optimize a threshold-independent metric
Is it a reasonable approach to train a probabilities classifier by optimizing a threshold-independent metric such as AUC, and then using the trained classifier to calibrate the decision threshold ...
- 57
0
votes
1
answer
2k
views
How do scale_pos_weight and sample_weight interact in XGBoost?
Question:
How do scale_pos_weight and sample_weight interact in XGBoost when used at the same time during training? Are they multiplied, added or something else?
Example when added:
Example when ...
- 485
1
vote
0
answers
38
views
Statistical assessment of block size for bootstrapped distribution fitting
I have a set of intensities from unordered independent events (with no date or timestamps), many of which constitute extremes, and I want to generate an extreme value distribution. The only ...
- 111
1
vote
0
answers
114
views
Time series prediction problem formatted correctly for LSTM neural networks?
I am new to machine learning, I am trying to find a way to predict voltage waveforms into the future. I have seen examples that successfully predict sinusoids or continuous voltage data based on ...
- 21
0
votes
1
answer
61
views
How should I understand the noise variance parameter(s) in multifidelity modeling using Emukit
I am learning the multi-fidelity modeling and have a question about Emukit's mixed_noise parameters, or more general, how should we determine the noise when the ...
- 43
5
votes
1
answer
575
views
Understanding Sequential Model-Based Optimization for machine learning
One of the hyperparameter tunning approaches that I came across recently is Sequential Model-Based Optimization (SMBO), which is a very smart approach that uses previous iterations in order to find ...
- 689
1
vote
0
answers
47
views
Hyperparameter Optimization with ASHA algorithm in Sherpa library: strange number of trials
I have developped a DL model to classify images and I am trying to optimize the hyperparameters. Reading the literature, I have found the ASHA algorithm and its implementation in SHERPA, like reported ...
- 151
2
votes
0
answers
52
views
What are effective methods to maximize an unknown noisy function?
I have a function that takes a few hundred parameters and which returns a score I want to optimize for - It's a piece of software attempting to play a game against another player. The parameters ...
- 21
1
vote
1
answer
21
views
Best Way to do Hyperparam Search and Cross-Validation
I'm making experiments to evaluate language models to brazilian portuguese datasets.
So, i've made so each dataset is divided in 10 parts, I want to use cross-validation to determine the model's ...
4
votes
1
answer
2k
views
Is there a risk of overfitting when hyperparameter tuning a model
Is there a risk of overfitting when hyperparameter tuning a model using Optuna (or another hyperparameter tuning method ), with evaluation on a validation set and a large number of trials?
While a ...
- 57
0
votes
1
answer
192
views
Is repeated hyperparameter tuning can lead to overfitting?
I'm performing hyperparameter tuning for a classifier. After I finish, I'm updating the hyperparameter search space and re-tuning the hyperparameters again. I repeat this process a few times. In ...
- 57
3
votes
2
answers
2k
views
Is it a bad practice to learn hyperparameters from the training data set?
I am working on a project where I am evaluating different machine learning models to be used as scoring functions during in-silico docking. It is a regression problem where the 3D structure data of a ...
