Questions tagged [hyperparameter]

A parameter that is not strictly for the statistical model (or data generating process), but a parameter for the statistical method. It could be a parameter for: a family of prior distributions, smoothing, a penalty in regularization methods, or an optimization algorithm.

Filter by
Sorted by
Tagged with
0
votes
0answers
40 views

What are the derivatives of Squared Exponential kernel function w.r.t. characteristic length scale (Gauss Process)

I'm writing a matlab code to implement Gaussian process. In the book: Gaussian Process for machine learning by Carl Edward Rasmussen and Christopher K. I. Williams, the authors define the squared ...
0
votes
0answers
15 views

Best score in SVM

i am new to machine learning and i took the house price dataset from kaggle.com to learn and understand SVM. for regression the best score would be 0.0 and for classification the best score ...
0
votes
1answer
19 views

Are the vectorization settings considered hyperparameters in ML?

Short definition of HP: "In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. Hyperparameter optimization or tuning is the problem of ...
1
vote
1answer
95 views

Picking lambda for LASSO

Preface: I am aware of this post: Why is lambda "within one standard error from the minimum" is a recommended value for lambda in an elastic net regression? (It is generally recommended to ...
2
votes
1answer
22 views

Is the validation data set used for building the testing model?

Let's assume I split my data into 70% training data, 20% validation data, and 10% testing data. For each hyperparameter I am building a model using the training data and determine the best ...
0
votes
0answers
27 views

Methods to optimize hyperparameters of an ARD kernel for Gaussian Process Regression

I am using Gaussian Process regression to build a model from my feature set, which consists of 40 parameters and ~250 samples in my training set. I've chosen an RBF kernel with ARD (different length ...
2
votes
1answer
22 views

Scale selection for the Hyperparameters

During the scale selection for hyperparameters I have a question generally if we have to select a random number in the interval $[2,5]$ It is fairly straight forward that we select either $2,3,4,5$ ...
2
votes
1answer
32 views

Nested Cross Validation: How to do the whole Shebang (Algorithmic Selection, Model Selection, Parameter Tuning, Preprocessing) [closed]

First post! If you don't want to read the background you can skip to the Problem heading below. Background Hello everyone, I'm a Physics student doing physics education research. My professor wants ...
0
votes
0answers
14 views

Query regarding Hyper parameters tuning in deeplearning

I have a question "Using an appropriate scale to pick hyperparameters", under the section "Hyperparameter tuning". The reason mentioned for choosing a log scale over a linear scale for the hyper ...
1
vote
2answers
46 views

Randomly sampling parameters for model selection

Suppose I'm fitting a complicated (e.g. neural network) model's parameters $\theta$ to some data $D$, and I'm trying to tune hyperparameters (e.g. number of layers, size of layers) $\eta$. Normally I ...
1
vote
2answers
53 views

Benefits of random search over other optimization methods in Neural Network hyperparameter tuning

When reading about methods of hyperparameter tuning of neural networks, I have mainly come across grid search and random search in textbooks and articles online. I was wondering why other optimization ...
0
votes
1answer
35 views

Perceptual Loss Layers Selection

I understand that in order to improve your generative model performance it is quite useful to compare your output and the target in the feature space, as stated in the paper Perceptual Losses for Real-...
2
votes
2answers
315 views

How to choose delta parameter in Huber Loss function?

In Huber loss function, there is a hyperparameter (delta) to switch two error function. Currently, I am setting that value manually. But, I cannot decide which values are the best. So, how to choose ...
2
votes
2answers
46 views

Why and when do we need to tune hyperparameters?

This might come as a basic question. But I need to understand why do we need to tune the hyper parameters in a machine learning model instead of going into a different model altogether. Or to put it ...
0
votes
1answer
88 views

What is the mathematical definition of the 'Elbow Method'?

In K-means algorithm, it is recommender to pick the optimal K, according to the Elbow Method. However all the tutorials explain the elbow method in these 4 steps: Run K-means for a range of K's ...
0
votes
0answers
9 views

Fourier transforms for noise reduction

Given a signal, which is regularly sampled over time and is noisy. The standard method is with a Fourier transform to reduce the noise and minimise the change to the signal. ...
1
vote
1answer
21 views

Identify suitable scoring metric for food prediction

I am using GridSearchCV to find the best parameter that help me tune XGBoost for a food prediction algorithm. I am struggling to identify the best scoring metric that would result in the best profit (...
0
votes
0answers
13 views

Is sampling the training set for hyperparameter optimisation a good speed up solution?

I am using Bayesian hyperparameter optimization for LSTM hyperparameters. This manages to find an optimum set of hyperparameters in much fewer iterations than a grid search. My problem is that I ...
1
vote
2answers
231 views

What is the “tree” structure in Tree Parzen Estimators?

Context In Algorithms for Hyper-Parameter Optimization, the authors propose a "tree-structured" configuration space. Here, a configuration space is a space of hyperparameters. Questions What ...
1
vote
1answer
59 views

How to decouple weight decay strength and model size?

Consider the neural networks' loss function with the cross entropy term and the $L^2$ weight decay term, which are usually written as: $$E = \frac{1}{N_{samples}} \sum_{i=1}^{N_{samples}} \text{...
0
votes
1answer
50 views

how to do the hyper parameter tunning for one class svm in r programming?

x is input (single column) tuned <- tune.svm(x=x, y =NULL, data=x, type= 'one-classification', tunecontrol = tune.control(sampling = "fix")) For this I am ...
0
votes
1answer
190 views

Gamma distribution and hyperparameters

The formula for mean and variance of a gamma distribution is given by a/b and a/b^2 (hyperparameters) respectively.Are they estimates of the posterior gamma distribution? Can prior, likelihood and ...
0
votes
0answers
6 views

Estimating Beta distribution parameter from Pert distribution ones

The Pert distribution is a modification of a Beta distribution defined by 3 parameters (a, b, c). Is it possible to specify the underlying Beta from those 3 parameters, transforming them into the ...
0
votes
0answers
50 views

Should hyperparamater epsilon be tuned in epsilon-SVR

$ϵ$ in $ϵ$-Support Vector Regression (ϵ-SVR) denotes the $ϵ$-tube within which no penalty is associated in the training loss function with points predicted within a distance epsilon from the actual ...
5
votes
3answers
312 views

introductory machine learning concept questions [closed]

I just started learning about machine learning and the concepts behind the different methods and I wanted to get some clarification on the couple concepts. I'm filling out a true or false handout and ...
0
votes
0answers
14 views

What scoring metric is optimal when performing hyperparameter optimization with a multiclass target variable?

I'm trying to find the optimal hyperparameter for different algorithms where the target variable has 3 classes. I was wondering if maximizing the average AUC over the 3 classes, which I'm currently ...
1
vote
0answers
12 views

Is Determinism important for Hyperparameter Tuning?

When training the Model on GPU, different results are retrieved for the same hyperparameters. This effect can be shut down by using CPU or Tensorflow 2.1. with deterministic settings. The Post on ...
0
votes
0answers
13 views

Optimizing neural networks for maximum utilization of data instead of faster training

A number of methods, some of which I'm familiar with, some not, are used to optimize neural networks for faster convergence. Tricks like batching, different backpropagation algorithms, and so on speed ...
2
votes
2answers
40 views

Optimizing hyperparameters of network with extremely long training time

As an example, let's say i am using a very deep fully convolutional autoencoder to segment lung scans. Input image resolutions will be large, since the features i hope to segment (things like early ...
0
votes
0answers
11 views

Cross validation of sample entropy parameters found using grid search

I have several time-series data spanning multiple weeks and I compute their complexity using sample entropy, which I then, in turn, correlate with a certain numeric value relevant to the particular ...
0
votes
0answers
12 views

Should I model co-variance structure in MCMC model for simulation. Or the posterior parameters will have some correlation due to how MCMC works

I am willing to simulate new data points coming from a dataset. The simulated datasets will be used for sampling experiments prior to training machine learning models to the original data using cross-...
1
vote
1answer
23 views

How do I use Cross-validation to evaluate/validate different settings (“hyperparameters”) in machine learning?

I've gone through a few posts about Cross-validation such as post1, post2, specially the scikit-learn doc, which says When evaluating different settings (“hyperparameters”) for estimators, such as ...
0
votes
0answers
30 views

Sampling from hyperprior with Gibbs sampling

Let's say I have some set of data, $\vec{y}$, where each element is sampled as $$ y_n \sim Normal(\mu,1/\tau) $$ where $\tau$ is the precision, $1/\sigma^2$, and I want to use Gibbs sampling, choosing ...
1
vote
1answer
90 views

How R randomforest sampsize works?

I am working on a predictive model (imbalanced data) and trying to undersample the majority class data. I wanted to get the representative sample of my majority class and somehow came to know about R'...
0
votes
1answer
37 views

Pause, store and resume hyperparameter search with GridSearchCV or RandomizedSearchCV

Suppose you are performing a big hyper-parameter search, using scikit learn RandomizedSearchCV or GridSearchCV. Suppose you are ...
0
votes
0answers
43 views

Tune, validate a model and perform feature selection

I am developing a classification model on some time related dataset. I would like to ask if my procedure for validating my classifier is correct and unbiased. In the single validation split, what I ...
0
votes
1answer
108 views

I am trying to build a monthly revenue prediction model using Prophet in Python but how can I optimize the hyperparameters of the model?

The hyperparameters which I am trying to optimize are: 'n_changepoints', 'changepoint_range', 'holidays_prior_scale', 'seasonality_prior_scale' and 'changepoint_prior_scale'. 'changepoint_range' has a ...
0
votes
0answers
21 views

OCSVM hyper parameters tunning for anomaly detection

I am implementing an One Class Support Vector Machine (OC-SVM) for an anomaly detection module. What I would like to do now is to find the best hyperparameters (nu, gamma, kernel) for my specific case....
0
votes
1answer
43 views

Classification learning curve: function of number of features

I have a binary classification problem where I am using linear SVM. I am interested in diagnosing underfitting/overfitting by visualizing learning curves. My models have different feature sizes; for ...
0
votes
0answers
12 views
0
votes
0answers
36 views

How to estimate the performance of Neural network

I have a feedforward neural network with two hidden layers built in keras. let say I have 40 observations. I split the data into train (e.g., 35 observations) and test (e.g., 5 observations) sets. ...
1
vote
0answers
83 views

cross validation and grid search

I read a lot of topics about cross validation, GridSearchCV, but I noticed a difference about how using train and test set. Generally, given a certain database, It s split in train and test. Example ...
0
votes
1answer
19 views

What should I be careful when using the word “supervised” in paper writing?

I am a biologist using machine learning tool for my research. I modified matrix decomposition ($V \approx WH$) to fit my data and wanted to describe about that in my paper. If I fixed one matrix ...
1
vote
0answers
18 views

How should I display hyperparameter sweeps graphically?

I performed a hyperparameter sweep for a dense neural network. One variable was "model type" where one model has a certain number of layers and nodes per layer. There was 10 of these. I also swept ...
1
vote
2answers
64 views

Select features first or optimize hyperparameters first?

I want to train a binary classification model using some tree ensemble (either xgboost or random forests). My dataset has some 50 features, and I believe some of them are redundant (there's ...
0
votes
1answer
14 views

Using probabilistic scores in Bayesian Optimisation

I was reading up on Bayesian Optimization and in one of the articles, I came across the following passage. We could just use the surrogate score directly. Alternately, given that we have chosen a ...
0
votes
1answer
53 views

Why use cross-validation to do hyperparameter tuning (instead of train/val/test split)?

I understand that to get an unbiased estimate of performance you need some sort of outer CV. However, this assumes you already have dealt with hyperparameter tuning. Nested CV suggests CV for the ...
2
votes
2answers
403 views

Difference between Feature engineering and hyperparameter optimizations?

Hyperparameter optimizations and feature engineering can(in my understanding) both be used to create a machine learning model. But what is the difference? And what is done to the y = wx + b formula in ...
3
votes
0answers
64 views

Parameter dimensionality reduction in a Kalman filter framework

My problem is related to parameter identification with maximum likelihood in a Kalman filter. This framework consists of a multivariate set-up, wherein the unobserved components of the initial ...
0
votes
0answers
20 views

Tuning hyperparameters using sklearn GridSearchCV or validation_curve?

I'm working on tuning a classifier (so far just a decision tree) and running my classifier through both sklearn's GridSearchCV and validation_curve. Is either of these methods preferred and when would ...

1
2
3 4 5
10