The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
0answers
11 views

Random forest low score on testing data (scikit-learn)

I am trying to train my model using Scikit-learn's Random forest (Regression) and have tried to use GridSearch with Cross-validation (CV=5) to tune hyperparameters. I fixed ...
4
votes
0answers
16 views

Advantages of Particle Swarm Optimization over Bayesian Optimization for hyperparameter tuning?

There's substantial contemporary research on Bayesian Optimization for tuning ML hyperparameters. The driving motivation here is that a minimal number of data points are required to make informed ...
2
votes
1answer
24 views

Levels of “hyperparameterization” in Hierarchical Modeling

Suppose we have observations $y$ that we wish to model as having being randomly sampled from a distribution with parameter $\theta$. General Bayesian approach assumes a prior distribution over ...
0
votes
0answers
26 views

How could a hyperparameter grid search be visualised?

Consider a hyperparameter grid search that looks at the training and testing scores of an estimator with respect to multiple parameters like training epochs, number of nodes in layer 1, number of ...
0
votes
0answers
49 views

normalization to zero mean and variance one logistic regression & random forrests

i was just thinking how does normalization to 0 mean and variance 1 (using an affine linear mapping) can impact the classification accuracy and the choice of hyperparameters when using: logistic ...
0
votes
0answers
75 views

Learn noise parameters from data where data and noise both have Gaussian distributions [closed]

Assuming $X$ and $N$ are two sets of vectors (observations) from a normal distribution, where $X$ represents clean data and $N$ represents noise; and $A$ a projection matrix. the scenario is that our ...
0
votes
0answers
23 views

Different Loss Functions used when fitting the model and tuning hyperparameters with cross validation

I usually do a two step process to fit a model: Given a hyper-parameter, fit the model using some criterion, such as MLE, etc. Do a k-fold cross validation and try a different hyper-parameter on ...
0
votes
0answers
49 views

Is this wikipedia article about KNN contradicting itself regarding “non-parametric”?

I understood that KNN (K-Nearest-Neighbors) was non-parametric, after reading the beginning of the wikipedia article here: ...
0
votes
0answers
28 views

How to determine the number of iterations for Latent Dirichlet Allocation

I am performing Latent Dirichlet Allocation for 240 test documents (trained model with 3361 documents). I am using 150 iterations and 120 burn in iterations. Is there a specific way to determine ...
1
vote
0answers
249 views

How to use XGboost.cv with hyperparameters optimization?

I want to optimize hyperparameters of XGboost using crossvalidation. However, it is not clear how to obtain the model from xgb.cv. For instance I call ...
4
votes
2answers
61 views

How exactly does one marginalize over parameters in an N-dimensional likelihood?

I see no equations for the following, so I'm not sure exactly what they are talking about: "For each model, we determine the best fit parameters from the peak of the N-dimensional likelihood surface. ...
1
vote
1answer
21 views

Can I use cross validation on a subset of the training set to select hyperparameters?

I am using R, and I had a dataset with 400000 rows and 800 columns, training a random forest model with only 100 trees on this dataset will take me about 1 and half hour on my laptop. So I went on and ...
2
votes
1answer
40 views

“Continuity” of SVM as a function of hyperparameters

Suppose we have some (large enough) labeled training set and use some exhaustive cross-validation technique (e.g. leave one out) for tuning hyperparameters of SVM (with some nonlinear kernel). Is it ...
2
votes
1answer
69 views

Can I do hyper-parameters optimization before model selection?

For every N model: Split in test and train subsets(Using the same seed for every N model) Randomized Search of parameters with 5 k-folds on train subset Select the best estimator obtained after the ...
2
votes
1answer
71 views

Hyperparameters Optimisation in Gaussian process for regression

I am trying to perform Gaussian Process for regression. I chose the SE Kernel : $K(x_i,x_j) = \exp(-\frac{||x_i-x_j||^2}{l}) + \sigma_n\delta_{i,j}$. I begin by maximize the log-likelihood with ...
0
votes
1answer
66 views

Which is better, MSE or classification rate, for minimizing the likelihood of overfitting?

During the process of selecting a classification model (with cross validation), if one uses MSE of the predicted posterior probabilities as a selection criterion, then a model ...
3
votes
2answers
153 views

Visualization figures for the grid search optimization steps?

I am using grid search in order to find the best values for the SVM parameters (namely C and ...
1
vote
1answer
141 views

How to optimize RBF parameters $C,\gamma$ with KSVM method?

I want to find the best choice of $C$ and $\gamma$ parameters for Radial Basis Function kernel. I am using kernlab instead of e1071 library. So how can i optimize RBF parameters $C$ and $\gamma$ with ...
0
votes
0answers
139 views

Practical hyperparameter optimization: Random vs. grid search

I'm currently going through Bengio's and Bergsta's Random Search for Hyper-Parameter Optimization [1] where the authors claim random search is more efficient than grid search in achieving ...
0
votes
0answers
76 views

marginal likelihood in linear bayesian regression (in weight-space)

I want to tune the hyperparameters namely the target deviance $\sigma_y$ and weight deviance $\sigma_w$ in bayesian linear regression. The posterior distribution in level-1 inference which is ...
1
vote
1answer
674 views

Hyperparameter Tuning - What is possible in terms of accuracy gain?

A question from a newbie: I played around with parameter tuning (grid, random search) in R (caret, xgboost) and my observation is as follows: in terms of accuracy gains I was able to get 3 - 7% but ...
1
vote
2answers
284 views

Nested cross-validation - how is it different from model selection via kfold CV on the training set?

I often see people talking about 5x2 cross-validation as a special case of nested cross validation. I assume the first number (here: 5) refers to the number of folds in the inner loop and the second ...
0
votes
0answers
28 views

set SVM parameter range values for tuning [duplicate]

I am newbie to using svm for classification. I want to tune svm parameters by .TrainAutofunction in EmguCV. But I don't know what are the range(min-max value) of below parameters that I should give to ...
0
votes
1answer
36 views

Marginal Likelihood of a Gaussian Process Model, Duplicate entries in kernel matrix

I am trying to fit a Gaussian process model using the toolbox and I got stuck in the following problem. Assuming that I have some duplicated data points in my training data, then those will map to ...
4
votes
1answer
147 views

Is an SVM's (maximum) likelihood uniquely defined as a function of hyperparameters?

I think that I must be reading this paragraph (below) incorrectly. Note that both types of evidence that we have defined in general depend on the inverse noise level $C$ and the kernel $K(x, ...
0
votes
0answers
16 views

ensemble model for SVM

I did a nested 5-cv and the resulting models are unstable (high variance among the hyper parameters C and gamma of SVM). So, I don't know how to choose C and gamma for the "final" model. I read that, ...
8
votes
4answers
670 views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
1
vote
1answer
379 views

How to select hyperprior distribution for Beta distribution parameter?

I have a parameter $\theta$ whose value should lie between $(0,1)$. Therefore, I am assuming the prior distribution of $\theta$ to be a beta distribution with hyper-priors $\alpha$ and $\beta$ ie. ...
3
votes
1answer
227 views

Understanding the effect of hyperparameters in machine learning experiments

In machine learning every algorithm has a set of hyperparameters which needs to be optimized for best prediction performance. The simplest method for this optimization is called grid search which ...
1
vote
0answers
100 views

Can you take a DNN that was trained without regularization, and continue training it with regularization?

If I've trained a DNN with out any regularization methods (e.g. weight decay, dropout etc.) and reached a good training error, can I somehow take that learned net and fine tune it with regularization? ...
3
votes
1answer
112 views

The role of $\gamma$ & $C$ in SVM

I'm using support vector machine method with the Gaussian kernel. Is it true that $\gamma$ and $C$ are hyper parameters of SVM? What is their role exactly?
1
vote
2answers
85 views

Are classifier hyperparameters selected within cross-validation or not?

I was reading this question about selecting hyper-parameters for a support vector machine classifier, where grid-search is presented as one option. Which one is correct, either ...
0
votes
1answer
201 views

Estimating correlation hyperparameters of a Gaussian Process

I have an actual function that I need to simulate using a GP model. I've not done this before so I'm unclear of the steps. I have used the true function at different values of the inputs ($\vec X1, ...
2
votes
1answer
173 views

Dirichlet Process Hyperparameter Estimation with Sampling

I have a dirichlet process for which I need to learn the concentration (strength) hyperparameter (with gamma prior on it). One way of doing is via maximizing the Likelihood. Another way of doing this ...
2
votes
1answer
152 views

What is meant by effective parameters in machine learning

My question might be a bit ambiguous, but I started to wonder what does the "effective parameters" mean in machine learning? I have heard few professors of machine learning in my university talk about ...
0
votes
0answers
73 views

learning hyper parameters: are we allowed to touch the prior parameters after observing the data?

There are many algorithms/applications that aim to learn the hyper parameters i.e. the parameters of a prior distribution from the observed data. A typical algorithm works in an iterative function ...
1
vote
2answers
329 views

Likelihood vs. noise kernel hyperparameter in GPML Toolbox

I'm using GPML toolbox by C.E.Rasmussen to solve the basic GP regression problem (presented in the book) with noisy observations. That is to say, estimate the underlying function $f$ of a static noisy ...
10
votes
3answers
2k views

Guideline to select the hyperparameters in Deep Learning

I'm looking for a paper that could help in giving a guideline on how to choose the hyperparameters of a deep architecture, like stacked auto-encoders or deep believe networks. There are a lot of ...
2
votes
0answers
226 views

Gaussian Process , selecting the hyperparameters

I am using Gaussian Process regression toolbox from the site http://www.gaussianprocess.org/gpml/code/matlab/doc/ I was able to use implement the code in matlab easily, following the guide lines. ...
1
vote
1answer
550 views

Grid Search for hyperparameter and feature selection

So I need to select my hyperparameters and also my features. A full grid search of the space of hyperparameters and features is too computationally intensive, so what I am doing instead is for each ...
1
vote
0answers
88 views

How to determine appropriate number of features and also which features to select?

So I have a dataset which I am using K fold cross validation on to select the number of features and which features should be selected. As I understand it, I would set the number of features to be ...
1
vote
0answers
23 views

How to get more continuity in regression forest output

I am using a regression forest. What I have noticed when I plot the quantile distribution of the forest's output is that over a long stretch of quantiles (e.g. $\tau \in [0.1,0.3]$), the output will ...
4
votes
2answers
229 views

How to select penalty parameter after cross validation?

Say I have a feature matrix $X$ and a target $y$. I use $k$-fold cross validation to generate $k$ out-of-sample MSE curves as a function of a penalty parameter $\lambda$ $$MSE_i(\lambda) \quad ...
4
votes
0answers
83 views

Prior elicitation with Normal-Gamma or Normal-Inverse-Gamma

I am looking for a way to have experts elicit a prior for a Normal-Inverse-Gamma Bayesian linear regression model. Is there any material suggesting intuitive ways to go about this?
2
votes
0answers
134 views

Maximizing incomplete likelihood

Given the conditional distribution $p(x|y)$ and the prior of the hidden variables $p(y|\theta)$ with unknown hyper-parameter $\theta$. Now we have observed i.i.d. samples of $x$. Besides the Bayes ...
2
votes
1answer
366 views

Relationship between the kernel and the value of C in SVM's

How exactly does the value of $C$ relate across different kernels that we can use for SVMs? As in, how does it vary when changing the polynomial degree of a kernel or while using a Gaussian kernel?
9
votes
1answer
1k views

Hyperprior density for hierarchical Gamma-Poisson model

In a hierarchical model of data $y$ where $$y \sim \textrm{Poisson}(\lambda)$$ $$\lambda \sim \textrm{Gamma}(\alpha, \beta)$$ it appears to be typical in practice to chose values ($\alpha, \beta)$ ...
2
votes
1answer
178 views

Fully Bayesian hyper-parameter selection in GPML

Is it possible to perform an approximated fully Bayesian (1) selection of hyper-parameters (e.g. covariance scale) with the GPML code, instead of maximizing the marginal likelihood (2) ? I think using ...
7
votes
1answer
919 views

Choosing an appropriate minibatch size for stochastic gradient descent (SGD)

Is there any literature that examines the choice of minibatch size when performing stochastic gradient descent? In my experience, it seems to be an empirical choice, usually found via ...
2
votes
1answer
384 views

Parameter learning of Markov random field

Given a Markov random field $\mathcal{G} = (\mathcal{V},\mathcal{E})$, the corresponding density function to which is expressed by $f(x) \propto \prod_{x\in\mathcal{V}} \psi_u(x) ...