16
votes
Accepted
Elementary explanation of Gaussian Processes
A stochastic process $X(t), t \in T$ is a Gaussian process (GP) if $\sum_i a_i X(t_i)$ is a Gaussian random variable for any such linear combination. Equivalently, it is a GP if all its finite-...
- 1,841
14
votes
What are some of the disavantage of bayesian hyper parameter optimization?
results are sensitive to parameters of the surrogate model, which are typically fixed at some value; this underestimates uncertainty; or else you have to be fully Bayesian and marginalize over hyper ...
- 94.1k
14
votes
Accepted
Why go through the trouble of expectation maximization and not use gradient descent?
There are several advantages of the EM algorithm over gradient descent:
Monotonic convergence. The EM algorithm never decreases the log-likelihood. This is not necessarily true for gradient descent.
...
- 21.6k
12
votes
Estimating a "most likely" distribution from min, max, mean, median, standard deviation
Obviously as a starting point to this kind of question, we must recognise that there are may possible probability distributions on the relevant support that have the same (finite) set of moments. ...
- 133k
11
votes
Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function
You are correct. The Gaussian process is a distribution over functions. As with other Bayesian methods, you start with a prior and combine it with data (observed outcome) through likelihood to get a ...
- 141k
8
votes
Why does Bayesian optimization work?
Gaussian Process models have two nice properties in the Bayesian Optimization context:
They can exactly fit the observed values of the black-box function $f$ (when there is no noise, at least)
They ...
- 94.1k
8
votes
Accepted
How to calculate the standard deviation for a Gaussian Process?
The closed form equation for the predicted covariance inverts $K(x_d, x_d)$. However, using the Cholesky decomposition is faster and more numerically stable than directly taking the inverse. You can ...
- 812
7
votes
Learning prior distribution from data
From a dataset $x_1,\ldots,x_n$, generated from a fixed distribution $f(\cdot;\theta_0)$, one can learn about the single value $\theta_0$ of the parameter that led to its generation, that is, the ...
- 108k
6
votes
What are good alternatives to grid search?
One nice option, which is at the same time very easy to implement is Random Search for Hyper-Parameter Optimization. It is also implemented in scikit-learn (see section 3.2.2).
The idea is to have an ...
- 14.3k
6
votes
Accepted
Intuitive Understanding of Expected Improvement for Gaussian Process
My question is are we searching for the point in the Gaussian Process model whose expected value (determined by mean and confidence) shall be decreased the most if sampled at that point?
No. At any ...
- 812
6
votes
Accepted
Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function
In addition to Tim's answer, here are some slight nitpicks/clarifications which might assist your intuitive understanding/prevent possible confusion in the future:
We start with a a-prior function, ...
- 785
6
votes
Accepted
How to run Bayesian optimization experiments in parallel?
The problem with partitioning the search space yourself upfront and then running Bayesian optimization independently on each of them is that it is quite likely that most of your partitions don't ...
- 11.2k
5
votes
Accepted
Bayesian Hyperparameter Tuning
What it does
Bayesian optimization is a general, black box optimization strategy that works in the regime where the objective function may be stochastic and we don't necessarily have an expression ...
5
votes
Bayesian optimizations of hyper-parameters - magical curve based on two data points?
Definition
A Gaussian Process (GP) is a Gaussian distribution over functions:
$$f(x) \sim GP(m(x), k(x, x^\prime))$$
where $m(x)$ is the mean function and $k(x, x^\prime)$ is a kernel function. The ...
- 3,329
5
votes
Accepted
what is the difference between Bayesian optimization and kriging?
I believe you mean Gaussian processes rather than Bayesian optimisation. Bayesian optimisation is the use of Gaussian processes for global optimisation. Essentially you use the mean and variance of ...
- 2,392
5
votes
what is the difference between Bayesian optimization and kriging?
I debated answering this because I have not used kriging in probably twelve years. They are also closely related. It is somewhat of a bridge between the extremes. Still, that kriging is the BLUP ...
- 7,810
5
votes
How to use the squared exponential kernel with multidimensional vector inputs?
As you've written it here, $\sigma$ and $\ell$ are scalars. You could use a similar kernel, sometimes called an "Automatic Relevance Determination" (ARD) kernel, where $\ell$ is a vector of the same ...
- 25.2k
5
votes
Accepted
Calculating the possible number of configurations
Each hyper parameter listed has $5,5,6,4,4,8$ different values listed. If multiplied, total number of combinations make $5\times5\times6\times4\times4\times8=19200$.
- 58.2k
5
votes
Accepted
Difference between Bayesian Optimization and Bayesian Statistical Inference
I'll take a go at this.
Bayesian optimization is concerned with finding optimal parameters for a model, with respect to an objective function, by intelligently sampling from the parameter space. A ...
- 335
5
votes
How to change priors in Bayesian estimation, if we get to know previous work was wrong?
Suppose you are trying to predict the outcome of an election on Proposition A, which needs 60% Yes votes to pass. Let $\theta$ be the unknown proportion of the electorate in favor.
Maybe you have a ...
- 57.6k
5
votes
Accepted
How does Bayesian Optimization balance exploration with exploitation?
In Bayesian optimization (BO), one chooses the next sampling point by maximizing the acquisition function $a(x)$, i.e.
\begin{equation}
x^* = \arg\max_{x\in\mathcal{X}} a(x).
\end{equation}
This ...
- 292
5
votes
Accepted
Terms and assumptions in trans-dimensional MCMC (RJ-MCMC) for Green 1995 paper
The key idea here is that we want a ratio of forward and backward proposal measures that is finite and non-zero. If $x$ and $x'$ have different numbers of dimensions, you will get infinite or zero ...
- 46.8k
4
votes
Accepted
Why are ("non-parametric") Gaussian Processes a good fit for Bayesian Optimization
Parametric models assume the samples are from a specific distribution e.g. from a mixture of Gaussians where the number of Gaussian components is known a priori. This is restrictive since for most ...
- 965
4
votes
How to find the "right" hyperparameters for the Gaussian process used Bayesian optimization
An alternative to MAP or MLE estimates is to sample hyperparameters (i'll call them $\theta$) from a distribution proportional to their probability, given your data and some prior assumptions. This is ...
- 785
4
votes
Accepted
Bayesian optimization neural network
One way of doing that could be by optimizing over the number of neurons in each layer, where the number of neurons can be 0 (no layer). In order to avoid duplicate configurations (e.g., 10, 10, 0 and ...
- 156
4
votes
Adaptive Parzen Estimator
I was also having some trouble understanding everything when reading the paper. To the specific questions you listed here, I might provide my understanding:
Parzen estimators (or Parzen windows) are ...
4
votes
What is the relation between a surrogate function and an acquisition function?
I think of an acquisition function as describing the utility of the point to be evaluated next in the Bayesian optimization framework.
To give more details, let's think about the general concept of ...
- 385
4
votes
Accepted
Why are frequentists uncomfortable with bayesian statistics when "optimization" algorithms used in frequentist statistics is bayesian?
I'm not sure I really understand your analogy, but it seems like your analogy doesn't fully capture either the objections or the philosophy of Bayesianism.
If I were going to criticize Bayesianism, I'...
- 39.6k
4
votes
Elementary explanation of Gaussian Processes
You are given points sampled from a function, that is not known, this is your data. Traditional curve fitting algorithms would try to find a function that fits the data the best. Gaussian process ...
- 141k
Only top scored, non community-wiki answers of a minimum length are eligible