# Tag Info

Accepted

### Elementary explanation of Gaussian Processes

A stochastic process $X(t), t \in T$ is a Gaussian process (GP) if $\sum_i a_i X(t_i)$ is a Gaussian random variable for any such linear combination. Equivalently, it is a GP if all its finite-...
• 1,831

### What are some of the disavantage of bayesian hyper parameter optimization?

results are sensitive to parameters of the surrogate model, which are typically fixed at some value; this underestimates uncertainty; or else you have to be fully Bayesian and marginalize over hyper ...
• 92.3k
Accepted

### Why go through the trouble of expectation maximization and not use gradient descent?

There are several advantages of the EM algorithm over gradient descent: Monotonic convergence. The EM algorithm never decreases the log-likelihood. This is not necessarily true for gradient descent. ...
• 21.4k

### Estimating a "most likely" distribution from min, max, mean, median, standard deviation

Obviously as a starting point to this kind of question, we must recognise that there are may possible probability distributions on the relevant support that have the same (finite) set of moments. ...
• 129k

### Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

You are correct. The Gaussian process is a distribution over functions. As with other Bayesian methods, you start with a prior and combine it with data (observed outcome) through likelihood to get a ...
• 140k

### Why does Bayesian optimization work?

Gaussian Process models have two nice properties in the Bayesian Optimization context: They can exactly fit the observed values of the black-box function $f$ (when there is no noise, at least) They ...
• 92.3k
Accepted

### How to calculate the standard deviation for a Gaussian Process?

The closed form equation for the predicted covariance inverts $K(x_d, x_d)$. However, using the Cholesky decomposition is faster and more numerically stable than directly taking the inverse. You can ...
• 812

### What are good alternatives to grid search?

One nice option, which is at the same time very easy to implement is Random Search for Hyper-Parameter Optimization. It is also implemented in scikit-learn (see section 3.2.2). The idea is to have an ...
• 14.2k

### Learning prior distribution from data

From a dataset $x_1,\ldots,x_n$, generated from a fixed distribution $f(\cdot;\theta_0)$, one can learn about the single value $\theta_0$ of the parameter that led to its generation, that is, the ...
• 106k
Accepted

### Intuitive Understanding of Expected Improvement for Gaussian Process

My question is are we searching for the point in the Gaussian Process model whose expected value (determined by mean and confidence) shall be decreased the most if sampled at that point? No. At any ...
• 812
Accepted

### Question of understanding regarding Bayesian Optimization, Gaussian process and acquisition function

In addition to Tim's answer, here are some slight nitpicks/clarifications which might assist your intuitive understanding/prevent possible confusion in the future: We start with a a-prior function, ...
• 785
Accepted

### How to run Bayesian optimization experiments in parallel?

The problem with partitioning the search space yourself upfront and then running Bayesian optimization independently on each of them is that it is quite likely that most of your partitions don't ...
• 11k
Accepted

### Bayesian Hyperparameter Tuning

What it does Bayesian optimization is a general, black box optimization strategy that works in the regime where the objective function may be stochastic and we don't necessarily have an expression ...

### Bayesian optimizations of hyper-parameters - magical curve based on two data points?

Definition A Gaussian Process (GP) is a Gaussian distribution over functions: $$f(x) \sim GP(m(x), k(x, x^\prime))$$ where $m(x)$ is the mean function and $k(x, x^\prime)$ is a kernel function. The ...
• 3,329
Accepted

### Optimizing a "black box" function: Linear Regression or Bayesian Optimization... what's the difference?

Your understanding is correct. BO inherently measures the uncertainty of regions of your search space. And the acquisition function governs the tradeoff between exploring a point in a region with ...
• 92.3k
Accepted

### what is the difference between Bayesian optimization and kriging?

I believe you mean Gaussian processes rather than Bayesian optimisation. Bayesian optimisation is the use of Gaussian processes for global optimisation. Essentially you use the mean and variance of ...
• 2,382

### what is the difference between Bayesian optimization and kriging?

I debated answering this because I have not used kriging in probably twelve years. They are also closely related. It is somewhat of a bridge between the extremes. Still, that kriging is the BLUP ...
• 7,770

### How to use the squared exponential kernel with multidimensional vector inputs?

As you've written it here, $\sigma$ and $\ell$ are scalars. You could use a similar kernel, sometimes called an "Automatic Relevance Determination" (ARD) kernel, where $\ell$ is a vector of the same ...
• 25.1k
Accepted

### Calculating the possible number of configurations

Each hyper parameter listed has $5,5,6,4,4,8$ different values listed. If multiplied, total number of combinations make $5\times5\times6\times4\times4\times8=19200$.
• 57.8k
Accepted

### Difference between Bayesian Optimization and Bayesian Statistical Inference

I'll take a go at this. Bayesian optimization is concerned with finding optimal parameters for a model, with respect to an objective function, by intelligently sampling from the parameter space. A ...
• 335

### How to change priors in Bayesian estimation, if we get to know previous work was wrong?

Suppose you are trying to predict the outcome of an election on Proposition A, which needs 60% Yes votes to pass. Let $\theta$ be the unknown proportion of the electorate in favor. Maybe you have a ...
• 56.9k
Accepted

### How does Bayesian Optimization balance exploration with exploitation?

In Bayesian optimization (BO), one chooses the next sampling point by maximizing the acquisition function $a(x)$, i.e. $$x^* = \arg\max_{x\in\mathcal{X}} a(x).$$ This ...
• 292
Accepted

### Terms and assumptions in trans-dimensional MCMC (RJ-MCMC) for Green 1995 paper

The key idea here is that we want a ratio of forward and backward proposal measures that is finite and non-zero. If $x$ and $x'$ have different numbers of dimensions, you will get infinite or zero ...
• 41.9k
Accepted

### Why are ("non-parametric") Gaussian Processes a good fit for Bayesian Optimization

Parametric models assume the samples are from a specific distribution e.g. from a mixture of Gaussians where the number of Gaussian components is known a priori. This is restrictive since for most ...
• 965

### How to find the "right" hyperparameters for the Gaussian process used Bayesian optimization

An alternative to MAP or MLE estimates is to sample hyperparameters (i'll call them $\theta$) from a distribution proportional to their probability, given your data and some prior assumptions. This is ...
• 785
Accepted

### Bayesian optimization neural network

One way of doing that could be by optimizing over the number of neurons in each layer, where the number of neurons can be 0 (no layer). In order to avoid duplicate configurations (e.g., 10, 10, 0 and ...
• 156

### What is the relation between a surrogate function and an acquisition function?

I think of an acquisition function as describing the utility of the point to be evaluated next in the Bayesian optimization framework. To give more details, let's think about the general concept of ...
• 385
Accepted

### Why are frequentists uncomfortable with bayesian statistics when "optimization" algorithms used in frequentist statistics is bayesian?

I'm not sure I really understand your analogy, but it seems like your analogy doesn't fully capture either the objections or the philosophy of Bayesianism. If I were going to criticize Bayesianism, I'...
• 38.1k