lacerbi
  • Member for 6 years, 7 months
  • Last seen this week
Converting random 6-faced dice results to decimal base
0 votes

If re-rolls are allowed (which is unclear from your question, but nothing seems to prevent it from the problem setup), there is a fairly easy solution. For simplicity (but it's not required), suppose ...

View answer
Optimization when Cost Function Slow to Evaluate
46 votes

The literature on evaluation of expensive black-box function is quite vast and it is usually based on surrogate-model methods, as other people pointed out. Black-box here means that little is known ...

View answer
Computing joint entropy from marginal distributions
Accepted answer
1 votes

If they are (conditionally) independent, as you say, then the joint entropy (conditioned on the same thing as above) is the sum of the marginal entropies. This follows immediately from the ...

View answer
Log marginal likelihood for Gaussian Process
Accepted answer
8 votes

The more general formulation for the log marginal likelihood (not marginal log likelihood, as you originally wrote - I edited it in your post) of a GP is $$\log p(y|X) = -\frac{1}{2}(y - m(X))^T(K+\...

View answer
Why should I be Bayesian when my model is wrong?
14 votes

Edits: Added reference to this paper in the body, as requested by the OP. I am giving an answer as a naive empirical Bayesian here. First, the posterior distribution allows you to do computations ...

View answer
Performance benchmarks for MCMC
Accepted answer
6 votes

After some online searching, I have come under the impression that a comprehensive benchmark of established MCMC methods, analogous to what one can find in the optimization literature, does not exist. ...

View answer
MCMC convergence
10 votes

It means that your chain most likely did not converge. By this I mean you should be wary of the entire chain, not just worry about the dimensions with low effective number of samples. Solutions Burn-...

View answer
When are genetic algorithms a good choice for optimization?
Accepted answer
22 votes

Genetic algorithms (GA) are a family of heuristics which are empirically good at providing a decent answer in many cases, although they are rarely the best option for a given domain. You mention ...

View answer
Objective function of a Genetic Algorithm
Accepted answer
2 votes

NoS means "numbers of" (that is, count the occurrences of the substring in the individual's chromosome). F(x) is the fitness you are computing (a scalar number). In particular, mNoS("110") where m is ...

View answer
When (and why) do Bayesians reject valid Bayesian methods?
Accepted answer
12 votes

I would like to correct an erroneous assumption in the original post, a mistake which is relatively common. The OP says: From what I have read and from answers to other questions I have asked here, ...

View answer
Preventing Pareto smoothed importance sampling (PSIS-LOO) from failing
Accepted answer
9 votes

For the record, I posted a similar question to the Stan users mailing list, which you can find here. I was answered by one of the authors of the original PSIS-LOO paper and by other contributors of ...

View answer
Hamiltonian/Hybrid MCMC 'mass matrix' terminology
Accepted answer
6 votes

A linear transformation of the position variables is equivalent to the inverse linear transformation of the momentum variables. Ideally, you want to sample from a (transformed) distribution whose ...

View answer
Covariance functions or kernels - what exactly are they?
Accepted answer
15 votes

In loose terms, a kernel or covariance function $k(x, x^\prime)$ specifies the statistical relationship between two points $x, x^\prime$ in your input space; that is, how markedly a change in the ...

View answer
Can Machine Learning or Deep Learning algorithms be utilised to "improve" the sampling process of a MCMC technique?
Accepted answer
28 votes

Yes. Unlike what other answers state, 'typical' machine-learning methods such as nonparametrics and (deep) neural networks can help create better MCMC samplers. The goal of MCMC is to draw samples ...

View answer
Best way to evaluate PDF estimation methods
11 votes

A1. This sounds like a sensible plan to me. Just to mention a couple of points. You'll want to test with different error metrics ($L^p$, K-L divergence, etc.) since methods will perform differently ...

View answer
How to prove this Gaussian Mixture inequality? (Fitting/Overfitting)
2 votes

This is more of an extended comment, so take it as such. Define: $$ f(x) \equiv \frac{1}{n} \sum_{i = 1}^n \mathcal{N}\left(x | x_i, \sigma_i^2 \right) $$ (I am using the standard notation for ...

View answer
Gaussian process with boundaries on unobserved variables
1 votes

Recapping and expanding on my discussion in the comments. If you had $(lb_i, ub_i)$ for every $i$, one approach would be the one I recommended to this other question, that is to have a single GP with ...

View answer
Fit a function f on dataset X such that f(X) fits a histogram
Accepted answer
2 votes

It seems that you want to do is to train a Gaussian process (GP) for regression with observation-dependent noise so as to approximate $f(\textbf{x})$. The idea is that your $y_i$ are noisy (uncertain) ...

View answer
What is the equivalent for cdfs of MCMC for pdfs?
4 votes

This is an attempt which I didn't completely work through, but too long for the comments section. It might be useful to put it here as another basic alternative for very low $k$. It does not require ...

View answer
Optimization of stochastic computer models
10 votes

(Expanding my comment to a proper answer.) As I mentioned, it depends on your goal. The expected value $\mathbb{E}[f(x)]$ is only one of many possible choices for the optimization target. For ...

View answer
cross-validation and over-fitting
Accepted answer
0 votes

There is another simple answer: a bug in the code. Whops. After a long debugging session, I found a subtle indexing mistake in the script that was reading the results of the 10-fold CV. I am writing ...

View answer
practical implementation detail of Bayesian Optimization
Accepted answer
5 votes

The norm is to use any global optimizer you like. The problem is that the EI surface is highly multi-modal and disconnected; optimizing this acquisition function is a nontrivial problem in itself. A ...

View answer
What optimization (maximization/minimization) methods exist contours with lots of kinks?
Accepted answer
7 votes

The state-of-the-art for non-convex optimization in a complex, ill-conditioned and multi-modal landscape is Covariance Matrix Adaptation - Evolution Strategies, aka CMA-ES, which in various versions (...

View answer
State-of-the-art sampling methods for different information about target density
Accepted answer
2 votes

(I am the OP.) For example, my answers would be: Slice sampling as per Neal (2003). It requires very little tuning; you only need to very vaguely know the length scales of your problem (and you can ...

View answer
Marginalization of GP regression hyperparameters with Laplace approximation
Accepted answer
4 votes

The best reference I could find online so far, and a very fitting one, is Ville Pietiläinen's MSc thesis: Approximations for Integration Over the Hyperparameters in Gaussian Processes (2010). ...

View answer