5
$\begingroup$

I'm trying to get into Bayesian model estimation (I'm interested in posterior parameter distributions). I could get away with Metropolis-Hastings and Gibbs Sampling for models with few parameters (< 10), but when parameter dimensionality is extremely high (for example, > 100), the algorithms never converge. I'm stuck with the "curse of dimensionality". I was wondering if any of you could recommend some references to face high dimensional models.

$\endgroup$
1
  • 2
    $\begingroup$ There's generally a choice of how you set up your sampler. In particular, naive implementations of MH and Gibbs may be poor, but implementations that consider the structure of the problem may be much better (initially both in terms of speed of each iteration and iterations to convergence, though as less naive approaches are used and still greater efficiencies are sought there's often a tradeoff between the two). Some indication of the problem might help identify better ways to organize the sampler. Sometimes a fairly simple reparameterization can have a big effect; for example. $\endgroup$
    – Glen_b
    May 22, 2017 at 4:17

1 Answer 1

1
$\begingroup$

Stan Is a good option, IMO. It's pretty general purpose and the learning curve is not too steep. There's also a great Google group where the developers respond very quickly to your questions/queries.

It implements Hamiltonian Monte Carlo (see here for a really cool, informal introduction) -- in particular its No-U-Turn Sampler variant -- which is supposedly better at exploring high-dimensional posteriors than Gibbs/MH. Even if you want to code it yourself, HMC and variants are excellent alternatives to Gibbs/MH. Radford Neal has done a lot of work on HMC and the Stan website has a list of publications which are a good starting point.

One drawback is that Stan/HMC can't deal with discrete parameters, so depending on your problem you might need to go with something different, but it's been able to do pretty much everything I've asked of it and saves a ton of coding time.

$\endgroup$
1
  • $\begingroup$ I could mention PyMC3 as well. Not sure about the differences, but found working with PyMC3 was quite straightforward coming from Python $\endgroup$
    – Tingiskhan
    Jun 5, 2017 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.