HMC: How many dimensions is too many?

Question

From what I have read Hamiltonian Monte Carlo is the "goto" MCMC method when your problem is high dimensional.

Practically speaking, how many dimensions 10's, 100's, 1,000's, 10,000's, 100,000's, ..., is too many? Computational cost will no doubt become an issue and I suppose the model being used is important to consider but those things aside, is there a practical limit to the number of dimensions when it comes to getting good samples from the desired distribution using HMC?

Also, how can we monitor the convergence (or lack there of I guess) for problems where the number of parameters is far too many to check trace plots, running means, autocorrelations, etc. for individual parameters?

Update: Found this post which mentions some non-visual diagnostics

Answer 1

Maximum number of parameters

It depends a lot on the structure of your problem. For example, my experience with various hierarchical linear models in Stan was that it starts being very slow (hours or days to complete) at around 10 000 - 30 000 params (some reproducible numbers are on my blog on Stan vs. INLA). When working with models involving ordinary differential equations and complex structure, 10 parameters may be too many. When fitting just a vector of independent normals (see below), Stan takes about 40 minutes to complete with 1e5 parameters, using default settings (1000 iter warmup, 1000 iter sampling, 4 chains). So having much more than 1e5 params is very likely to be impractical.

The longest part of a Stan run is however the warmup phase when the hyperparameters of the algorithm are tweaked. If you could provide good values for those by yourself (which is hard), you might be able to push the performance even further.

Also, the support of MPI for within-chain parallelism and offloading matrix operations to GPU should be added to Stan soon (See e.g. discussion here http://discourse.mc-stan.org/t/parallelization-again-mpi-to-the-rescue/455/11, and here http://discourse.mc-stan.org/t/stan-on-the-gpu/326/10) so even larger models are likely to become practical in the near future.

Diagnostics in high dimension

The HMC implementation in Stan provides multiple useful diagnostics that work even with large number of parameters: divergent transitions, n_eff (effective sample size) and split Rhat (potential scale reduction). See Stan manual, section "Initialization and Convergence Monitoring" for detailed explanation of those.

---

R code for a simple model - just a set of independent normals which can scale in the number of parameters, fit in Stan:

library(rstan)
model_code = "
data {
 int N;
}

parameters {
 vector[N] a;
}

model {
 a ~ normal(0,1);
}
"

model = stan_model(model_code = model_code)

fit_large = sampling(model, data = list(N = 1e5))