Analogue of landscape conjecture in likelihood theory or Bayes?

Question

The so-called landscape conjecture in machine learning says that in high dimensions, most critical points of the loss surface are saddle points rather than poor local minima.

Out of curiosity I was wondering if something similar can be conjectured about high-dimensional likelihood functions and posteriors, for instance in hierarchical models. Common practice is to estimate these models with gradient-based MCMC sampling algorithms (e.g., Hamiltonian Monte Carlo), which intuitively should perform poorly in (truly) multimodal distributions because they struggle to move from one mode to another. For instance, Michael Betancourt mentioned how, because of the potential multimodality of the posterior, there only exist necessary but not sufficient criteria for Markov chain convergence.

But if in say 10,000 dimensions nearly all modes are actually saddle points, then there's less to worry. In a podcast episode, Andrew Holbrook (UCLA) mentioned that "arguments have been made that [...] in higher dimensions, saddle points [...] abound much more frequently than real basins," and we therefore "don't need to be worried so much about getting trapped in local modes" when it comes to convergence of HMC in practice.

Is there more that can be said about this phenomenon (e.g., is this to be expected due to asymptotic results?), and (as the question title suggests) can this be tied to the landscape conjecture in machine learning?

Answer 1

Two simple thoughts:

• it's not enough for local extrema to be rare as a proportion of stationary points; they need to be rare in absolute terms (10 local maxima out of a million stationary points is worse than 5 local maxima out of 10 stationary points)
• MCMC sometimes needs to worry about about multimodality from relabelling in a way that optimisation doesn't. Many models (eg mixture models) are identified only up to labelling of latent variables, which doesn't matter if you just want one of the maxima but can matter if if you want a Markov chain to mix over the entire parameter space -- if it's stuck in one set of labels it is ipso facto not seeing the entire space. You can often fix this by forcing some unique labelling, but I'm not sure that's a general solution.

One data point:

• I have a problem based on a real scientific question that has lots of local maxima. It's a source apportionment problem based on particulate air pollution composition. Given a $T\times S$ data matrix $X$ and known error variances $\Omega$, we want to find a $T\times K$ matrix $G$ and a $K\times S$ matrix $F$ such that $X=GF+E$, with all the entries of $F$ and $G$ non-negative, minimising $\sum_{t,s} e^2_{ts}/\omega_{ts}$. This has lots of local maxima because of the non-negativity constraints -- there tend to be maxima at edges and corners.

One other note:

The form I've seen the landscape conjecture argued, it's based on the idea that a high-dimensional stationary point is unlikely to be a maximum because that would require all the second partial derivatives to be negative. The problem with this argument is that maxima exist for non-local reasons. For example, there is a global maximum in a lot of problems not because random patterns of second derivatives line up to give a maximum but because the likelihood for that problem is necessarily bounded above. I agree that I'd expect large numbers of stationary points that aren't maxima, but I'm not convinced the number of maxima should be very small.