Skip to main content
Cross Validated

Return to Question

added another reference
Source Link
Durden
Durden
  • 1.5k
  • 20
  • 29

The so-called landscape conjecture in machine learning says that in high dimensions, most critical points of the loss surface are saddle points rather than poor local minima.

Out of curiosity I was wondering if something similar can be conjectured about high-dimensional likelihood functions and posteriors, for instance in hierarchical models. Common practice is to estimate these models with gradient-based MCMC sampling algorithms (e.g., Hamiltonian Monte Carlo), which intuitively should perform poorly in (truly) multimodal distributions because they struggle to move from one mode to another. For instance, Michael Betancourt mentioned how, because of the potential multimodality of the posterior, there only exist necessary but not sufficient criteria for Markov chain convergence.

But if in say 10,000 dimensions nearly all modes are actually saddle points, then there's less to worry. In a podcast episode, Andrew Holbrook (UCLA) mentioned that "arguments have been made that [...] in higher dimensions, saddle points [...] abound much more frequently than real basins," and we therefore "don't need to be worried so much about getting trapped in local modes" when it comes to convergence of HMC in practice.

Is there more that can be said about this phenomenon (e.g., is this to be expected due to asymptotic results?), and (as the question title suggests) can this be tied to the landscape conjecture in machine learning?

The so-called landscape conjecture in machine learning says that in high dimensions, most critical points of the loss surface are saddle points rather than poor local minima.

Out of curiosity I was wondering if something similar can be conjectured about high-dimensional likelihood functions and posteriors, for instance in hierarchical models. Common practice is to estimate these models with gradient-based MCMC sampling algorithms (e.g., Hamiltonian Monte Carlo), which intuitively should perform poorly in (truly) multimodal distributions because they struggle to move from one mode to another. But if in say 10,000 dimensions nearly all modes are actually saddle points, then there's less to worry. In a podcast episode, Andrew Holbrook (UCLA) mentioned that "arguments have been made that [...] in higher dimensions, saddle points [...] abound much more frequently than real basins," and we therefore "don't need to be worried so much about getting trapped in local modes" when it comes to convergence of HMC in practice.

Is there more that can be said about this phenomenon (e.g., is this to be expected due to asymptotic results?), and (as the question title suggests) can this be tied to the landscape conjecture in machine learning?

The so-called landscape conjecture in machine learning says that in high dimensions, most critical points of the loss surface are saddle points rather than poor local minima.

Out of curiosity I was wondering if something similar can be conjectured about high-dimensional likelihood functions and posteriors, for instance in hierarchical models. Common practice is to estimate these models with gradient-based MCMC sampling algorithms (e.g., Hamiltonian Monte Carlo), which intuitively should perform poorly in (truly) multimodal distributions because they struggle to move from one mode to another. For instance, Michael Betancourt mentioned how, because of the potential multimodality of the posterior, there only exist necessary but not sufficient criteria for Markov chain convergence.

But if in say 10,000 dimensions nearly all modes are actually saddle points, then there's less to worry. In a podcast episode, Andrew Holbrook (UCLA) mentioned that "arguments have been made that [...] in higher dimensions, saddle points [...] abound much more frequently than real basins," and we therefore "don't need to be worried so much about getting trapped in local modes" when it comes to convergence of HMC in practice.

Is there more that can be said about this phenomenon (e.g., is this to be expected due to asymptotic results?), and (as the question title suggests) can this be tied to the landscape conjecture in machine learning?

Source Link
Durden
Durden
  • 1.5k
  • 20
  • 29

Analogue of landscape conjecture in likelihood theory or Bayes?

The so-called landscape conjecture in machine learning says that in high dimensions, most critical points of the loss surface are saddle points rather than poor local minima.

Out of curiosity I was wondering if something similar can be conjectured about high-dimensional likelihood functions and posteriors, for instance in hierarchical models. Common practice is to estimate these models with gradient-based MCMC sampling algorithms (e.g., Hamiltonian Monte Carlo), which intuitively should perform poorly in (truly) multimodal distributions because they struggle to move from one mode to another. But if in say 10,000 dimensions nearly all modes are actually saddle points, then there's less to worry. In a podcast episode, Andrew Holbrook (UCLA) mentioned that "arguments have been made that [...] in higher dimensions, saddle points [...] abound much more frequently than real basins," and we therefore "don't need to be worried so much about getting trapped in local modes" when it comes to convergence of HMC in practice.

Is there more that can be said about this phenomenon (e.g., is this to be expected due to asymptotic results?), and (as the question title suggests) can this be tied to the landscape conjecture in machine learning?