1
$\begingroup$

This paper (p. 1751) claims that variational methods do not suffer the label switching problem inherent to Bayesian estimation of mixture models. However, I struggle to find additional references and it is not clear to me why is so.

$\endgroup$
5
  • $\begingroup$ Hopefully fixed, thanks for the heads up $\endgroup$
    – econ86
    May 6 at 19:29
  • 1
    $\begingroup$ Concerning the part of your question where you state you struggle to find additional references. The paper states "VB does not suffer from the label-switching problem when fitting mixture models (see Celeux et al. 2000)". Have you looked into the reference in the quotation? $\endgroup$
    – microhaus
    May 6 at 19:43
  • 1
    $\begingroup$ Sure. That paper refers to the label switching problem itself, nothing to do with VB $\endgroup$
    – econ86
    May 6 at 19:45
  • 1
    $\begingroup$ @microhaus: in this paper of ours, we do not show that VB does not suffer from the problem, do not cover VB at all, but instead propose solutions to overcome the label switching problem within the genuine posterior distribution. $\endgroup$
    – Xi'an
    May 6 at 19:49
  • 1
    $\begingroup$ @Xi'an. Thank you for clarifying this. I don't have intimate knowledge of the paper, rather wanted to exhaust the possibility that econ86 had omitted checking the reference. $\endgroup$
    – microhaus
    May 6 at 19:58
1
$\begingroup$

The standard VB approximation to the mixture posterior as in McGrory and Titterington (2007) is using conjugate distributions

  • $(\omega_1,\ldots,\omega_k)\sim \mathcal D(\alpha_1,\ldots,\alpha_k)$
  • $\mu_i|\sigma_is\sim\mathcal N(\xi_i,\delta_i\sigma_i^2)$
  • $\sigma_i^{-2}\sim\mathcal G(\alpha_i,\beta_i)$

and the hyperparameters in these distributions are optimised to bring the approximation to be as close as possible to the true posterior. Label switching$¹$ means that any permutation of the indices does not change the likelihood, but VB being an optimisation technique, once iterations have started they will not switch to a permutation of the present hyperparameters, just like EM converges to a particular mode of the likelihood, depending on the choice of the starting value.


$¹$The reference to our paper (Celeux et al., 2000) is intended to point out the label switching difficulties, not to show that VB does not suffer from label switching.

$\endgroup$
4
  • $\begingroup$ I see. But if the hyperparameters are not optimized and for instance set to exchangeable priors (e.g. $\alpha_1 = \dots = \alpha_k$), are we under the same type of unidentifiability even if conducting VB? $\endgroup$
    – econ86
    May 6 at 19:49
  • $\begingroup$ In my understanding, VB is always optimised within the chosen family, in order to minimise the approximation error. Picking an arbitrary value for the hyperparameters does not seem to make sense. $\endgroup$
    – Xi'an
    May 6 at 19:51
  • $\begingroup$ Shouldn't its importance be relatively minor with a lot of data, just as priors in standard Bayesian settings? (Also, could you please point to a reference on optimizing the hyperparameters?) $\endgroup$
    – econ86
    May 6 at 19:54
  • 1
    $\begingroup$ The paper you cited does exactly that. And the whole VB literature. $\endgroup$
    – Xi'an
    May 6 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.