5
$\begingroup$

I have been reading several papers on using a bound on log sum of exponentials to be used in variational inference. One example of this case that can happen is in correlated topic models. Where to create simplical topic distribution, we use a logistic normal. Used in the ELBO the expression will contain a variational expectation of log sum of exponentials. However alongside the linear terms containing the parameter the optimization problem cannot be solved in closed form.

There are different bounds introduced to decouple some of the issued with this, but some require still other numerical optimization, such as Newton Raphson method. I wanted to know if there are any specific bounds that I can use that result in closed form solution.

One of these bounds is exploiting the Jensen's ineqaulity as follows: $$\mathbb{E}_{q}\Big[\ln\ \sum_{k}\exp(\theta_k)\Big]\leq \ln\ \Big( \sum_{k}\mathbb{E}_{q}\ [\exp(\theta_k)]\Big)$$ where $\theta \sim \text{Normal}(\mu,\Lambda^{-1}),$.

Others as in Blei and Lafferty 2006 use the following bound: $$\mathbb{E}_{q}\Big[\ln\ \sum_{k}exp(\theta_k)\Big]\leq \ln\ \omega +\dfrac{\sum_{k}\exp(\theta_k)-\omega}{\omega}.$$ Although useful, I was wondering if there are any ways that can help with single step closed-form solution to the optimization problem.

Improve this question
$\endgroup$
4
  • $\begingroup$ Are you looking for a single-step, closed form solution in the vein of, say, linear regression? Models consisting of conditionally conjugate exponential families have closed form updates, but I'm not sure this is what you're after. $\endgroup$
    – Sean Easter
    Commented Jun 29, 2017 at 16:34
  • $\begingroup$ So, imagine the same model as correlated topic model, where instead of Dirichlet multinomial the logistic normal multinomial is introduced. This model is no more conditionally conjugate. Hence regularly there are no more closed form solutions, unless we use some sort of local variational approximation to the part causing trouble. $\endgroup$
    – A.Yazdiha
    Commented Jun 29, 2017 at 21:46
  • $\begingroup$ Sure, but that's what I mean to confirm: You're looking for a bound that lets you derive closed-form updates for each variational parameter, in order to iterate over until convergence of the variational objective, is that correct? $\endgroup$
    – Sean Easter
    Commented Jun 29, 2017 at 21:51
  • $\begingroup$ yes, that's correct $\endgroup$
    – A.Yazdiha
    Commented Jun 30, 2017 at 8:39

1 Answer 1

Reset to default
3
$\begingroup$

This paper here provides many different ways to approximate the quantity of interest. Some of them introduce variational parameters that admit closed form coordinate ascent updates that you can iterate over until convergence. The context is slightly different (mixed logit models), but the underlying problem of approximating the expectation of the log-sum-exp term is the same.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.