I have been reading several papers on using a bound on log sum of exponentials to be used in variational inference. One example of this case that can happen is in correlated topic models. Where to create simplical topic distribution, we use a logistic normal. Used in the ELBO the expression will contain a variational expectation of log sum of exponentials. However alongside the linear terms containing the parameter the optimization problem cannot be solved in closed form.

There are different bounds introduced to decouple some of the issued with this, but some require still other numerical optimization, such as Newton Raphson method. I wanted to know if there are any specific bounds that I can use that result in closed form solution.

One of these bounds is exploiting the Jensen's ineqaulity as follows: $$\mathbb{E}_{q}\Big[\ln\ \sum_{k}\exp(\theta_k)\Big]\leq \ln\ \Big( \sum_{k}\mathbb{E}_{q}\ [\exp(\theta_k)]\Big)$$ where $\theta \sim \text{Normal}(\mu,\Lambda^{-1}),$.

Others as in Blei and Lafferty 2006 use the following bound: $$\mathbb{E}_{q}\Big[\ln\ \sum_{k}exp(\theta_k)\Big]\leq \ln\ \omega +\dfrac{\sum_{k}\exp(\theta_k)-\omega}{\omega}.$$ Although useful, I was wondering if there are any ways that can help with single step closed-form solution to the optimization problem.

  • $\begingroup$ Are you looking for a single-step, closed form solution in the vein of, say, linear regression? Models consisting of conditionally conjugate exponential families have closed form updates, but I'm not sure this is what you're after. $\endgroup$ Jun 29 '17 at 16:34
  • $\begingroup$ So, imagine the same model as correlated topic model, where instead of Dirichlet multinomial the logistic normal multinomial is introduced. This model is no more conditionally conjugate. Hence regularly there are no more closed form solutions, unless we use some sort of local variational approximation to the part causing trouble. $\endgroup$
    – A.Yazdiha
    Jun 29 '17 at 21:46
  • $\begingroup$ Sure, but that's what I mean to confirm: You're looking for a bound that lets you derive closed-form updates for each variational parameter, in order to iterate over until convergence of the variational objective, is that correct? $\endgroup$ Jun 29 '17 at 21:51
  • $\begingroup$ yes, that's correct $\endgroup$
    – A.Yazdiha
    Jun 30 '17 at 8:39

This paper here provides many different ways to approximate the quantity of interest. Some of them introduce variational parameters that admit closed form coordinate ascent updates that you can iterate over until convergence. The context is slightly different (mixed logit models), but the underlying problem of approximating the expectation of the log-sum-exp term is the same.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.