# log-sum-exp bound to be used in variational inference

I have been reading several papers on using a bound on log sum of exponentials to be used in variational inference. One example of this case that can happen is in correlated topic models. Where to create simplical topic distribution, we use a logistic normal. Used in the ELBO the expression will contain a variational expectation of log sum of exponentials. However alongside the linear terms containing the parameter the optimization problem cannot be solved in closed form.

There are different bounds introduced to decouple some of the issued with this, but some require still other numerical optimization, such as Newton Raphson method. I wanted to know if there are any specific bounds that I can use that result in closed form solution.

One of these bounds is exploiting the Jensen's ineqaulity as follows: $$\mathbb{E}_{q}\Big[\ln\ \sum_{k}\exp(\theta_k)\Big]\leq \ln\ \Big( \sum_{k}\mathbb{E}_{q}\ [\exp(\theta_k)]\Big)$$ where $$\theta \sim \text{Normal}(\mu,\Lambda^{-1}),$$.

Others as in Blei and Lafferty 2006 use the following bound: $$\mathbb{E}_{q}\Big[\ln\ \sum_{k}exp(\theta_k)\Big]\leq \ln\ \omega +\dfrac{\sum_{k}\exp(\theta_k)-\omega}{\omega}.$$ Although useful, I was wondering if there are any ways that can help with single step closed-form solution to the optimization problem.

• Are you looking for a single-step, closed form solution in the vein of, say, linear regression? Models consisting of conditionally conjugate exponential families have closed form updates, but I'm not sure this is what you're after. Jun 29, 2017 at 16:34
• So, imagine the same model as correlated topic model, where instead of Dirichlet multinomial the logistic normal multinomial is introduced. This model is no more conditionally conjugate. Hence regularly there are no more closed form solutions, unless we use some sort of local variational approximation to the part causing trouble. Jun 29, 2017 at 21:46
• Sure, but that's what I mean to confirm: You're looking for a bound that lets you derive closed-form updates for each variational parameter, in order to iterate over until convergence of the variational objective, is that correct? Jun 29, 2017 at 21:51
• yes, that's correct Jun 30, 2017 at 8:39