Difference between stochastic variational inference and variational inference?

Question

Very simple, as the question header says: what is the difference between SVI and VI?

I cannot seem to find a clear-cut definition online.

Answer 1

Have a look at the paper Stochastic Variational Inference:

The coordinate ascent algorithm in Figure 3 is inefficient for large data sets because we must optimize the local variational parameters for each data point before re-estimating the global variational parameters. Stochastic variational inference uses stochastic optimization to fit the global variational parameters.

So instead of getting the gradient from the full dataset, you obtain the natural gradient from batches.

Answer 2

Stochastic VI means you don't use the exact, complete, information you have [because it's too complicated, or computationally expensive] but rather a stochastic version of it.

While the paper about SVI only deals with the Exponential Family, and one type of stochasticity, I think the term should also apply to any general purpose VI algorithm where you use Stochastic-Gradient-Ascent (e.g., Automatic Differentiation VI / ADVI), or any VI algorithm that uses some form of stochasticity.

In my opinion you can divide the literature into 2:

1. SVI for Expo. Family (e.g., Stochastic Variational Inference, Hoffman et al. 2013)
2. SVI in General (e.g., ADVI, Kucukelbir et al. 2016)

The stochasticity doesn't have to come only from optimizing the "local" parameters by sampling from your dataset $x$, as in the global-local type of problems that the SVI paper looks at, but it can also come from using MC integration and sampling your parameters $\theta$ as the ADVI paper uses.