Very simple, as the question header says: what is the difference between SVI and VI?
I cannot seem to find a clear-cut definition online.
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
Have a look at the paper Stochastic Variational Inference:
The coordinate ascent algorithm in Figure 3 is inefficient for large data sets because we must optimize the local variational parameters for each data point before re-estimating the global variational parameters. Stochastic variational inference uses stochastic optimization to fit the global variational parameters.
So instead of getting the gradient from the full dataset, you obtain the natural gradient from batches.
Stochastic VI means you don't use the exact, complete, information you have [because it's too complicated, or computationally expensive] but rather a stochastic version of it.
While the paper about SVI only deals with the Exponential Family, and one type of stochasticity, I think the term should also apply to any general purpose VI algorithm where you use Stochastic-Gradient-Ascent (e.g., Automatic Differentiation VI / ADVI), or any VI algorithm that uses some form of stochasticity.
In my opinion you can divide the literature into 2:
The stochasticity doesn't have to come only from optimizing the "local" parameters by sampling from your dataset $x$, as in the global-local type of problems that the SVI paper looks at, but it can also come from using MC integration and sampling your parameters $\theta$ as the ADVI paper uses.