What is the difference between inducing location and minibatch data points in Stochastic Variational Gaussian Process?

Question

I'm reading about Gaussian Processes (GP) and specifically about scalable Gaussian processes, namely Stochastic Variational Gaussian Process (SVGP).

I've been using the GPFlow-library for implementing GP regression for big data, and the example code I've been using can be found from this source: https://gpflow.github.io/GPflow/develop/notebooks/advanced/gps_for_big_data.html

I see the term "inducing locations" being used in this context, which are explained in the tutorial as, quote:

The main idea behind SVGP is to approximate the true GP posterior with a GP conditioned on a small set of “inducing” values. This smaller set can be thought of as summarizing the larger dataset.

Later on in the same example code, the authors talk about "minibatches", which I assume is the same idea as in deep learning context: you use a small subset of the data to perform gradient descent -like of iterative training for the model, because you can't use all the data at once due to computational issues.

My confusion comes from the difference between the "inducing location" and "minibatch" data points. It is not fully clear to me the differing roles of these subsets of the data, because they seem very similar in their purpose. I think this would come clear for me if I read couple of papers and a book on the subject, but maybe I could save weeks worth of time and get the general idea from the community :)

So my question is: What is the difference between the minibatch and the inducing location -subsets of the data in SVGP ? Can they be equal?

Answer 1

I might have found a solution myself, but I'm not sure of it. The suggested solution comes from this paper:

Gaussian Processes for Big Data,

which is referenced in the GPFlow-page. If I have understood correctly, the relation between the inducing location and minibatch data points is illustrated by the Gaussian process hyperparameter (e.g. the prior kernel variance) update equation (see page 3 in the reference article):

$$\boldsymbol\theta_{1(t+1)})=\textbf{S}^{-1}_{(t)}\textbf{m}_{(t)}+\mathcal{l}\left(\beta\textbf{K}^{-1}_{mm}\textbf{K}_{mn}\textbf{y}-\textbf{S}^{-1}_{(t)}\textbf{m}_{(t)}\right)$$

The relevant symbols regarding my question here are the matrices $\textbf{K}^{-1}_{mm}$ and $\textbf{K}_{mn}$. The whole point of using the "inducing locations" is to have a representative approximation of the kernel matrix $\textbf{K}_{nn}$ with the help of matrix $\textbf{K}_{mm}$ and $\textbf{K}_{mn}$, where $n$ is the total number of data points and $m$ is the number of "inducing" data points, which represent the whole data set of size $n$. We also assume $m<<n$.

For calculating $\textbf{K}^{-1}_{mm}$, we need the $m$ inducing data points, and to calculate $\textbf{K}_{mn}$ we need the inducing points + some batch (i.e. subset) from the set of all data points. In calculating $\textbf{K}_{mn}$, this matrix can still be pretty big, because $n$ is large, so instead, we might take some subset (minibatch) of size $p \leq n-m$ from the set of remaining non-inducing data points for which we can calculate $\textbf{K}_{mp}\approx\textbf{K}_{mn}$.

If I have understood this correctly, this is the relation between the inducing and minibatch data points.