I have a hard time understanding what it means that in Gaussian Process (GP) regression, every point is a new dimension. I'm reading the distill article which usually does a very good job explaining things but I have still some troubles: This is the Figure where I get lost: enter image description here First I'm a bit confused because they use $X$ as test data and $Y$ as training data. From general machine learning, I would've expected it the other way around ($y$ usually being the target variable). Then they didn't label the axes. The different colors are a nice idea but don't help me particularly, since I'm not even sure what we find on this axis.

From the leftmost Subfigure we learn that there are 10 $x$ values (I assume $x$ are instances of the random variable $X$.) In the text they write that that means the GP has 10 dimensions. I'm confused why they are all depicted on one axis and how these are different dimensions, even though they all come from $X$. Is this Subfigure representing a 10-dimensional space?

Since $X∼\mathcal{N}(μ_X​,Σ_{XX}​)$, I'm not sure what the middle Subfigure indicates. Our output should be $f(x)$ which will be plotted on the vertical axis in the third Subfigure. Yet in the covariance matrix, there are 10 pairwise interactions between the dark pink point and all the others. What is the mean and variance here and why do they evaluate $k(\mathrm{dark~pink}, \mathrm{light~pink})$ instead of $k(\mathrm{dark~pink}, \mathrm{dark~pink})$ as might be expected from $\Sigma_{XX}$? How do we get from that particular $k$ in the middle Subfigure to the $f(x)$ in the right Subfigure.

My guess: One goes through $k(\mathrm{dark~pink}, h)$, where $h \in \{ \mathrm{hues}\})$ which gives $f(x)$ which is a 10-dimensional point along all axes spanned. So the 3rd Subfigure is the representation of this 10-dimensional point rather than the $f(x)$ of all the different hues.

Later in the article, conditionals are introduced: enter image description here The noise-free and centered prediction with a conditional on the training data $Y$ would be $$X∣Y​∼\mathcal{N}(Σ_{XY}​ Σ_{YY}^{-1}​Y​,Σ_{XX}​-Σ_{XY}​Σ_{YY}^{-1}Σ_{YX}​)$$ where I am not sure how the different parts belong to the Figure. $Σ_{XY}$ is probably the pink-orange kernel (basically the $10 \times 2$ and $2 \times 10$ off-blockdiagonals.) Conditioning on one of the points in $Y$ we yield another 12-dimensional (?) point, $f(x)$.

It is completely lost on me what the arrow in the last Subfigure means (but I'm sure it does, since it is on distill.pub and they don't just throw in random graphics, usually.)

Thanks for the help and bearing with me.

  • $\begingroup$ I am afraid that your question may be too broad for site like this. You seem to ask us (1) explain you what GP's are, (2) go through the article almost line by line, commenting different parts of it. Could you make a specific question out of this? Otherwise it seems someone would have to write parallel article explaining GP's and commenting the distill... $\endgroup$ – Tim Apr 4 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.