I am reading a textbook Gaussian Process for Machine Learning by C.E. Rasmussen and C.K.I. Williams and I am having some trouble understanding what does distribution over functions mean. In the textbook, an example is given, that one should imagine a function as a very long vector (in fact, it should be infinitely long?). So I imagine a distribution over functions to be a probability distribution drawn "above" such vector values. Would it then be a probability that a function will take this particular value? Or would it be a probability that a function will take a value that is in a given range? Or is distribution over functions a probability assigned to a whole function?
Quotes from the textbook:
Chapter 1: Introduction, page 2
A Gaussian process is a generalization of the Gaussian probability distribution. Whereas a probability distribution describes random variables which are scalars or vectors (for multivariate distributions), a stochastic process governs the properties of functions. Leaving mathematical sophistication aside, one can loosely think of a function as a very long vector, each entry in the vector specifying the function value f(x) at a particular input x. It turns out, that although this idea is a little naive, it is surprisingly close what we need. Indeed, the question of how we deal computationally with these infinite dimensional objects has the most pleasant resolution imaginable: if you ask only for the properties of the function at a finite number of points, then inference in the Gaussian process will give you the same answer if you ignore the infinitely many other points, as if you would have taken them all into account!
Chapter 2: Regression, page 7
There are several ways to interpret Gaussian process (GP) regression models. One can think of a Gaussian process as defining a distribution over functions, and inference taking place directly in the space of functions, the function-space view.
From the initial question:
I made this conceptual picture to try to visualize this for myself. I am not sure if such explanation that I made for myself is correct.
After the update:
After the answer of Gijs I updated the picture to be conceptually more something like this: