I don't quite get why samples from prior in Gaussian Processes look like they do. I build covariance matrix, for example using radial basis kernel or a polynomial one, then I draw samples from multivariate normal Gaussian with mean $0$ and covariance that I calculate and the result is smooth, look like a continuous function. I don't get why. My intuition, that is obviously wrong, would tell me that there will be some discontinuities. Similarly with periodic kernel, I can visualize the covariance function using some heatmap, I can see the periodic pattern, but I don't understand why when I plug that into a multivariate Gaussian it yields periodic functions in return. I would be grateful for any help.
5
-
$\begingroup$ Why shouldn't it be smooth? It might be easier to answer if you explain us why is it counterintuitive for you? $\endgroup$– TimCommented Oct 25, 2021 at 8:29
-
$\begingroup$ What's counterintuitive to me: I want to draw a sample from the prior, sample consisting of N points. So I construct N-dimensional Gaussian and then I draw one sample from it. So every point of my function that I sample corresponds to one dimension in the multivariate Gaussian, do I get it right? And I don't quite get why, when my first point has value let's say 1, my second point has value somewhere around 1 as well, why doesn't the third point have value -50 or something? $\endgroup$– gabeCommented Oct 27, 2021 at 8:20
-
$\begingroup$ I'm afraid I don't follow. Why -50? What do you mean? $\endgroup$– TimCommented Oct 27, 2021 at 8:23
-
$\begingroup$ Let's day I have normal Gaussian. one dimensional. I draw one sample, let's say it's positive, then the second sample doesn't have to be positive as well, it can be negative. It doesn't have to be anywhere near the dirst sample. So why in GPs value of the first point is kind of close to the second point, which in turn is close to third point, etc. making the function continuous. This is what I don't get. $\endgroup$– gabeCommented Oct 27, 2021 at 9:27
-
$\begingroup$ Hi @gabe, I am struggling to grasp this intuitively as well. I do not understand what makes the samples (which is just a random sample from multivariate normal) lie in such a smooth fashion when I plot them. Could you kindly share if you now have a better intuition? $\endgroup$– swag2198Commented May 15, 2022 at 1:22
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The covariance function encodes prior beliefs about the nature of the function. It basically says how similar the output of the Gaussian process should be as a function of the input features. If you have a covariance function (such as the RBF or polynomial covariance functions) that give a high value if the input features are very similar, then that means you will get a smooth function. The output values for similar input vectors will be similar because that is what the covariance function specifies.
If you want a non-smooth function, I think the Matern covariance may be better. See also this related question.
answered Oct 25, 2021 at 8:32