3
$\begingroup$

In Gaussian Process (GP), the kernel (co-variance function) is used to measure the similarity between one point and a given point. There are so many kernel functions for GP, and I wonder how to select a suitable kernel. For instance, if my time-series data are not periodic, should I choose the Squared Exponential (SE) kernel?

In addition, could anyone explain why the SE kernel is so popular as well? what is the feature of this kernel?

Thank you for your help in advance.

$\endgroup$
2
$\begingroup$

One possibility you might try is simulating Gaussian Processes with different kernels. In that way, you can get a feel for what the different kernels will produce. This can most easily be done by selecting a grid of values and simulating from the multivariate normal implied by that grid. To make things easier, just use a zero vector for your mean function. You can also see with this method if the properties of your simulated draws tend to match up with how your time series data looks.

For example, you will see that the squared exponential kernel is very smooth. In fact, draws from a Gaussian Process with a squared exponential kernel will be continuous with probability one and also in fact infinitely differentiable with probability one. This is one property of the squared exponential that makes it very useful. Another reason for why it gets a lot of use is its clear connection with a Gaussian density.

Other kernels such as the Ornstein–Uhlenbeck covariance function will produce much rougher draws and may be more desirable in terms of a model.

$\endgroup$
  • $\begingroup$ Thank you very much for your reply. However..I am still a little confused on:1) there are almost infinite number of kernels, so it looks impossible to try them one by one. Is there a general guideline to initially select a suitable range of kernels (for instance...periodic/non-periodic...)? 2) I am still confused on the SE kernel. Please could you say more details on the SE kernel? and would you mind to explain why 'continuous with probability one' and 'infinitely differentiable with probability' are great advantages? Thank you. $\endgroup$ – kakanana Aug 2 '14 at 7:25
  • $\begingroup$ You're welcome. For 1) I don't have any authoritative advice on selecting kernels in general besides the advice of plotting a few and seeing how they line up with properties of the data you're interested in, e.g. sample autocorrelation, etc. I do believe this is a good exercise in general and should give you a feel for the different kernels. Just select some of the most popular ones or the ones you are specifically interested in and it shouldn't take more than an afternoon to code it all up. $\endgroup$ – Samuel Benidt Aug 2 '14 at 17:51
  • $\begingroup$ In regards to the periodicity issue, there are such things as periodic kernels, which you may find useful for modeling periodic time series. en.wikipedia.org/wiki/… For 2) continuous and infinitely differentiable are more analytic properties that can be useful in derivations. In terms of modeling, you can interpret these properties in layman's terms as the curves will be very smooth, which may or may not describe the time series data you are interested in. $\endgroup$ – Samuel Benidt Aug 2 '14 at 17:58
  • $\begingroup$ thank you for your comment. Just one more question....I am trying to use Gaussian Process Regression to make predictions for the missing data of my time-series data. I want to select the SE kernel (not necessary to create my own kernel) but I must explain the reason why I chose this kernel in the report... This is the issue which has confused me for a long time. There are a number of kernels, I am not sure how to evaluate which kernel suits my data best... Thank you for your patience. $\endgroup$ – kakanana Aug 2 '14 at 20:54
  • $\begingroup$ Here's a good way that you might justify a kernel choice in a report. First - fit 2 or 3 different kernel functions that you might think are reasonable. Second -calculate test statistics of interest such as sample autocovariance at different distances on the original data. Next - simulate time series data repeatedly from each of the fitted models. Apply the same test statistics to the simulated data. Make histograms of the test statistics and see where the test statistics for the original data lie in the plot (or calculate some posterior predictive p-value) Finally-write up the results. $\endgroup$ – Samuel Benidt Aug 3 '14 at 1:23
0
$\begingroup$

Set aside a second set of training data, and "train" your model architecture using that.

i.e. 1) select an arbitrary kernel 2) train it using training set 1 3) evaluate it on training set 2 (using accuracy, precision, recall, whatever) 4) if !tired: goto 1) 5) else: return kernel with highest evaluation score from step 3)

It would probably make sense to start with "simple" kernels, and gradually try more complicated ones. The simple models will perform nominally on training set 2. As the kernel get more complicated, the model will start to perform better. As the kernel gets insanely complicated, the model will perform worse on training set 2, as the insanely complicated model starts overfitting. This is is good time to stop.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.