19
$\begingroup$

I have just started reading about GPs and analogous to the regular Gaussian distribution it is characterized by a mean function and the covariance function or the kernel. I was at a talk and the speaker said that the mean function is usually quite uninteresting and all the inference effort is spent on estimating the correct covariance function.

Can someone explain to me why that should be the case?

$\endgroup$
24
$\begingroup$

I think I know what the speaker was getting at. Personally I don't completely agree with her/him, and there's a lot of people who don't. But to be fair, there are also many who do :) First of all, note that specifying the covariance function (kernel) implies specifying a prior distribution over functions. Just by changing the kernel, the realizations of the Gaussian Process change drastically, from the very smooth, infinitely differentiable, functions generated by the Squared Exponential kernel

enter image description here

to the "spiky", nondifferentiable functions corresponding to an Exponential kernel (or Matern kernel with $\nu=1/2$)

enter image description here

Another way to see it is to write the predictive mean (the mean of the Gaussian Process predictions, obtained by conditioning the GP on the training points) in a test point $x^*$, in the simplest case of a zero mean function:

$$y^*=\mathbf{k}^{*T}(K+\sigma^{2}I)^{-1}\mathbf{y}$$

where $\mathbf{k}^*$ is the vector of covariances between the test point $x^*$ and the training points $x_1,\ldots,x_n$, $K$ is the covariance matrix of the training points, $\sigma$ is the noise term (just set $\sigma=0$ if your lecture concerned noise-free predictions, i.e., Gaussian Process interpolation), and $\mathbf{y}=(y_1,\ldots,y_n)$ is the vector of observations in the training set. As you can see, even if the mean of the GP prior is zero, the predictive mean is not zero at all, and depending on the kernel and on the number of training points, it can be a very flexible model, able to learn extremely complex patterns.

More generally, it's the kernel which defines the generalization properties of the GP. Some kernels have the universal approximation property, i.e., they are in principle capable to approximate any continuous function on a compact subset, to any prespecified maximum tolerance, given enough training points.

Then, why should you care at all about the mean function? First of all, a simple mean function (a linear or orthogonal polynomial one) makes the model much more interpretable, and this advantage must not be underestimated for model as flexible (thus, complicated) as the GP. Secondly, in some way the zero mean (or, for what's worth, also the constant mean) GP kind of sucks at prediction far away from the training data. Many stationary kernels (except the periodic kernels) are such that $k(x_i-x^*) \to 0 $ for $\operatorname{dist}(x_i,x^*)\to\infty$. This convergence to 0 can happen surprisingly quickly, expecially with the Squared Exponential kernel, and particularly when a short correlation length is necessary to fit the training set well. Thus a GP with zero mean function will invariably predict $y^*\approx 0$ as soon as you get away from the training set.

Now, this could make sense in your application: after all, it is often a bad idea to use a data-driven model to perform predictions away from the set of data points used to train the model. See here for many interesting and fun examples of why this can be a bad idea. In this respect, the zero mean GP, which always converges to 0 away from the training set, is safer than a model (such as for example an high degree multivariate orthogonal polynomial model), which will happily shoot out insanely large predictions as soon as you get away from the training data.

In other cases, however, you may want your model to have a certain asympotic behavior, which is not to converge to a constant. Maybe physical consideration tell you that for $x^*$ sufficiently large, your model must become linear. In that case you want a linear mean function. In general, when the global properties of the model are of interest for your application, then you have to pay attention to the choice of the mean function. When you are interested only in the local (close to the training points) behavior of your model, then a zero or constant mean GP may be more than enough.

$\endgroup$
  • $\begingroup$ Delta, do you know what would be a good mean function? $\endgroup$ – An old man in the sea. Nov 5 '18 at 16:09
  • $\begingroup$ @Anoldmaninthesea it depends a lot on the application. As I explained, unless you need an interpretable model, or you're interested in predictions "far away" from your training set, it would be probably better to concentrate your efforts on improving the covariance function, rather than the mean function $\endgroup$ – DeltaIV Nov 5 '18 at 17:36
  • 1
    $\begingroup$ Delta, well in my case I need to try to make some predictions which may be far away from the observed data... I've asked this question here stats.stackexchange.com/questions/375468/… $\endgroup$ – An old man in the sea. Nov 5 '18 at 17:42
  • $\begingroup$ @Anoldmaninthesea ok. I don't have time today, but tomorrow I'll have a look at your question and attempt an answer. $\endgroup$ – DeltaIV Nov 5 '18 at 17:46
5
$\begingroup$

We can't speak on behalf of the person who was giving the lecture; perhaps the speaker had a different idea in mind when the speaker made that statement. However, in the case that you are attempting to construct posterior predictions from a GP, a constant mean function has a closed-form solution that can be computed exactly. However, in the case of a more general mean function, you must resort to approximate methods, e.g. simulation.

Additionally, the covariance function controls how rapidly (and where) deviations from the mean function occur, so it's often the case that a more flexible/rigid covariance function can be "good enough" to approximating a more ornate mean function -- which again grants access to the convenience properties of a constant mean function.

$\endgroup$
  • $\begingroup$ Thanks for that explanation. Yeah, I could not ask my question and was wondering if there is a principled reason for this. $\endgroup$ – Luca Jul 5 '16 at 14:26
5
$\begingroup$

I'll give you an explanation that probably wasn't meant by the speaker. In some applications means are always boring. For instance, let's say we're forecasting sales with autoregressive model $y_t=c+\gamma y_{t-1}+e_t$. The long run mean is obviously $E[y_t]\equiv\mu=\frac{c}{1-\gamma}$. Is it interesting?

It depends on your objective. If you're after the store valuation, then it tells you that you must increase $c$ or decrease $\gamma$ to increase the value of the store because the value is given by: $$V=\frac{\mu}{r}$$ where $r$ is the discount factor. So, the mean is clearly interesting.

If you're interested in the liquidity, i.e. you have enough cash to cover expenses in next couple of months, then the mean is almost irrelevant. You're looking at next month's cash forecast: $$y_1=c+\gamma y_0$$ So this month's sales $y_0$ are a factor now.

$\endgroup$
5
$\begingroup$

Well one very good reason is that the mean function may not live in the the space of functions you wish to model. each input point, $x_i$, may have a corresponding posterior mean, $\mu(x_i)$. However, these posterior mean points are the expectation before you see any other data. So there are many cases where no situation where the future data observed will create that mean function.

Simple example: Imagine fitting a sine function with unknown offset but known period and amplitude one. The prior mean is zero for all $x$ but a constant line does not live in the space of sine functions we described. The covariance function gives us that additional structural information.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.