I have just started reading about GPs and analogous to the regular Gaussian distribution it is characterized by a mean function and the covariance function or the kernel. I was at a talk and the speaker said that the mean function is usually quite uninteresting and all the inference effort is spent on estimating the correct covariance function.
Can someone explain to me why that should be the case?
I think I know what the speaker was getting at. Personally I don't completely agree with her/him, and there's a lot of people who don't. But to be fair, there are also many who do :) First of all, note that specifying the covariance function (kernel) implies specifying a prior distribution over functions. Just by changing the kernel, the realizations of the Gaussian Process change drastically, from the very smooth, infinitely differentiable, functions generated by the Squared Exponential kernel
to the "spiky", nondifferentiable functions corresponding to an Exponential kernel (or Matern kernel with $\nu=1/2$)
Another way to see it is to write the predictive mean (the mean of the Gaussian Process predictions, obtained by conditioning the GP on the training points) in a test point $x^*$, in the simplest case of a zero mean function:
$$y^*=\mathbf{k}^{*T}(K+\sigma^{2}I)^{-1}\mathbf{y}$$
where $\mathbf{k}^*$ is the vector of covariances between the test point $x^*$ and the training points $x_1,\ldots,x_n$, $K$ is the covariance matrix of the training points, $\sigma$ is the noise term (just set $\sigma=0$ if your lecture concerned noise-free predictions, i.e., Gaussian Process interpolation), and $\mathbf{y}=(y_1,\ldots,y_n)$ is the vector of observations in the training set. As you can see, even if the mean of the GP prior is zero, the predictive mean is not zero at all, and depending on the kernel and on the number of training points, it can be a very flexible model, able to learn extremely complex patterns.
More generally, it's the kernel which defines the generalization properties of the GP. Some kernels have the universal approximation property, i.e., they are in principle capable to approximate any continuous function on a compact subset, to any prespecified maximum tolerance, given enough training points.
Then, why should you care at all about the mean function? First of all, a simple mean function (a linear or orthogonal polynomial one) makes the model much more interpretable, and this advantage must not be underestimated for model as flexible (thus, complicated) as the GP. Secondly, in some way the zero mean (or, for what's worth, also the constant mean) GP kind of sucks at prediction far away from the training data. Many stationary kernels (except the periodic kernels) are such that $k(x_i-x^*) \to 0 $ for $\operatorname{dist}(x_i,x^*)\to\infty$. This convergence to 0 can happen surprisingly quickly, expecially with the Squared Exponential kernel, and particularly when a short correlation length is necessary to fit the training set well. Thus a GP with zero mean function will invariably predict $y^*\approx 0$ as soon as you get away from the training set.
Now, this could make sense in your application: after all, it is often a bad idea to use a data-driven model to perform predictions away from the set of data points used to train the model. See here for many interesting and fun examples of why this can be a bad idea. In this respect, the zero mean GP, which always converges to 0 away from the training set, is safer than a model (such as for example an high degree multivariate orthogonal polynomial model), which will happily shoot out insanely large predictions as soon as you get away from the training data.
In other cases, however, you may want your model to have a certain asympotic behavior, which is not to converge to a constant. Maybe physical consideration tell you that for $x^*$ sufficiently large, your model must become linear. In that case you want a linear mean function. In general, when the global properties of the model are of interest for your application, then you have to pay attention to the choice of the mean function. When you are interested only in the local (close to the training points) behavior of your model, then a zero or constant mean GP may be more than enough.
Well one very good reason is that the mean function may not live in the the space of functions you wish to model. each input point, $x_i$, may have a corresponding posterior mean, $\mu(x_i)$. However, these posterior mean points are the expectation before you see any other data. So there are many cases where no situation where the future data observed will create that mean function.
Simple example: Imagine fitting a sine function with unknown offset but known period and amplitude one. The prior mean is zero for all $x$ but a constant line does not live in the space of sine functions we described. The covariance function gives us that additional structural information.
I'll give you an explanation that probably wasn't meant by the speaker. In some applications means are always boring. For instance, let's say we're forecasting sales with autoregressive model $y_t=c+\gamma y_{t-1}+e_t$. The long run mean is obviously $E[y_t]\equiv\mu=\frac{c}{1-\gamma}$. Is it interesting?
It depends on your objective. If you're after the store valuation, then it tells you that you must increase $c$ or decrease $\gamma$ to increase the value of the store because the value is given by: $$V=\frac{\mu}{r}$$ where $r$ is the discount factor. So, the mean is clearly interesting.
If you're interested in the liquidity, i.e. you have enough cash to cover expenses in next couple of months, then the mean is almost irrelevant. You're looking at next month's cash forecast: $$y_1=c+\gamma y_0$$ So this month's sales $y_0$ are a factor now.
