6
$\begingroup$

Scikit learn allows us to fit Gaussian processes $GP(0,K(.,.))$ such that $K:T\times T \to \mathbb{R}$ is a covariance function (kernel), however it doesn't let us specify a mean function $m: T\times \mathbb{R}$. I'm trying to work solely with scikit-learn, so working with only the tools it provides. Is it a reasonable approach to fit a mean function using another method such as the multiple linear regression and then fit a Gaussian process to the residuals?

Should I do something like Iterated Weighted Least Squares (IWLS) where you alternate between fitting the fixed effects under the current estimate of the covariance matrix and then reestimate the covariance function parameters using the current residuals?

$\endgroup$
4
  • 1
    $\begingroup$ I'm not very familiar with scikit-learn, but if you can specify the covariance function as the sum of a linear covariance function and any other (e.g. matern, sq exp), this is equivalent to adding in a mean function $\endgroup$
    – jcken
    Apr 6, 2023 at 11:04
  • $\begingroup$ @jcken sklearn does have a dot-product linear kernel. That sounds exactly like what I want. Could you elaborate a little bit on the equivalence of the linear covariance and the mean function? I know that the sum of p.d. kernels are also p.d., but I don't see how the equivalence works out. $\endgroup$ Apr 6, 2023 at 19:33
  • 1
    $\begingroup$ Thanks, I had the same question. Chris Fonnesbeck wrote an article about Scikit-Learn's GPR module in 2017, in which he states "The GaussianProcessRegressor does not allow for the specification of the mean function, always assuming it to be the zero function, highlighting the diminished role of the mean function in calculating the posterior." So unless something has changed since then it looks like the answer is 'no'. $\endgroup$
    – Bill
    Sep 2, 2023 at 18:42
  • $\begingroup$ @Bill I guessed so. Thanks for the additional info. Looking at the theory, the universal kriging predictor (BLUP under a GP) is just the ordinary kriging predictor over the residuals of fitting said linear model. So fitting, for example, a parametric or non-parametric (splines, etc) linear model then using the residuals to fit and predict seems fine. $\endgroup$ Sep 4, 2023 at 16:02

1 Answer 1

1
$\begingroup$

Usually mean function is not of your greatest interest when using Gaussian Processes. If you care about it, it can be done within the GP model, as discussed for example here. If your scikit-learn does not support non-zero mean functions, you can simply use some model to find the mean, subtract if from the data, and fit GP to the de-meaned data. There is rather no need to do it iteratively.

$\endgroup$
1
  • 1
    $\begingroup$ This is only true when you have sufficient data. One of the potential advantages of Gaussian process regression is to make predictions with few (or even no) data when you want to rely on an explicit basis function, which you may have from prior knowledge or first principles. $\endgroup$
    – Bill
    Sep 2, 2023 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.