Gaussian Process Regression with positive prediction weights

Question

I want to do Gaussian Process Regression for a density function $f(x)$ with a gaussian kernel function $k(x, x')$.

Given the training data $\mathbf{x} = (x_1, x_2, \dots, x_N)$ and $\mathbf{f} = (f(x_1), f(x_2), \dots, f(x_N))$, the prediction at a new point $x^*$ is: $$ f(x^*) = \mathbf{k}^T \mathbf{C}_N^{-1}\mathbf{f} $$ where $$ \mathbf{k} = \begin{bmatrix} k(x_1, x^*)\\ \vdots\\ k(x_N, x^*) \end{bmatrix} \mathbf{C}_N = \begin{bmatrix} k(x_1, x_1) &\dots &k(x_1, x_N)\\ &\vdots\\ k(x_N, x_1) &\dots &k(x_N, x_N) \end{bmatrix} $$ As mentioned in Bishop's machine learning book, I can rewrite $$ f(x^*) = \sum_{i = 1}^N a_i k(x_i, x^*) $$ with $a_i$ is the $i^{th}$ component of $\mathbf{C}_N^{-1}\mathbf{f}$.

As I use an gaussian kernel $k$, I can directly sample my density function, if all $a_i$ are positive.

My questions is if there exists a way to constraint positive $a_i$ in the GP regression?

Answer 1

I see a practical (read: quick and dirty) way to approach this. But in general Gaussian processes seem to be bad models for densities. Gaussian processes at training and prediction points are multivariate normal random variables, which means they cannot be positive with certainty.

If you are only interested in the mean function, a quick and dirty way to get around this might be to impose a non-zero prior mean. This might lift your mean function in areas where you have little data above zero. If this makes sense or not depends on the nature of your problem. You might also want to use the constraint that the paths integrate to one. This constraint can be implemented in a clean way (see Chapter 9.4. of Rasmussen, Williams, where they discuss derivatives but integrals are the same in spirit)

Depending on what you need an even better approach might be to regress on $\log(f)$ instead of $f$.

Answer 2

Yes - provided you are will to stop learning. The biggest bottle neck of GPs is the inversion of K (the covariance matrix) which is order $O(n^3)$. But if you stop learning and keep $K^{-1}$ somewhere handy you can reuse it for multiple predictions and as such your $a$ is a constant. Unfortunately most people want systems that keep learning with new observations. In this case $a$ will change on each update.