Gaussian Process goodness of fit Let's say I got a Gaussian Process model $M$ based on some training data. Now I get a stream of sample data of a certain batch size coming in.
The GP does not model a time series, but it's trying to regress the value at certain locations $x$, that will be visited mutliple times.
I know that at some point there will be an abrupt change in the distribution the data batches are generated from. (At least on certain locations $x$)
Now I'm looking for a statistically sound way of detecting this change, that is I want to find the point, when the GP doesn't model the data well enough anymore.
I thought I would base this on detecting "big" changes in the likelihood function $L(\theta | x)$. However I'm not sure how to interpret "big" here, as the values of the likelihood function only have a meaning in comparison, but on their own.
Note: I'm asking this, because some people said my original question was too abstract. I didn't want to change the whole question, though, as there already were some answers.
 A: What do you think would cause the change? Is it a change in the mean, the variance or something else? The type of change you expect should determine the parameter you should test for change. Is this a sudden change? If so this could be done using intervention analysis for time series such as is done with Box Jenkins models for statiomary Gaussian time series. The autobox software can handle this for you.
A: So following is the solution I came up with. Please correct me if I'm wrong :)
Assumptions
The model change is abrupt.
Idea
My idea was the following: We determine the goodness of the fit by means of a model comparison. So we create a very simple model for which we know how to calculate the likelihood. As soon as this models becomes more likely than the original GP model we assume there was a model change.
We assume the data $D$ is normalized to have a zero mean.
null model likelihood
As a simplification of the GP we choose a normal distribution with zero mean and a large variance at each test location.
$\log p(D|M_0) = \sum\limits_{d \in D}\log p(d_i|M_0)$ with $d_i = (x_i, y_i)$
$\log p(d|M_0) = -\frac{1}{2}\log(2\pi)-\frac{1}{2}\log(\sigma^2)-\frac{1}{2\sigma^2}x_i^2$ 
GP model likelihood
For a GP with covariance matrix $K_y$ the likelihood is given by:
$\log p(D|M_{GP}) = -\frac{1}{2}\mathbf{y}^\top K_y^{-1} \mathbf{y} -\frac{1}{2}\log |K_y| -\frac{n}{2}\log{2\pi}$
Making a decision based on the Bayes factor
Assuming a uniform prior over the models, we calculate the log Bayes factor as follows:
$\log B_{01} = \log p(D|M_0) - \log p(D|M_{GP})$
According to this paper if $\log B_{01}$ is greater than 2 we assume there was a change in the underlying model.
