How to test if the process that generated a time-series has changed over time

Question

Problem

I have time-series data generated by a machine over two disjoint periods of time - roughly one month in 2016 and another month in 2018.

It is hypothesized by domain experts that at each time step $t$, an observed variable $Y^t$ can be explained by another set of observed variables, $X_1^t, \ldots, X_d^t$.

How can I test whether this process has changed over time? Note that I am not trying to test if the distribution for the variable $Y$ has changed over time. I want to test if the relationship between the $X_i$s and $Y$ has changed over time.

Current approach

Suppose I fit a time-series model (e.g., a Gaussian Process) on the data from 2016 to predict $Y^t$ given $X_1^t, \ldots, X_d^t$ as a way to model the underlying process that generated $Y^t$.

The domain experts have suggested that maybe we can try to use this model to predict the variables $Y^t$s given the $X^t$s from 2018 and use the residuals somehow to infer that the model (representing the process in 2016) is or is no longer the same in 2018. I am uncertain how to continue after this point.

What I'm considering

1. Should I test if the residues from 2016 and 2018 are generated from the same distribution, or perform a goodness of fit test using something like Kolmogorov-Smirnov test? My concern with this approach is that the out-of-sample data from 2018 is likely to have larger errors than the in-sample training data from 2016, so this test will likely give rise to false positives. Is there any way to adjust/account for this effect?

2. Should I fit two models, one for 2016 and another for 2018, and use some way to test that these two models are "same" or "different"? For e.g., one possibility is to compute the KL divergence between the 2 Gaussian Processes fitted respectively on the 2016 and 2018 data. Are there any other suggestions or problem with this approach?

3. I saw some posts on cointegration. But I do not fully understand this concept. Is this relevant?

In general, how might one approach this type of problem? I've tried searching for this online, but maybe due to a lack of precision of my query (I'm not familiar in this area), I'm not getting many relevant results. I appreciate even simple hints/comments on the topic(s)/keywords to search, or books/papers to look through.

Kindly note that I am looking for principled (preferably statistical) approaches and not methods based on heuristics. Good examples are the answers suggesting the Chow test and its variants below.

Answer 1

Structural change can be tested not only with Chow test mentioned by @John Stax Jakobsen.

There are plenty other tests, especially family of fluctuation tests usually works well.

Here you have nice introduction to R package strucchange that computes them. If you're not R user, read just theory, it is well described there.

Answer 2

If it is reasonable to model the relationship with a linear regression, then an easy way to test for a structural break is the chow test.

see the wiki article here

Answer 3

First, I would just fit some black box models (e.g. GBM or random forest) which directly take into account the time variable $T$, e.g. $Y_t=F(X_t^1, \ldots, X_d^t; T)$. It might be helpful to test various granularity of $T$, such as measured in calendar years (2016, 2018), months passed since 2016, etc. Then, in order to assess the importance of $T$ one could either look at variable importance plots (see e.g. Section “15.3.2 Variable Importance” in Elements of Statistical Learning) or simply drop the $T$ variable, refit the model and compare the model performance.

Alternatively, you could stick to your model (Gaussian process) and compare the 2016 and 2018 residuals. I agree with your intuition that the comparison of distribution of in-sample (2016) and out-of-sample (2018) residuals would get misleading results. However, this can be fixed quickly by partitioning your data as follows: split 2016 data into training subset (used to fit the model) and validation subset (used to assess the quality of your model), also define the second validation dataset using subset of 2018 data. Then, just fit your model using training subset and test the performance (calculate residuals, MSE, etc) on two validation subsets (2016 and 2018). In order to rule out chance (your result might differ just because of bad luck) you might want to repeat the whole exercise (splitting data, fitting model, assessing performance on validation datasets) several times.

Also, as you mentioned, you could fit two different models (one based on 2016 data, another based only on 2018 data). In this case I would also split the data for each year into training and validation subsets and assess the model performance based on the validation subsets. As a measure of similarity you could use: RMSE, QQ-plots, statistical tests you mentioned or calculating the confidence intervals for predictions coming from both models and checking whether the confidence intervals overlap.

Answer 4

One of the features that I put into my favorite forecasting package was the CHOW Test to investigate that breakpoint in parameters that was the most significant. Before I did that I had to treat/adjust for pulses/seasonal pulses.

Of course if there are identified level Shifts or Time Trends this test is bypassed.

The CHOW Test premises independent errors with in each group as it is required under the F test that he uses.

My implementation includes the possibility of contemporary and/or lagged user specifed causals within the GLM.