30
$\begingroup$

I have read that using R-squared for time series is not appropriate because in a time series context (I know that there are other contexts) R-squared is no longer unique. Why is this? I tried to look this up, but I did not find anything. Typically I do not place much value in R-squared (or Adjusted R-Squared) when I evaluate my models, but a lot of my colleagues (i.e. Business Majors) are absolutely in love with R-Squared and I want to be able to explain to them why R-Squared in not appropriate in the context of time series.

$\endgroup$
3
  • 3
    $\begingroup$ Google search: "spurious regression in econometrics". Or check out Granger and Newbold's paper. Others may supply more details in answers. $\endgroup$ Commented Jun 8, 2014 at 1:18
  • 1
    $\begingroup$ @Richard Hardy could you please elaborate on "If we take sample R2 as a measure of its population counterpart, it breaks down under integrated time series.". $\endgroup$ Commented Jun 13, 2019 at 20:47
  • $\begingroup$ Depending on the order of integration, population or true $R^2$ might not even be defined. Also, autocorrelation leads to spurious $R^2$ $\endgroup$
    – Firebug
    Commented Sep 22, 2021 at 11:45

4 Answers 4

25
$\begingroup$

Some aspects of the issue:

If somebody gives us a vector of numbers $\mathbf y$ and a conformable matrix of numbers $\mathbf X$, we do not need to know what is the relation between them to execute some estimation algebra, treating $y$ as the dependent variable. The algebra will result, irrespective of whether these numbers represent cross-sectional or time series or panel data, or of whether the matrix $\mathbf X$ contains lagged values of $y$ etc.

The fundamental definition of the coefficient of determination $R^2$ is

$$R^2 = 1 - \frac {SS_{res}}{SS_{tot}}$$

where $SS_{res}$ is the sum of squared residuals from some estimation procedure, and $SS_{tot}$ is the sum of squared deviations of the dependent variable from its sample mean.

Combining, the $R^2$ will always be uniquely calculated, for a specific data sample, a specific formulation of the relation between the variables, and a specific estimation procedure, subject only to the condition that the estimation procedure is such that it provides point estimates of the unknown quantities involved (and hence point estimates of the dependent variable, and hence point estimates of the residuals). If any of these three aspects change, the arithmetic value of $R^2$ will in general change -but this holds for any type of data, not just time-series.

So the issue with $R^2$ and time-series, is not whether it is "unique" or not (since most estimation procedures for time-series data provide point estimates). The issue is whether the "usual" time series specification framework is technically friendly for the $R^2$, and whether $R^2$ provides some useful information.

The interpretation of $R^2$ as "proportion of dependent variable variance explained" depends critically on the residuals adding up to zero. In the context of linear regression (on whatever kind of data), and of Ordinary Least Squares estimation, this is guaranteed only if the specification includes a constant term in the regressor matrix (a "drift" in time-series terminology). In autoregressive time-series models, a drift is in many cases not included.

More generally, when we are faced with time-series data, "automatically" we start thinking about how the time-series will evolve into the future. So we tend to evaluate a time-series model based more on how well it predicts future values, than how well it fits past values. But the $R^2$ mainly reflects the latter, not the former. The well-known fact that $R^2$ is non-decreasing in the number of regressors means that we can obtain a perfect fit by keeping adding regressors (any regressors, i.e. any series' of numbers, perhaps totally unrelated conceptually to the dependent variable). Experience shows that a perfect fit obtained thus, will also give abysmal predictions outside the sample.

Intuitively, this perhaps counter-intuitive trade-off happens because by capturing the whole variability of the dependent variable into an estimated equation, we turn unsystematic variability into systematic one, as regards prediction (here, "unsystematic" should be understood relative to our knowledge -from a purely deterministic philosophical point of view, there is no such thing as "unsystematic variability". But to the degree that our limited knowledge forces us to treat some variability as "unsystematic", then the attempt to nevertheless turn it into a systematic component, brings prediction disaster).

In fact this is perhaps the most convincing way to show somebody why $R^2$ should not be the main diagnostic/evaluation tool when dealing with time series: increase the number of regressors up to a point where $R^2\approx 1$. Then take the estimated equation and try to predict the future values of the dependent variable.

$\endgroup$
3
  • $\begingroup$ Good explanation but then why this this is added as a standard output of software in statistical package $\endgroup$
    – user86409
    Commented Aug 20, 2015 at 11:49
  • $\begingroup$ @brijesh Regression-tradition, I would say. $\endgroup$ Commented Aug 20, 2015 at 13:12
  • 5
    $\begingroup$ Great answer! However, it contains little information that is particular to time series. Prediction vs. in-sample fit applies to other data types probably as much as for time series. On the other hand, one key aspect that is particular to time series is missing. I mean regressing integrated variables. If we take sample $R^2$ as a measure of its population counterpart, it breaks down under integrated time series. (I could write this up as an answer but do not have the time right now.) $\endgroup$ Commented Apr 24, 2018 at 9:17
3
$\begingroup$

Some extra comments to the post above. When dealing with time-series an R squared (or adjusted R^2) would always be greater if explanatory variables were not differenced. However, when it goes to out-of-time fit, the error term would be significantly higher for non-differenced time series. This happens because of trends presented in the data and generally well-known issue. But it is a good way showing why this measure should probably, be last on the list when choosing most appropriate time-series model.

$\endgroup$
0
1
$\begingroup$

In a basic linear regression, there are several equivalent formulations of $R^2$.

  1. Squared Pearson correlation between the feature ($x$) and outcome ($y$).

  2. Squared Pearson correlation between the predicted ($\hat y$) and true ($y$) values.

  3. Comparison of the square loss incurred by the model to the square loss incurred by a baseline model that always predicts the overall mean, $\bar y$.

This last one can be expressed as:

$$ R^2=\dfrac{\text{Baseline Square Loss} - \text{Model Square Loss}}{\text{Baseline Square Loss}}$$$$=1-\dfrac{\text{Model Square Loss}}{\text{Baseline Square Loss}}$$$$= 1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

In a time series model, it is not necessarily the case that all of these are equal of that all even make sense. Consequently, it is not obvious which formulation of $R^2$ is being criticized. I will tackle each of these.

  1. It is not obvious what would go into such a calculation. If you have one lag in an $AR(1)$ model, perhaps you could calculate $\text{cor}\left(x_{t-1}, x_t\right)$, but your time series model might not even be $AR$, let alone $AR(1)$, and then this stops making sense for about the same reason that it does not make sense for a linear regression with two features.

  2. This misses many differences that can be present. For instance, the time series $\left(1, 2, 3, 4, 5\right)$ has perfect correlation with $\left(11, 21, 31, 41, 51\right)$. Does either seem like a good set of predictions of the other?

  3. This makes sense to me. You have some kind of model performance. Give it context. How does it compare to a simple benchmark of always predicting the same value, regardless of the features? This is analogous to comparing the performance of a classifier to the performance achieved by always predicting the majority category (reduction in error rate compared to this naïve guessing).

The criticism I might have of $R^2$ in time series predictions, even for this last formulation of $R^2$, is that the benchmark is not very good and gives a lot of room for improvement (e.g., integrated time series, discussed in other answers and in comments) that makes for an $R^2$ value of $0.99$ when the predictions might leave a lot to be desired.

(Really, this is not a unique criticism by time series predictions. In "regular" regression, naïvely predicting $\bar y$ might be too naïve and give an inflated sense of model performance. The calculation always means exactly what it means, but it is easy to see a value like $0.99$ and think your model gets an $\text{A}$-grade, rather than thinking critically about how good your model predictions are, such as if a simple benchmark would have achieved a value like $0.98$.)

A second criticism is that $R^2$ is, often, an in-sample measure of performance, while time series predictions often want to predict the future. I see reasonable ways to give an out-of-sample $R^2$, both for "regular" regression and for time series forecasts, though you have to know the appropriate modification. Upon making appropriate modifications, however, such as the linked calculation of a time series $R^2$, taken from a publication in the Review of Financial Studies (one of the elite journals in academic finance), such an $R^2$-style calculation, which follows the philosophy of comparing to a benchmark despite the details having some differences from the "regular" regression $R^2$, could make a lot of sense.

(I have seen several articles that discuss how to determine appropriate benchmarks in time series forecasting. As I come across them, I may come back to edit in citations. I believe Rob Hyndman published a few of them.)

$\endgroup$
1
  • $\begingroup$ +1 I get a really bad feeling when people don't use separate test data to evaluate their model, with whatever metric you want. $\endgroup$
    – Ggjj11
    Commented Apr 16 at 17:33
0
$\begingroup$

@Richard Hardy, I agree with you. To my experience the test performance is not the main point why the R2 may be misleadingly good (it should be obvious that we need to use hold out test data for a reliable performance estimation). Instead, evaluating the $R^2 = 1 - \frac {SS_{res}}{SS_{tot}}$ with residuals from multiple time points in an integrated time series (and not treating these multiple times points separately) lets the R2 value look much too good in many applications.

You can easily envision a model having predictions catching up on a trend, so that the model appears to have $R^2$ close to 1 (when evaluated on all residuals, because the residuals are smaller than only predicting the global mean of the samples). It is obvious that computing the R^2 on all the residuals of a time series with trend may easily really good (because you are better than the overall mean if you are able to predict some trend). However taking the conditional mean value of training outcomes at timepoints t could even beat your model easily (also on holdout test data).

To solve this, people sometimes come up with computing rolling R2 (and a certain window size in time) on hold out test data to better reflect performance over time (e.g. here in finance a rolling R2 with window size 5 years was used to identify time periods where the model was (locally) good or bad:

https://books.google.de/books?id=SxVBDwAAQBAJ&pg=PA39

The window size is obviously a parameter of your evaluation and must be chosen carefully to reflect the aspects you are interested in.

@Constantin: While you point to the main issue I do not share the general advise to not have a look at the R2. It may just be that you need to look at rolling R2 on holdout test data and if you only want one number, you might aggregate e.g. via the min median, mean, std (for "stability" checks) or getting the last entry ... of the rolling R^2 time series (depending on your goals)

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.