1
$\begingroup$

I have the following linear model based on multivariate timeseries:

lm(formula = Y ~ X1 + X2 + X3 + X4 + X5 + X6 + X7, data = model.data, na.action = na.omit)

Residuals:
       Min         1Q     Median         3Q        Max 
-0.0030132 -0.0002101  0.0000004  0.0002101  0.0035819 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -2.355e-06  3.813e-06  -0.618  0.53683    
X1           4.322e-02  6.964e-03   6.206 5.49e-10 ***
X2          -5.200e-02  8.202e-03  -6.340 2.32e-10 ***
X3          -3.222e-02  8.182e-03  -3.939 8.21e-05 ***
X4           9.367e-07  2.016e-07   4.647 3.37e-06 ***
X5           5.131e-02  9.980e-03   5.141 2.74e-07 ***
X6           2.911e-02  1.005e-02   2.897  0.00377 ** 
X7          -1.677e-07  2.608e-08  -6.430 1.29e-10 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

Residual standard error: 0.0003898 on 35773 degrees of freedom
  (3750 observations deleted due to missingness)
Multiple R-squared: 0.006066,   Adjusted R-squared: 0.005872 
F-statistic: 31.19 on 7 and 35773 DF,  p-value: < 2.2e-16

All the IVs have a low enough p-value to be considered significant. $R^2$ however is less than 0.01.

Now let's take a look at the mean absolute error of this model. I report a "Naive Model MAE" calculated as the difference between the last observed $Y_t$ and the next observed value $Y_{t+1}$ and a "Model MAE" as the mean of the abs of the residuals.

The same Naive MAE formula is used for the out-of-sample data set. The Fcast MAE is calculated as the difference between the predicted values based on the model fit and the actual observed values in the out of sample data. Here the MAE values are multiplied by 10000 for readability.

AIC -460198.9 
Naive Model MAE 4.005496 (0.0004005496)
Model MAE 2.812995 (0.0002812995)

Naive Fcast MAE 2.436187 (0.0002436187)
Fcast MAE 1.710664  (0.0001710664)
Fcast Yt+1 Impr % 29.78

So although $R^2$ is very low, the forecast MAE relative to a naive benchmark is quite "interesting". Intuitively I am struggling to capture this relationship between $R^2$ and MAE.

On mpkitas suggestion I also ran a regression based on the naive $Y_t$ forecast to compare $R^2$, in addition to MAE:

 summary(lm(data$NextY ~ data$Y, na.action = na.omit))

Call:
lm(formula = data$NextY ~ data$Y, na.action = na.omit)

Residuals:
       Min         1Q     Median         3Q        Max 
-0.0029678 -0.0002104  0.0000016  0.0002113  0.0035384 

Coefficients:
                         Estimate Std. Error t value Pr(>|t|)
(Intercept)            -1.590e-06  2.067e-06  -0.770    0.442
Y                       3.447e-03  5.231e-03   0.659    0.510

Residual standard error: 0.000391 on 35781 degrees of freedom
  (3748 observations deleted due to missingness)
Multiple R-squared: 1.213e-05,  Adjusted R-squared: -1.581e-05 
F-statistic: 0.4341 on 1 and 35781 DF,  p-value: 0.51 

How can $R^2$ be so low if the forecast is far to be random?

$\endgroup$
  • $\begingroup$ What is the connection between the model shown here and the time series? Are any of the IVs equivalent to time or a lagged value of Y? $\endgroup$ – whuber Mar 27 '12 at 19:26
  • $\begingroup$ All IVs are at time t and regressed against $Y_{t+1}$. The IVs series are logically different from Y, that is no ARIMA or autocorrelation is used. $\endgroup$ – Robert Kubrick Mar 27 '12 at 19:40
  • 1
    $\begingroup$ So the question seems to be, you get terrible $R^2$ when regressing $Y_{t+1}$ against a bunch of $X_t$ and you're puzzled why there's a correlation between $Y_{t+1}$ and $Y_t$? What I'm missing here is the relationship between the model described in the first half of this question and the "MAE" calculation in the second half, which appears to be completely different. $\endgroup$ – whuber Mar 27 '12 at 21:04
  • 1
    $\begingroup$ I didn't check the correlation between fitted values and observations. I can calculate it and add it to the question if it helps the answer. I am puzzled about how the predicted values improve the naive forecast by almost 30%, despite the quasi-null $R^2$. About the MAE: good point! I multiply the actual MAE value by 10000 as most values are < 0.001. $\endgroup$ – Robert Kubrick Mar 28 '12 at 0:51
  • 1
    $\begingroup$ Well yes, but do you calculate the $R^2$ for naive model? $\endgroup$ – mpiktas Mar 28 '12 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.