3
$\begingroup$

Let's say I build a Generalized Least Squares model. I follow the standard procedure and first estimate a LM model. Then I create an error-response covariance matrix based on the residuals of this model. Now I build an LM model again only this time I specify weights based on the error-response covariance matrix.

Now suppose I want to predict with the GLS model out-of-sample to test for model stability. I want to confirm that I can simply perform a prediction using the coefficients estimated by GLS and there is no need to furnish weights anymore (especially since in a prediction scenario where residuals are not available the error-response covariance matrix cannot be generated).

Follow-up question:

We proceed to score on test data with coefficients from the training data. (The dimension of the test data consists of a cross-section of N individuals and T observations.) We would like to produce consistent standard errors. Therefore, instead of calculating standard error of the estimate in the OLS fashion, we weight the residuals (see below) by a "GLS weights" vector:

OLS calc of SEE: sqrt( sum( ( residuals from linear model ) ^ 2 ) ) / residualDegreeFreedom )

GLS calc of SEE: sqrt( sum( ( residuals from linear model) ^ 2 * glsWeight ) ) / sum( glsWeight ) * length( glsWeight ) / residualDegreeFreedom )

"gls weight" is a vector calculated in the usual way as the inverse of the variance of the residuals of each cross-section at a date (i.e. a vector of length T). However, here I am using the residuals from the test data as opposed to the training data (indeed this is required otherwise the dimension of out-of-time residuals would not match the dimension of GLS weights vector).

What is counter-intuitive is that if I want to measure the SEE of the GLS model out-of-sample on one individual, I am required to score all individuals out of sample (otherwise constructing the GLS weights vector would be impossible since there is no variance of residuals).

Question is - Am I required to use the GLS weights when calculating the SEE out-of-sample, or can I simply use the OLS calculation of SEE?

$\endgroup$
5
$\begingroup$

Suppose we have a GLS model:

$$y=X\beta+u,$$

with

$$Euu'=\Omega.$$

Suppose we want to predict $y^*$:

$$y^*=x^*\beta+u^*,$$

Goldberger proved that the best linear unbiased prediction for $y^*$ is the following:

$$\hat{y}=x^*\hat{\beta}+w'\Omega^{-1}\hat{u},$$

where

$$\hat\beta=(X'\Omega^{-1}X)^{-1}X\Omega^{-1}y,\quad \hat{u}=y-X\hat\beta$$

and

$$w=Eu^*u$$

So the answer to your first question would be that if you use simple prediction, then your prediction will not be optimal. On the other hand to use this formula you need to know $w$. And for that you need to know more about $\Omega$. Goldberger in his article discusses several special cases.

As for your second question it is a bit unclear for me what are you trying to achieve. The problem with GLS model is that if we use OLS standard errors of the coefficients then they are biased. The formulas you give are for calculating the standard error of the error term. But this only makes sense for OLS model, since for GLS model the error term in general will not have unique variance.

If you are going for prediction variance, then @whuber comment holds, you cannot calculate it in this setup. The basic problem for that is you predict one observation, so you get one number. And variance of one number is zero. What you can calculate is theoretical prediction variance, but this then depends on the model you are trying to test.

If you want to calculate PRESS: the sum of squares of residuals from jackknife procedure and weight them with $\Omega$, I think you will run into the same problem of how to calculate $\Omega$ out of sample.

$\endgroup$
  • $\begingroup$ The Goldberg paper and your explanation clarified very much - thank you! I believe I can proceed then by comparing the out-of-sample predictions against the residuals (to calculate omega and w'). Also the variance will no longer be zero in this scenario. $\endgroup$ – Ram Ahluwalia Aug 18 '11 at 14:45
  • $\begingroup$ To clarify for posterity Goldberger defines w as the Tx1 vector of covariance of the residuals of the prediction with the vector of residuals from the sample data. u* is defined as the 1x1 scalar value of the prediction residual, and u is the Tx1 vector of sample residuals. $\endgroup$ – Ram Ahluwalia Aug 18 '11 at 15:19
  • $\begingroup$ can you clarify w? Goldberger defines w as the Tx1 vector of covariances of the prediction residuals with the vector of sample residuals. How can a covariance matrix have shape Tx1 as opposed to TxT? Or is each element of vector W computed by taking the scalar u* and multiplying by the Tx1 vector of sample residuals? $\endgroup$ – Ram Ahluwalia Aug 18 '11 at 15:32
2
$\begingroup$

The $\beta$'s that come out of GLS are estimates of the predictor effects that contribute to the mean response and the assumption of GLS is that the mean of $Y|X$ is the same for each data point - only non-constant variance and serial correlation are allowed for. So, to predict a new data point, yes, you would just plug in the predictor values into the linear predictor. The observed prediction error will usually take a form like

$$ \sum_{i} (Y_{i} - \hat{Y}_{i} )^2 $$

which potentially is the mean of a sum of non-independent random variables if the test data points are heteroskedastic/autocorrelated in the same way the training data was. Fortunately, the linearity of expectation,

$$ E \big( \sum_{i} X_{i} \big) = \sum_{i} E(X_{i}) $$

is true regardless of whether or not the $X_{i}$ are independent so your prediction error will not be biased.

$\endgroup$
  • 1
    $\begingroup$ Thanks! Quick follow-up : Let's say I have no scored the model on out-of-sample data, have captured residuals, and now would like to measure the standard error of the model. I assume in this case, much like in the in-sample case, I will go ahead and use GLS weights to modify the error reporting. $\endgroup$ – Ram Ahluwalia Aug 17 '11 at 22:07
  • 1
    $\begingroup$ (+1) This is correct, but I believe the key point is that to compare a prediction to a value, it is helpful (and in many applications essential) to have an estimate of the prediction variance. That's impossible to do with the setup described in the question. You can get one component of that variance from the error estimates of the parameters, but there is no way to infer the covariance of the prediction residual with the rest of the data, which is the other component of the prediction variance. $\endgroup$ – whuber Aug 17 '11 at 22:08
  • $\begingroup$ @whuber - well, if you believe the test data is from the same population as the training data and that the structure of the covariance matrix estimated from the model is correct, then don't you have an estimate of the covariance matrix of the $Y$'s? In that case, you could calculate an estimate of the variance of $\sum (Y_{i}-\hat{Y}_{i})^{2}$ using that knowledge, yes? $\endgroup$ – Macro Aug 17 '11 at 22:11
  • $\begingroup$ But apparently the covariance matrix varies willy-nilly from one point to the next. In short, you need to model the covariance matrix as a function of the independent variables (or of additional covariates) if you want to carry out your suggestion (which I think is a great idea). $\endgroup$ – whuber Aug 17 '11 at 22:13
  • $\begingroup$ @Quant, I'm not sure I understand your comment. If your test data is from the same population as the training data, you should be able to character the prediction variance using the estimated covariance matrix (see above). $\endgroup$ – Macro Aug 17 '11 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.