7
$\begingroup$

I have a multiple regression model, which I can estimate either with OLS or GLS. The weights for the GLS are estimated exogenously (the dataset for the weights is different from the dataset for the model). I'm trying to determine if one estimation technique is "better" than the other by looking at the coefficients, t-stats, f-stat, $R^2$, ... . I guess it's a question about the weights in general. If I have multiple sets of weights (including uniform), how can I choose the set that will give the "best" estimation?

$\endgroup$
3
  • 2
    $\begingroup$ Note that $R^2$ of OLS will by definition be higher than for any other linear model (including GLS). Thus it does not make much sense to compare the OLS and the GLS models by looking at $R^2$, since you know the answer before even looking at the numbers, regardless of the data you have. $\endgroup$ Jun 2 '15 at 9:33
  • $\begingroup$ Why is this the case? OLS maximizes $R^2$ while GLS maximizes weighted $R^2$. In a degenerate case I can make $R^2$ in GLS equal to 1 by setting the weights to 1 for any 2 observations and 0 for all the other ones. $\endgroup$
    – mss
    Jun 2 '15 at 15:45
  • $\begingroup$ Oh, I thought you followed the same (classical) definition of $R^2$ both for OLS and for GLS. If you modify the $R^2$ for GLS as you say, then my remark does not apply. $\endgroup$ Jun 2 '15 at 16:14
15
$\begingroup$

The real difference between OLS and GLS is the assumptions made about the error term of the model. In OLS we (at least in CLM setup) assume that $Var(u)=\sigma^2 I$, where I is the identity matrix - such that there are no off diagonal elements different from zero. With GLS this is no longer the case (it could be, but then GLS = OLS). With GLS we assume that $Var(u) = \sigma^2 \Sigma$, where $\Sigma$ is the variance-covariance matrix.

Many text books introduce GLS with WLS, which is the GLS function that eliminates heteroskedasticity (or tries to). This means that the usual t/F statistics can be valid for the GLS estimation, but not for the OLS. This is less troublesome today, since you can just compute robust variance estimates and base you inference on that - same as you normally would.

This implies that difference between OLS and GLS is in the variance of the estimates. And the real reason, to choose, GLS over OLS is indeed to gain asymptotic efficiency (smaller variance for n $\rightarrow \infty$. It is important to know that the OLS estimates can be unbiased, even if the underlying (true) data generating process actually follows the GLS model. If GLS is unbiased then so is OLS (and vice versa).

You can very easily proof this, but basically the assumptions for consistency/unbiasedness do not rely on the variance of the estimates at all. A more subtle point is that, unless you know the actual GLS function, it is not unbiased but only consistent.

I would therefore argue that choosing between OLS and GLS based on estimates and $R^2$ is the wrong way to think about it. The estimates of both OLS and GLS should be close to one another, if not numerically then in the size of the “impact”. If they are not, then it would most likely indicate that you have a function form misspecification(s), of that you have left out variables.

I don’t know whether or not, excluding the GLS weights covariates is justified in your case - but perhaps it worth trying to include them in the OLS estimation and see what happens? It might make the reader less “suspicious” about your conclusion (but this is pure speculation on my part).

$\endgroup$
2
  • $\begingroup$ Nice answer .... $\endgroup$
    – IrishStat
    Jun 1 '15 at 21:28
  • $\begingroup$ I'm trying to use the weights not to deal with heteroscedasticity, but to introduce a notion of "importance" of an observation. Let's say I have a prior knowledge that some observations represent a true relations more than others. I guess this could be thought of as heteroscedasticity, i.e. "important" observations have lower standard error. As a back of the envelope type adjustment, I wanted to modify the design matrix by repeating "important" observations. So, if observation $\#1$ is twice as important as $\#2$, I would add another row to the design matrix with $\#1$ observation. $\endgroup$
    – mss
    Jun 1 '15 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.