I have a multiple regression model, which I can estimate either with OLS or GLS. The weights for the GLS are estimated exogenously (the dataset for the weights is different from the dataset for the model). I'm trying to determine if one estimation technique is "better" than the other by looking at the coefficients, t-stats, f-stat, $R^2$, ... . I guess it's a question about the weights in general. If I have multiple sets of weights (including uniform), how can I choose the set that will give the "best" estimation?
$\begingroup$
$\endgroup$
6
-
3$\begingroup$ Note that $R^2$ of OLS will by definition be higher than for any other linear model (including GLS). Thus it does not make much sense to compare the OLS and the GLS models by looking at $R^2$, since you know the answer before even looking at the numbers, regardless of the data you have. $\endgroup$– Richard HardyCommented Jun 2, 2015 at 9:33
-
1$\begingroup$ Why is this the case? OLS maximizes $R^2$ while GLS maximizes weighted $R^2$. In a degenerate case I can make $R^2$ in GLS equal to 1 by setting the weights to 1 for any 2 observations and 0 for all the other ones. $\endgroup$– mssCommented Jun 2, 2015 at 15:45
-
$\begingroup$ Oh, I thought you followed the same (classical) definition of $R^2$ both for OLS and for GLS. If you modify the $R^2$ for GLS as you say, then my remark does not apply. $\endgroup$– Richard HardyCommented Jun 2, 2015 at 16:14
-
$\begingroup$ @Richard Hardy, does your comment about R^2=1-SS_residual/SS_total being higher for OLS than GLS hold true when the R^2 are computed on a hold out test dataset (or on hold out via cross validation)? I fail to see that R^2 would be higher for OLS in general ... $\endgroup$– Ggjj11Commented Nov 3 at 20:54
-
$\begingroup$ @Ggjj11, in the context of the question I take $R^2$ to be used in the usual sense of being the in-sample metric. I do not like to use the same notation for out-of-sample $R^2$, as I find that confusing. That the out-of-sample $R^2$ is not uniquely defined does not help either (by just extending the three equivalent definitions of the in-sample metric we get nonequivalent out-of-sample metrics). $\endgroup$– Richard HardyCommented Nov 4 at 7:09
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
2
The real difference between OLS and GLS is the assumptions made about the error term of the model. In OLS we (at least in CLM setup) assume that $Var(u)=\sigma^2 I$, where I is the identity matrix - such that there are no off diagonal elements different from zero. With GLS this is no longer the case (it could be, but then GLS = OLS). With GLS we assume that $Var(u) = \sigma^2 \Sigma$, where $\Sigma$ is the variance-covariance matrix.
Many text books introduce GLS with WLS, which is the GLS function that eliminates heteroskedasticity (or tries to). This means that the usual t/F statistics can be valid for the GLS estimation, but not for the OLS. This is less troublesome today, since you can just compute robust variance estimates and base you inference on that - same as you normally would.
This implies that difference between OLS and GLS is in the variance of the estimates. And the real reason, to choose, GLS over OLS is indeed to gain asymptotic efficiency (smaller variance for n $\rightarrow \infty$. It is important to know that the OLS estimates can be unbiased, even if the underlying (true) data generating process actually follows the GLS model. If GLS is unbiased then so is OLS (and vice versa).
You can very easily proof this, but basically the assumptions for consistency/unbiasedness do not rely on the variance of the estimates at all. A more subtle point is that, unless you know the actual GLS function, it is not unbiased but only consistent.
I would therefore argue that choosing between OLS and GLS based on estimates and $R^2$ is the wrong way to think about it. The estimates of both OLS and GLS should be close to one another, if not numerically then in the size of the “impact”. If they are not, then it would most likely indicate that you have a function form misspecification(s), of that you have left out variables.
I don’t know whether or not, excluding the GLS weights covariates is justified in your case - but perhaps it worth trying to include them in the OLS estimation and see what happens? It might make the reader less “suspicious” about your conclusion (but this is pure speculation on my part).
-
-
$\begingroup$ I'm trying to use the weights not to deal with heteroscedasticity, but to introduce a notion of "importance" of an observation. Let's say I have a prior knowledge that some observations represent a true relations more than others. I guess this could be thought of as heteroscedasticity, i.e. "important" observations have lower standard error. As a back of the envelope type adjustment, I wanted to modify the design matrix by repeating "important" observations. So, if observation $\#1$ is twice as important as $\#2$, I would add another row to the design matrix with $\#1$ observation. $\endgroup$– mssCommented Jun 1, 2015 at 21:39