2
$\begingroup$

For a linear regression model I tried on a dataset, when I fitted OLS, the output is as follows:

fit = lm(price ~., data = art)

# Coefficients:
#                                  Estimate Std. Error t value Pr(>|t|)    
# (Intercept)                     2.326e+03  8.863e+02   2.625 0.008777 ** 
# temp_stagelate                  3.029e-01  8.735e-02   3.467 0.000544 ***
# temp_stagemature                4.009e-01  1.154e-01   3.473 0.000533 ***
# temp_stagemiddle                5.346e-01  8.646e-02   6.184 8.55e-10 ***
# temp_stagena                    1.766e-01  8.645e-02   2.042 0.041322 *  
# tonedark                       -4.306e-01  5.511e-02  -7.814 1.20e-14 ***
# tonelight                      -4.267e-01  6.214e-02  -6.866 1.05e-11 ***
# subjectflower-animal           -3.997e-01  6.972e-02  -5.734 1.24e-08 ***
# subjectlandscape                1.609e-02  6.429e-02   0.250 0.802480    
# size.square                     2.454e-04  1.696e-05  14.469  < 2e-16 ***
# size.sqq                       -6.205e-09  9.412e-10  -6.593 6.44e-11 ***
# coloringink and color           3.665e-01  5.522e-02   6.637 4.81e-11 ***
# further.inscribed.or.signedyes  2.997e-01  7.868e-02   3.810 0.000146 ***
# signedyes                       9.487e-01  2.058e-01   4.610 4.45e-06 ***
# inscribedyes                    1.865e-01  5.966e-02   3.126 0.001812 **
#   
# Residual standard error: 0.6806 on 1511 degrees of freedom
# Multiple R-squared:  0.726,    Adjusted R-squared:  0.7186 

and when I tried to fit a weighted least squares (WLS) model,

gls(price ~. , data=art, weights = varFixed(~size.square)) 

the output is of the following:

# Standardized residuals:
#     Min          Q1         Med          Q3         Max 
# -4.27128213 -0.58938641 -0.06659419  0.53758125  6.03938177 
# 
# Coefficients:
#                                    Value Std.Error    t-value p-value
# (Intercept)                    1851.4155  924.1690   2.003330  0.0454
# temp_stagelate                    0.3717    0.1020   3.645983  0.0003
# temp_stagemature                  0.6992    0.1257   5.563261  0.0000
# temp_stagemiddle                  0.4881    0.1032   4.729729  0.0000
# temp_stagena                      0.2973    0.0978   3.038328  0.0024
# tonedark                         -0.4489    0.0571  -7.867553  0.0000
# tonelight                        -0.4451    0.0627  -7.093028  0.0000
# subjectflower-animal             -0.3605    0.0716  -5.036949  0.0000
# subjectlandscape                  0.0625    0.0669   0.934295  0.3503
# size.square                       0.0003    0.0000  13.128687  0.0000
# size.sqq                          0.0000    0.0000  -5.179399  0.0000
# coloringink and color             0.4053    0.0560   7.235716  0.0000
# further.inscribed.or.signedyes    0.3425    0.0866   3.955705  0.0001
# signedyes                         0.8654    0.2979   2.905369  0.0037
# inscribedyes                      0.3211    0.0574   5.593102  0.0000
# 
# Residual standard error: 0.01437576 
# Degrees of freedom:  1511 residual

I get a much smaller residual standard error. I am wondering if the two residual errors I have are comparable, and if the smaller residual standard error in the WLS model indicates that the WLS yields a better fit?

Does a smaller residual standard error in general indicate a better fit?

$\endgroup$
9
  • 1
    $\begingroup$ May it be the case that for WLS you consider standardized residuals while for OLS you consider raw residuals? Then the residual standard errors may not be comparable. $\endgroup$ Dec 17, 2015 at 16:14
  • $\begingroup$ Hmm, I am not sure whether the function gls really does WLS, but you may know better. You could do WLS partly mechanically: model1=lm(y~X); resid1=residuals(model1); weights=diag(resid1 %*% t(resid1))^(-1); model2=lm(y~X,weights=weights). $\endgroup$ Dec 17, 2015 at 16:30
  • $\begingroup$ Thanks! But when I tried the procedure you mentioned above, I resulted in a much higher adjusted R-sqaure but when I checked the testing MSE, it is not getting any better. What could the reason causing this? $\endgroup$
    – lll
    Dec 17, 2015 at 17:15
  • $\begingroup$ I wonder if you have tried the mechanical way of doing WLS? Are you getting similar results to those from gls? I am a bit suspicious regarding how gls works since it uses maximum likelihood estimation which is noever required in standard WLS or GLS... $\endgroup$ Dec 17, 2015 at 17:16
  • $\begingroup$ OK, I see. But MSE of WLS is now larger than that of OLS, isn't it? At least that part will be right then. Regarding $R^2_{adj.}$, that is a good question! I tried it with simulated data and got very high $R^2$ and $R^2_{adj.}$ for independent data... This does not seem right. And that may merit a separate question. $\endgroup$ Dec 17, 2015 at 17:18

3 Answers 3

2
$\begingroup$

As @RichardHardy says, Ordinary Least Squares (OLS) can be used when you can reasonably assume that your data is homoscedastic. Weighted Least Squares (WLS) can be used when your data is heteroscedastic (but uncorrelated) and Generalised Least Squares (GLS) accounts for correlation and heterscedasticity.

When you compute gls(price ~. , data=art, weights = varFixed(~size.square)) you are assuming heteroscedasticity without correlation. Moreover, you assume that the variance of each osbservation is proportional to size.square.

The residual variance (the square of the residual standard error) obtained in the output of gls is only the proportionality constant, it does not contain the part that is proportional with size.square.

I don't know your data but you could try the following (i.e. WLS) and I think you will find the result of gls (see e.g. D.N. Gujarati, ''Basic econometrics''):

  1. The intercept is to be replaced by a column vector $x_0$ with elements all equal to 1
  2. replace all your independent variables $x_i$ by $v_i = x_i/\sqrt{size.square}$, also the $x_0$ that replaces the intercept;
  3. Do the same for your dependent variable, i.e. $y=price/\sqrt{size.square}$;
  4. regress this 'new' dependent variable $y$ on all the ''new'' independent variables $v_i$ using a model without an intercept, i.e. lm ( y ~ v0 + v1 ... + vn - 1)

Note: you weight each observation by $1/\sqrt{size.square}$ and then perform least squares (lm), hence the name ''weighted least squares''.

The residual standard error of the regression lm ( y ~ v0 + v1 ... + vn - 1) will be 0.01437 (see your example outcome for gls), so it is the resudual error after dividing each observation by $\sqrt{size.square}$ (and thus also after dividing the standard deviation by that value).

$\endgroup$
3
$\begingroup$

Given the regressand ($y$) and the regressors ($x$'s), OLS will by costruction yield the lowest in-sample residual standard error among the linear estimators. After all, OLS minimizes the residual standard error (hence the term "least squares"). You should not be getting a smaller error from WLS than from OLS; it looks as if you are using different data for OLS than for WLS, which could explain the phenomenon. All in all, comparing OLS and WLS in terms of residual standard error is pretty meaningless -- you know the answer before you start the exercise.

The use of WLS may be justified if you believe that different observations have different error variances, i.e. $\text{Var}(\varepsilon_1)=\dotsc=\text{Var}(\varepsilon_n)$ does not hold. Then WLS may be more efficient than OLS (as long as you are able to obtain weights that are roughly proportional to inverse error variances).

$\endgroup$
3
  • $\begingroup$ I used the same data set thought, so I am wondering what is causing this situation... $\endgroup$
    – lll
    Dec 17, 2015 at 15:50
  • $\begingroup$ What about data = all for OLS versus data = art for WLS? Also, could you report comparable results? Now the OLS output differs from GLS output. Perhaps try summary of one and the other. $\endgroup$ Dec 17, 2015 at 15:51
  • $\begingroup$ it's a typo. I added the summary $\endgroup$
    – lll
    Dec 17, 2015 at 16:17
-2
$\begingroup$

In terms of fit, both will give you the same estimates, so there is no difference.

In terms of inference, however, there will be. Smaller standard errors mean estimates are more precisely estimated. If you fish for stars in your regression table, then you should choose the smallest standard errors.

If you are more scientific, you will first ask yourself whether you should use a correction for heteroskedasticity at all. For that, you have a plethora of tests to run.

$\endgroup$
1
  • 3
    $\begingroup$ Hmm, how should OLS and WLS give the same estimates? $\beta_{OLS}=(X'X)^{-1}X'y$ while $\beta_{WLS}=(X'WX)^{-1}X'Wy$ where in general $W \neq I$; hence, $\beta_{OLS} \neq \beta_{WLS}$. This will also result in different fitted values, different model residuals etc. $\endgroup$ Dec 17, 2015 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.