Comparing OLS versus WLS residual standard error, is the smaller the better?

Question

For a linear regression model I tried on a dataset, when I fitted OLS, the output is as follows:

fit = lm(price ~., data = art)

# Coefficients:
#                                  Estimate Std. Error t value Pr(>|t|)    
# (Intercept)                     2.326e+03  8.863e+02   2.625 0.008777 ** 
# temp_stagelate                  3.029e-01  8.735e-02   3.467 0.000544 ***
# temp_stagemature                4.009e-01  1.154e-01   3.473 0.000533 ***
# temp_stagemiddle                5.346e-01  8.646e-02   6.184 8.55e-10 ***
# temp_stagena                    1.766e-01  8.645e-02   2.042 0.041322 *  
# tonedark                       -4.306e-01  5.511e-02  -7.814 1.20e-14 ***
# tonelight                      -4.267e-01  6.214e-02  -6.866 1.05e-11 ***
# subjectflower-animal           -3.997e-01  6.972e-02  -5.734 1.24e-08 ***
# subjectlandscape                1.609e-02  6.429e-02   0.250 0.802480    
# size.square                     2.454e-04  1.696e-05  14.469  < 2e-16 ***
# size.sqq                       -6.205e-09  9.412e-10  -6.593 6.44e-11 ***
# coloringink and color           3.665e-01  5.522e-02   6.637 4.81e-11 ***
# further.inscribed.or.signedyes  2.997e-01  7.868e-02   3.810 0.000146 ***
# signedyes                       9.487e-01  2.058e-01   4.610 4.45e-06 ***
# inscribedyes                    1.865e-01  5.966e-02   3.126 0.001812 **
#   
# Residual standard error: 0.6806 on 1511 degrees of freedom
# Multiple R-squared:  0.726,    Adjusted R-squared:  0.7186 

and when I tried to fit a weighted least squares (WLS) model,

gls(price ~. , data=art, weights = varFixed(~size.square)) 

the output is of the following:

# Standardized residuals:
#     Min          Q1         Med          Q3         Max 
# -4.27128213 -0.58938641 -0.06659419  0.53758125  6.03938177 
# 
# Coefficients:
#                                    Value Std.Error    t-value p-value
# (Intercept)                    1851.4155  924.1690   2.003330  0.0454
# temp_stagelate                    0.3717    0.1020   3.645983  0.0003
# temp_stagemature                  0.6992    0.1257   5.563261  0.0000
# temp_stagemiddle                  0.4881    0.1032   4.729729  0.0000
# temp_stagena                      0.2973    0.0978   3.038328  0.0024
# tonedark                         -0.4489    0.0571  -7.867553  0.0000
# tonelight                        -0.4451    0.0627  -7.093028  0.0000
# subjectflower-animal             -0.3605    0.0716  -5.036949  0.0000
# subjectlandscape                  0.0625    0.0669   0.934295  0.3503
# size.square                       0.0003    0.0000  13.128687  0.0000
# size.sqq                          0.0000    0.0000  -5.179399  0.0000
# coloringink and color             0.4053    0.0560   7.235716  0.0000
# further.inscribed.or.signedyes    0.3425    0.0866   3.955705  0.0001
# signedyes                         0.8654    0.2979   2.905369  0.0037
# inscribedyes                      0.3211    0.0574   5.593102  0.0000
# 
# Residual standard error: 0.01437576 
# Degrees of freedom:  1511 residual

I get a much smaller residual standard error. I am wondering if the two residual errors I have are comparable, and if the smaller residual standard error in the WLS model indicates that the WLS yields a better fit?

Does a smaller residual standard error in general indicate a better fit?

Answer 1

As @RichardHardy says, Ordinary Least Squares (OLS) can be used when you can reasonably assume that your data is homoscedastic. Weighted Least Squares (WLS) can be used when your data is heteroscedastic (but uncorrelated) and Generalised Least Squares (GLS) accounts for correlation and heterscedasticity.

When you compute gls(price ~. , data=art, weights = varFixed(~size.square)) you are assuming heteroscedasticity without correlation. Moreover, you assume that the variance of each osbservation is proportional to size.square.

The residual variance (the square of the residual standard error) obtained in the output of gls is only the proportionality constant, it does not contain the part that is proportional with size.square.

I don't know your data but you could try the following (i.e. WLS) and I think you will find the result of gls (see e.g. D.N. Gujarati, ''Basic econometrics''):

1. The intercept is to be replaced by a column vector $x_0$ with elements all equal to 1
2. replace all your independent variables $x_i$ by $v_i = x_i/\sqrt{size.square}$, also the $x_0$ that replaces the intercept;
3. Do the same for your dependent variable, i.e. $y=price/\sqrt{size.square}$;
4. regress this 'new' dependent variable $y$ on all the ''new'' independent variables $v_i$ using a model without an intercept, i.e. lm ( y ~ v0 + v1 ... + vn - 1)

Note: you weight each observation by $1/\sqrt{size.square}$ and then perform least squares (lm), hence the name ''weighted least squares''.

The residual standard error of the regression lm ( y ~ v0 + v1 ... + vn - 1) will be 0.01437 (see your example outcome for gls), so it is the resudual error after dividing each observation by $\sqrt{size.square}$ (and thus also after dividing the standard deviation by that value).

Answer 2

Given the regressand ($y$) and the regressors ($x$'s), OLS will by costruction yield the lowest in-sample residual standard error among the linear estimators. After all, OLS minimizes the residual standard error (hence the term "least squares"). You should not be getting a smaller error from WLS than from OLS; it looks as if you are using different data for OLS than for WLS, which could explain the phenomenon. All in all, comparing OLS and WLS in terms of residual standard error is pretty meaningless -- you know the answer before you start the exercise.

The use of WLS may be justified if you believe that different observations have different error variances, i.e. $\text{Var}(\varepsilon_1)=\dotsc=\text{Var}(\varepsilon_n)$ does not hold. Then WLS may be more efficient than OLS (as long as you are able to obtain weights that are roughly proportional to inverse error variances).

Answer 3

In terms of fit, both will give you the same estimates, so there is no difference.

In terms of inference, however, there will be. Smaller standard errors mean estimates are more precisely estimated. If you fish for stars in your regression table, then you should choose the smallest standard errors.

If you are more scientific, you will first ask yourself whether you should use a correction for heteroskedasticity at all. For that, you have a plethora of tests to run.