Residuals dont sum to zero: Stata bug? Or User error?

Question

This is my first post to this wonderful community. I am having a vexing problem with Stata 13 MP 64 bit.

The use of .xtpcse and .predict are not generating residuals that sum to zero ( residuals = observed values - fitted values) . Residuals summed to a positive 41,716.38 (from pairs of 13,443 observed and fitted values. The absolute values of obs values had average of 27.7), so the fitted values were low by about 3.1 on average. Why???

I found the same problem when I manually calculated the predicted values.

Here is my main model with its stata output. Further below, I discuss the consequences when I used .reg with and without aweights and .xtpcse with and without aweights.

xtpcse rep_code_vlist  avr_ph_i   avr_ph_ii  stg1_pricon_d   stg2_pricon_d    prim_reg_season  stg2_pricon_r_x_olmom    poc_decimal_91  poc_decimal91_x_18t22_x_ph2 poc_decimal91_x_23t30_x_ph2 poc_decimal91_x_31t37_x_ph2 poc_decimal91_x_38t43_x_ph2      _18_to_22 _23_to_30  _31_to_37 _38_to_43 _44_to_66         shd1     shd2     shd3     shd4     shd5     shd6     shd7     shd8     shd9     shd10     shd11     shd12     shd13     shd14     shd15     shd16     shd17     shd18     shd19     shd20     shd21     shd22     shd23     shd24     shd25     shd26     shd27     shd28     shd29     shd30     shd31     shd32     shd33     shd34     shd35     shd36     shd37     shd38     shd39     shd40     shd41     shd42     shd43     shd44     shd45     shd46     shd47     shd48     shd49     shd50     shd51     shd52     shd53     shd54     shd55     shd56     shd57     shd58     shd59      w1     w2     w3     w4     w5     w6     w7     w8     w9     w10     w11     w12     w13     w14     w15     w16     w17     w18     w19     w20     w21     w22     w23     w24     w25     w26     w27     w28     w29     w30     w31     w32     w33     w34     w35     w36     w37     w38     w39     w40     w41         y10   y12     y14    y16  pres_year  [aw= aweight ], corr(ar1)

Here are the results. I trucated the list of variables, coefs, etc.

>Prais-Winsten regression, correlated panels corrected standard errors (PCSEs)
>
>Group variable:   state_house_district          Number of obs      =     13444
>Time variable:    weeks_after_jan_1_2006        Number of groups   =        60
>Panels:           correlated (unbalanced)       Obs per group: min =       149
>Autocorrelation:  common AR(1)                                 avg =  224.0667
>Sigma computed by casewise selection                           max =       249
>Estimated covariances      =      1830          R-squared          =    0.8216
>Estimated autocorrelations =         1          Wald chi2(121)     =   1347.17
>Estimated coefficients     =       122          Prob > chi2        =    0.0000
>
>---------------------------------------------------------------------------------------------
>                            |           Panel-corrected
>             rep_code_vlist |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
>----------------------------+----------------------------------------------------------------
>
>                   avr_ph_i |  -20.75933   9.694859    -2.14   0.032     -39.7609   -1.757752
>                  avr_ph_ii |   172.3479   19.48763     8.84   0.000     134.1528    210.5429
>              stg1_pricon_d |   17.34981   11.01055     1.58   0.115    -4.230477    38.93009
>              stg2_pricon_d |    42.2724   11.29455     3.74   0.000     20.13548    64.40932
>            prim_reg_season |  -32.76939    10.0791    -3.25   0.001    -52.52406   -13.01472
>      stg2_pricon_r_x_olmom |   29.97346   15.29307     1.96   0.050    -.0004098    59.94734
>             poc_decimal_91 |  -.9604848   4.702387    -0.20   0.838    -10.17699    8.256025
>                 .
>                 .
>                 .
>
>                        y14 |   12.81982   8.375178     1.53   0.126    -3.595228    29.23486
>                        y16 |   26.23057   6.787419     3.86   0.000     12.92747    39.53366
>                  pres_year |    25.0513   8.327703     3.01   0.003       8.7293    41.37329
>                      _cons |   -13.2098   31.56652    -0.42   0.676    -75.07905    48.65945
>----------------------------+----------------------------------------------------------------
>                        rho |   .1980228
>---------------------------------------------------------------------------------------------

I then used . . .

>. predict fitted_values

I got the results that I discussed above.

When I tried .reg with aweights, residuals still did not sum to zero. I also tried .xtpcse without aweights and had the same problem. I tried a few other variations with similar outcomes. Also, R2 fluctated wildly between models. Coefficients were fairly stable between models.

When I used the nonconstant option along with .xtpcse aweights, the sum of residuals skyrocketted to a minus 118,205.527 and the R2 plummeted from .8216 to that of 0.5217.

The only estimator and model that gave residuals that summed to zero was .reg when aweights were NOT USED. However, again, .reg witih aweights gave residuals that did not sum to zero.

I also tried closing stata and starting the software (Stata 13/MP 64 bit), but the same problem persisted. I am using Windows 10 64 bit.

Any ideas as to what is going on? Bug or user error? I am dumbfounded. I dont have the budget to upgraded to a newer version of stata.

Answer 1

The residuals should not typically sum to zero in weighted least squares, generalised least squares, or mixed models for unbalanced panel data.

The regression coefficients are solving some equation of the form $GX\beta=GY$ where $G^TG$ is the weight matrix. This equation implies $G(Y-X\beta)=0$.

For ordinary least squares, the corresponding equation is $X\beta=Y$, implying $Y-X\beta=0$, with the quantity on the left being the sum of the residuals.

For weighted least squares or generalised least squares, then, it's not the sum of the residuals that's exactly zero, but a suitable weighted sum. The whole point of the estimation procedure is not to give all residuals equal weight, so it doesn't.

Using the nonconstant option changes the intercept from -13 to zero, in just over 13 thousand observations, so you shouldn't be surprised by a change in the residual sum that's anything up to about $13\times 13000=169000$