1
$\begingroup$

With a small book-exercise with four metric variables on 10 cases (one dependent/outcome, three independent/predictor) I ran linear regression in SPSS and R, and ANOVA (in SPSS declaring the predictors as "covariates").
I found the output of the SSQ (Sum-of-Squares) different - and obviously from this also the F-test statistic and the p-values. Except from the last predictor the displayed values are different (the predictors in Rmay be reordered and the analysis be rerun to find all SPSS- coefficients).

By reengineering the computations in matrix-formulae I could reproduce the SPSS-values as well as the R-values and found, that SPSS uses the (partial) SSQ based on the logic of the "usefulness"-coefficients for each predictor (which is sort of semipartial coefficient), while R simply uses the (hierarchically) partial SSQ. (Unfortunately I'm not sure how to express that two approaches correctly so this toy-characterizing might be improved) .

Q: Has that property of different focuses/philosophies been discussed anywhere? Is there some advantage of one over the other?


Data: (taken from M. Backhaus et al., multivariate Verfahren)

predictors                   outcome-item
---------------------------+-------------
Preis   VerkFoer  Vertreter  Absatzmenge
12.50      2000      109      2298
10.00       550      107      1814
 9.95      1000       99      1647
11.50       800       70      1496
12.00         0       81       969
10.00      1500      102      1918
 8.00       800      110      1810
 9.00      1200       92      1896
 9.50      1100       87      1715
12.50      1300       79      1699

The comparision of the output: bild

$\endgroup$
  • $\begingroup$ SPSS usue by default type III SS while R uses default type I SS. You can request both to use the same type. Please searh ANOVA SS types in the internet and this site. Thousands of pages discuss it. $\endgroup$ – ttnphns Dec 19 '15 at 20:12
  • $\begingroup$ As far as I know the type "I" and "III" are different only in the case of interactions, but which are not introduced here. No, it's really a different idea what to display to the user -. which is in the case of SPSS the "usefulness"-concept and in R the simple partial coproduct. So we find the three coefficients shown by SPSS if we do three models in R as model.1: absatzmenge ~ preis + vertreter + verkfoerd and model.2: absatzmenge ~ vertreter + verkfoerd + preis and model.3: absatzmenge ~ verkfoerd + preis +vertreter so that each of the 3 predictors is one time at the end of the(...) $\endgroup$ – Gottfried Helms Dec 19 '15 at 20:52
  • $\begingroup$ (...) formula. Then each of that last (=completely) partialled out sum-of-squares gives one coefficient which is shown by SPSS. And I do vaguely remember a 30 year old discussion where someone proposed the use of semipartials instead of partials in the regression. (But this is a complete imprecise memory) @ttphns $\endgroup$ – Gottfried Helms Dec 19 '15 at 20:55
  • $\begingroup$ @ttnphns: Upps, I see. Your hint is correct, using SS typ I gives the same coefficients now as in R , and thus the same idea of showing the partialled square-sums. (I got from another Answer/Question which I scanned here before the impression that SS Typ I and SS Typ III would only differ on interaction - that was wrong, so sorry, that part of my comment was wrong.) But still is my question: which output is the more meaningful one? It was annoying to me to show the students the SPSS output and been later confronted with the R-output both claiming to be the procedure Anova . $\endgroup$ – Gottfried Helms Dec 19 '15 at 21:04
  • 1
    $\begingroup$ As far as I know the type "I" and "III" are different only in the case of interactions. If the design appears to be unbalanced disproportional SS I and SS III may differ even in the absence of interaction term. $\endgroup$ – ttnphns Dec 19 '15 at 22:59
1
$\begingroup$

I want to illustrate the difference of the use of SS type I and SS type III as hinted by @ttnphns in his comment. This explains the different result gotten by the software and says also how to arrive at the compatible coefficients. However, this does not yet answer my second question: which is the better one/meaningful one? This is of course dependent on the question, and although we find a lot of discussion in SSE as also hinted by ttnhpns I found it difficult to digest the many contributions to understand what's really behind the scene.
So I thought it might be instructive (at least it was for me) to show how the SS I and SS III results are computed by the matrix-formula applied on the raw data (see my initial question). I added one column with "const=1" to my data and did no centering or standardizing.

[55] CoProd = data' * data    // just the analogue of a covariance matrix  

CoProd         |  const          preis          VerkFörd       Vertreter    |    Absatz      
------         +  ------         ------         ------         ------       +    ------      
const          |        10.000        104.950      10250.000        936.000 |       17262.000
preis          |       104.950       1123.003     108550.000       9736.550 |      180338.650
VerkFörd       |     10250.000     108550.000   13172500.000     981650.000 |    19132900.000
Vertreter      |       936.000       9736.550     981650.000      89370.000 |     1643436.000
------         +  ------         ------         ------         ------       +    ------      
Absatz         |     17262.000     180338.650   19132900.000    1643436.000 |    30838452.000
------         +  ------         ------         ------         ------       +    ------      

In the diagonal we find simply the SSq (Sum of Squares), in the off-diagonal entries the crossproducts of the data. For instance the value of the full variation of the dependent variable ("Absatz") is 30838452.000 in the bottom-right cell.

Next we find a "coordinatesystem" as we would do in principal components ("Loadings matrix"); an initial solution can be taken by a cholesky decomposition.

[58] PL = cholesky(CoProd)  

PL             |  [const]        [preis]        [VerkFörd]     [Vertreter]  |    [Absatz]    
------         +  ------         ------         ------         ------       +    ------      
const          |         3.162          0.000          0.000          0.000 |           0.000
preis          |        33.188          4.642          0.000          0.000 |           0.000
VerkFörd       |      3241.335        210.288       1619.268          0.000 |           0.000
Vertreter      |       295.989        -18.691         16.168         33.907 |           0.000
------         +  ------         ------         ------         ------       +    ------      
Absatz         |      5458.724       -177.932        911.997        284.368 |         310.685
------         +  ------         ------         ------         ------       +    ------      

In principal components this were the "coordinates" in the euclidean(orthogonal) "factor-space" . The brackets in the header-line indicate, that the columns are to be understood as "partialled".

As in PL we have "coordinates" we get now in the following the (partialled sums of) squares by elementwise squaring (in principal components analysis this were now (partial) "covariances") :

[61] PL2 = PL  ^# 2          // compute squares elementwise 

PL²            |  [const]        [preis]        [VerkFörd]     [Vertreter]  |    [Absatz]    
------         +  ------         ------         ------         ------       +    ------      
const          |        10.000          0.000          0.000          0.000 |           0.000
preis          |      1101.450         21.552          0.000          0.000 |           0.000
VerkFörd       |  10506250.000      44221.094    2622028.906          0.000 |           0.000
Vertreter      |     87609.600        349.339        261.406       1149.655 |           0.000
------         +  ------         ------         ------         ------       +    ------      
Absatz         |  29797664.400      31659.900     831737.936      80864.894 |       96524.870
------         +  ------         ------         ------         ------       +    ------      

The values which we got now by R Anova -procedure were simply the values in the last row "Absatz" which represent so-to-say partially "explained" squaresums.

SPSS would give us from this table only the value 80964 (and the remaining values from the following tables) which is the (partial) squaresum in "Absatz" which is explained by "Vertreter" (after partialling out all other items). This coefficient is also analoguous to the concept of usefulness in regression (here "usefulness" of the item "Vertreter")

The value 96524 in the column of [Absatz] is of course the "unexplained squaresum"

To get the other coefficients for the SS III solution we rotate the PL-matrix such that the previous variable "VerkFoerd" has now individual loading on the 4'th axis. In R this would mean to redefine the formula for the Anova-model.

[64] PL = rot(PL,"drei",4´1´2´3,1..4)

PL             |    [Vertreter]     [const]      [preis]        [VerkFörd]  |  [Ab-satz]    
------         +  ------         ------         ------         ------       +  ------      
const          |         3.131          0.444          0.000          0.000 |         0.000
preis          |        32.569          6.706          4.156          0.000 |         0.000
VerkFörd       |      3283.680        -70.136        498.741       1461.604 |         0.000
Vertreter      |       298.948          0.000          0.000          0.000 |         0.000
------         +  ------         ------         ------         ------       +  ------      
Absatz         |      5497.395        112.162        129.903        700.802 |       310.685
------         +  ------         ------         ------         ------       +  ------      

and get also the squares of the entries:

[67] PL2 = PL  ^#  2

PL²            |    [Vertreter]     [const]      [preis]        [VerkFörd]  |  [Ab-satz]    
------         +  ------         ------         ------         ------       +  ------      
const          |         9.803          0.197          0.000          0.000 |         0.000
preis          |      1060.763         44.964         17.275          0.000 |         0.000
VerkFörd       |  10782552.562       4919.035     248743.030    2136285.373 |         0.000
Vertreter      |     89370.000          0.000          0.000          0.000 |         0.000
------         +  ------         ------         ------         ------       +  ------      
Absatz         |  30221348.172      12580.307      16874.660     491123.992 |     96524.870
------         +  ------         ------         ------         ------       +  ------      

We do this again for the remaining two predictors - rotating means:isolating one predictor's coordinate and square the related coordinate in "Absatz" to get the partial squaresums.

[106] PL = rot(PL,"drei",3´4´1´2,1..4)

PL             |  [VerkFörd]      [Vertreter]     [const]      [preis]      |  [Ab-satz]    
------         +  ------         ------         ------         ------       +  ------      
const          |         2.824          1.352          0.443          0.000 |         0.000
preis          |        29.909         12.935          6.760          3.934 |         0.000
VerkFörd       |      3629.394          0.000          0.000          0.000 |         0.000
Vertreter      |       270.472        127.337          0.000          0.000 |         0.000
------         +  ------         ------         ------         ------       +   ------      
Absatz         |      5271.652       1708.854        144.040       -103.379 |       310.685
------         +  ------         ------         ------         ------       +   ------      


[109]   PL2 = PL  ^#  2

PL²            |  [VerkFörd]      [Vertreter]     [const]      [preis]      |  [Ab-satz]    
------         +  ------         ------         ------         ------       +  ------      
const          |         7.976          1.828          0.197          0.000 |         0.000
preis          |       894.523        167.315         45.691         15.474 |         0.000
VerkFörd       |  13172500.000          0.000          0.000          0.000 |         0.000
Vertreter      |     73155.189      16214.811          0.000          0.000 |         0.000
------         +  ------         ------         ------         ------       +   ------      
Absatz         |  27790310.299    2920182.204      20747.480      10687.148 |     96524.870
------         +  ------         ------         ------         ------       +   ------      




[112] PL = rot(PL,"drei",2´3´4´1,1..4)

PL             |  [preis]        [VerkFörd]      [Vertreter]     [const]    |  [Ab-satz]    
------         +  ------         ------         ------         ------       +  ------      
const          |         3.132          0.064          0.372         -0.223 |         0.000
preis          |        33.511          0.000          0.000          0.000 |         0.000
VerkFörd       |      3239.213       1637.071          0.000          0.000 |         0.000
Vertreter      |       290.546         24.745         65.884          0.000 |         0.000
------         +  ------         ------         ------         ------       +   ------      
Absatz         |      5381.439       1039.218        822.122       -161.799 |       310.685
------         +  ------         ------         ------         ------       +   ----

[115]   PL2 = PL  ^#  2

PL²            |  [preis]        [VerkFörd]      [Vertreter]     [const]    |  [Ab-satz]    
------         +  ------         ------         ------         ------       +  ------      
const          |         9.808          0.004          0.138          0.050 |         0.000
preis          |      1123.003          0.000          0.000          0.000 |         0.000
VerkFörd       |  10492498.904    2680001.096          0.000          0.000 |         0.000
Vertreter      |     84416.914        612.338       4340.748          0.000 |         0.000
------         +  ------         ------         ------         ------       +   ------      
Absatz         |  28959889.834    1079973.646     675884.824      26178.826 |     96524.870
------         +  ------         ------         ------         ------       +   ------ 

All four predictors have now shown their "usefulness"-coefficient, we look at it below.


Finally in this sequence of matrix-calculations we compute the Regression-coefficients B for the items using the inversion of the upper-left submatrix of the PL . This gives us the columns in the metric of the predictors (see the "1"-coordinates in their own columns).

Note that the vectorspace has now non-orthogonal axes (the items which provide the metric are correlated)

[76] PLInv = inv(PL[1..4,1..4])
[77] PLInv = merge(einh(5)*sqrt(N),PLInv)
[78] B = PL * PLInv

B              |  const          preis          VerkFörd       Vertreter    | Absatz      
------         +  ------         ------         ------         ------       + ------      
const          |         1.000          0.000          0.000          0.000 |        0.000
preis          |         0.000          1.000          0.000          0.000 |        0.000
VerkFörd       |         0.000          0.000          1.000          0.000 |        0.000
Vertreter      |         0.000          0.000          0.000          1.000 |        0.000
------         +  ------         ------         ------         ------       + ------      
Absatz         |       725.548        -26.281          0.479          8.387 |      982.471
------         +  ------         ------         ------         ------       + ------      


To have the set of coefficients for the SS I and the SS III type (as given by R and SPSS ) together we copy all rows of the PL² which contain the "Absatz"-squaresums:

PL²            |  [const]         [preis]        [VerkFörd]     [Vertreter] |  [Ab-satz]    
Absatz         |  29797664.400      31659.900     831737.936      80864.894 |     96524.870

PL²            |  [Vertreter]     [const]        [preis]        [VerkFörd]  |  [Ab-satz]    
Absatz         |  30221348.172      12580.307      16874.660     491123.992 |     96524.870

PL²            |  [VerkFörd]      [Vertreter]    [const]        [preis]     |  [Ab-satz]    
Absatz         |  27790310.299    2920182.204      20747.480      10687.148 |     96524.870

PL²            |  [preis]         [VerkFörd]     [Vertreter]    [const]     |  [Ab-satz]    
Absatz         |  28959889.834    1079973.646     675884.824      26178.826 |     96524.870

The solution for SS I is now the first row of coefficients, and that of SS III the fourth column:

               |  SS I (`R`)    SS III  (`SPSS`)
------         +  -----------     ------------      
const          |                   26178.826
preis          |    31659.900      10687.148
VerkFörd       |   831737.936     491123.992
Vertreter      |    80864.894      80864.894
------         +  -----------     ------------      
Residual       |    96524.870      96524.870
------         +  -----------     ------------      

And the difference is, that the SPSS default -(Type SS III) gives us the set of "usefulnesses" of the items (and also their F-value and p-value) and the "Anova"-procedure in R (using the Type SS I definition by default) gives us the set of so-to-say "explained partial Squaresums" in hierarchical order (which has been defined by the order on the formula-string for the procedure).

(P.S. Of course show the triangular PL-matrices immediately, that if the predictors are uncorrelated and thus the upper part of the PL is even diagonal, that type SS I and type SS III coefficients become identical)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.