# Inconsistency in mixed-effects model estimation results (Stata and SPSS)

For my thesis there's a big chance that I will need some sort of mixed-effects specification. I have some (non-syntax) experience with SPSS but feel that it won't suffice for my analysis. I have very basic knowledge in Stata and decided to experiment more with that package.

I decided to try and replicate results from SPSS in Stata for a basic model. I have data on 4059 students in 65 schools, investigating the influence of entry level score (standlrt) of students on their final exam score (normexam).

In a previously followed course which had a brief introduction to multilevel modeling, my teacher provided me with a syntax in SPSS.

MIXED
normexam  WITH standlrt
/FIXED = standlrt
/PRINT = SOLUTION TESTCOV
/RANDOM INTERCEPT standlrt  | SUBJECT(school) COVTYPE(VC) .


Now I tried replicating these results in Stata but the results are not consistent. Magnitude and sometimes even sign of the betas differ.

First I use xtset school to indicate that my data is clustered. Then I use

xtmixed normexam standlrt || school: standlrt .

What may be the cause of these inconsistent results?

ps. this is not homework, and I hope I specified my first question properly.

pps. a possibility may be that the 'problem' has multiple optima but I don't think this is the case in such a basic model, also because it's an uni-variate regression. Also, the iterative procedures performed while estimating may have different results, but I only think this would have big effects like sign changes.

EDIT

This is my Stata output

xtmixed normexam standlrt || school: standlrt

Mixed-effects REML regression                   Number of obs      =      4059
Group variable: school                          Number of groups   =        65

Obs per group: min =         2
avg =      62.4
max =       198

Wald chi2(1)       =    768.21
Log restricted-likelihood = -4667.8385          Prob > chi2        =    0.0000

------------------------------------------------------------------------------
normexam |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
standlrt |   .5570213   .0200971    27.72   0.000     .5176317    .5964108
_cons |  -.0080944   .0400842    -0.20   0.840     -.086658    .0704691
------------------------------------------------------------------------------

------------------------------------------------------------------------------
Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
school: Independent          |
sd(standlrt) |   .1214197   .0191066      .0891958    .1652852
sd(_cons) |   .3032317   .0309434      .2482638    .3703699
-----------------------------+------------------------------------------------
sd(Residual) |   .7440605   .0083943      .7277885    .7606962
------------------------------------------------------------------------------
LR test vs. linear regression:       chi2(2) =   438.60   Prob > chi2 = 0.0000


And this is my SPSS output

-2 Restricted Log Likelihood    9335,677

Type III Tests of Fixed Effects(a)
|---------|------------|--------------|-------|----|
|Source   |Numerator df|Denominator df|F      |Sig.|
|---------|------------|--------------|-------|----|
|Intercept|1           |60,466        |,041   |,841|
|---------|------------|--------------|-------|----|
|standlrt |1           |56,936        |768,207|,000|
|---------|------------|--------------|-------|----|
a. Dependent Variable: normexam = final exam scores.

Estimates of Fixed Effects(a)
|---------|--------|----------|------|------|----|-----------------------------------|
|Parameter|Estimate|Std. Error|df    |t     |Sig.|95% Confidence Interval            |
|         |        |          |      |      |    |-----------------------|-----------|
|         |        |          |      |      |    |Lower Bound            |Upper Bound|
|---------|--------|----------|------|------|----|-----------------------|-----------|
|Intercept|-,008094|,040084   |60,466|-,202 |,841|-,088262               |,072073    |
|---------|--------|----------|------|------|----|-----------------------|-----------|
|standlrt |,557021 |,020097   |56,936|27,717|,000|,516777                |,597266    |
|---------|--------|----------|------|------|----|-----------------------|-----------|
a. Dependent Variable: normexam = final exam scores.

Covariance Parameters

Estimates of Covariance Parameters(a)
|-------------------------------------|--------|----------|------|----|-----------------------------------|
|Parameter                            |Estimate|Std. Error|Wald Z|Sig.|95% Confidence Interval            |
|                            |--------|        |          |      |    |-----------------------|-----------|
|                                     |        |          |      |    |Lower Bound            |Upper Bound|
|----------------------------|--------|--------|----------|------|----|-----------------------|-----------|
|Residual                             |,553626 |,012492   |44,319|,000|,529676                |,578659    |
|----------------------------|--------|--------|----------|------|----|-----------------------|-----------|
|Intercept [subject = school]|Variance|,091949 |,018766   |4,900 |,000|,061635                |,137174    |
|----------------------------|--------|--------|----------|------|----|-----------------------|-----------|
|standlrt [subject = school] |Variance|,014743 |,004640   |3,177 |,001|,007956                |,027319    |
|----------------------------|--------|--------|----------|------|----|-----------------------|-----------|
a. Dependent Variable: normexam = final exam scores.


As you can see, the log likelihoods are the same. Additionally, the fixed effects tables are the same. However the random effects are different. I'm not very skilled in interpretation yet but the results seem to differ.

These are the settings for the variance-covariance matrix

 Model
covariance(vartype)    variance-covariance structure of the random
effects

vartype                  Description
-------------------------------------------------------------------------
independent              one variance parameter per random effect, all
covariances zero; the default unless a factor
variable is specified
exchangeable             equal variances for random effects, and one
common pairwise covariance
identity                 equal variances for random effects, all
covariances zero; the default for factor
variables
unstructured             all variances and covariances distinctly
estimated


And I read online that COVTYPE(VC) requests the default (variance component) structure for random effects, which assumes all random effects are independent.

-
Just to clarify: which results are not consistent? The estimated parameters of the fixed effects (standlrt) and the estimates of the variance/covariance matrix of the random effects? Remember that by default Stata gives you the estimates of the standard deviations of the random effects. Are you sure you're comparing apples with apples? (It may sound like a stupid question, but it's good to rule out the simplest sources of error first) –  boscovich Apr 29 '12 at 16:53
The last line of the SPSS code includes a particular structure of the random effects. I am not sure but this looks like SAS, where "VC" means that the random intercept is independent of the random effect "standlrt". Are you sure your STATA code includes such a structure too ? –  Stéphane Laurent Apr 29 '12 at 17:32
Please post the full output from both packages. –  StasK Apr 29 '12 at 17:49
By default in this case Stata uses a structure that "allows a distinct variance for each random effect within a random-effects equation and assumes that all covariances are zero". Stata calls it an independent structure, but afaik it's also known to be a variance components (VC) structure. So, it should be the same. –  boscovich Apr 29 '12 at 18:35
Thanks for the replies so far. I added the outputs. –  C. Pieters Apr 29 '12 at 20:49

.1214197 ^ 2 = .014742744 (standlrt)

@C.Pieters The Stata postestimation command estat recov displays the random-effects covariance/correlation matrix, so that you don't have to square the results by hand (if you want the variances). –  boscovich Apr 29 '12 at 21:13