Misspecified levels in multilevel mixed logit model

Question

I'm estimating a couple 3 level logit models using Stata 12 and am faced with a dilemma about how (or if) I should specify my third level.

The data is court cases nested within judges nested within circuits. The only random effect is the intercept. There's over a million cases, about 500 judges, and 20 circuits. The problem with the 3rd level is that some judges operate in multiple circuits (multiple membership). This creates a dilemma and this is where I'd love to hear feedback from those who know more.

I can specify the model with k independent variables (and several interactions) in stata syntax as:

xtmelogit y x1 x2 x3##x4 xk ||_all:  R.circuit || judge: , intp(1)

The intp(1) sets the integration points to 1 which results in the Laplacian approximation which, I've read, results in more or less accurate coefficients but potentially serious bias in the variance components. I've had a model such as this running for a week now and it's on it's second iteration so I'm guessing if it converges it'll take at least another week. I suspect that IF I were to specify the standard 7 integration points the model would take months to complete. And forget checking the results for sensitivity to integration points.

On the otherhand, I can estimate the model as 2 levels, the syntax is as follows:

xtmelogit y x1 x2 x3##x4 xk ib11.circuit || judge: , intp(7)

This model includes the circuits as dummies (circuit 11 as the reference) nested within judges. This is the model I've been using. Problem is, the levels are obviously misspecified. The upside is I have more confidence in the coefficients and the variance components because I can use a reasonable number of integration points and I have verified that the results are not sensitive to the number of integration points used. But I'm wondering what impact the level misspecification is having. It's certainly screwing with the variance components but the alternative model isn't likely to have accurate variance components anyway. The real concern for me is whether the random intercept and coefficients are biased in some unforeseen way.

Does anyone have any input or sage advice?

Edited to add: I've thought more about this and the core issue is simply that I want to control for the effect of circuit. There have been other papers published with this data where cases are nested within circuits (2 levels) and they've shown an important effect of circuit on the outcome. In my model with the circuit dummies they have substantial odds ratios with some as high as 7, 8, or 9 (as it turns out the 11th circuit is the most lenient). I don't want to ignore this effect but rather control for it as a nuisance variable, I can live with it being mis-estimated as long as that doesn't bias my intercepts and my coefficients. I guess what I'm asking, then, is what's the least damning way of doing this since the technically correct way appears to be out the window?

Answer 1

Since it looks like no one is going to bite, here's how I've settled the issue.

I ran a unconditional 3 level logit model with the levels appropriately specified (laplace approximation)

xtmelogit outcome || _all:R.circuit || judge: ,intp(1)

from this I predicted the random effect for both circuits and judges.

predict reffect* , reffect

Next I generated the odds for each circuit

gen circuitodds=exp(reffect1)

reffect1 is the random effect for the circuit and reffect2 is for the judge level - Stata names them sequentially according to the order the levels are specified.

Next I estimated an unconditional 2 level model with the circuit represented by dummy variables

xtmelogit outcome ib11.circuit ||judge: , intp(11)

This gets me an odds ratio for each circuit, but Stata doesn't report the baseline odds (probably because it would be a function of the random effects which aren't actually estimated as part of the model - only their distribution is estimated).

So I then plugged all this into excel and checked to see if I could use the odds ratios from the 2 level model to basically recreate the predicted odds for each circuit from the 3 level model.

Since circuit 11 is the reference I simply took the odds ratio for each circuit (from the 2 level model) and multiplied that times the predicted odds for circuit 11 (from the 3 level model) and compared the resultant odds with the predicted odds for that circuit (again from the 3 level model).

Basically, I'm checking to see if the odds ratios from the incorrect model accurately quantify the relationship between the odds for each circuit from the correctly specified model. The results are very very close which makes me think that specifying the model as 2 levels with the cross-nested 3rd level represented by dummy variables is a satisfactory solution.

The assumption, of course, is that the comparison of the unconditional models is representative of what I would see IF I could compare the full models.

I'd be happy to hear if anyone sees any flaw in this method of validation or if anyone can offer a better way of checking my model (mis)specification.