Comparing coefficients of two different probit models---is this “bad statistics”?

Apologies for any stupid mistakes, or if the answer to this question is trivial: I have no formal statistical training. Long story short: can we meaningfully compare coefficients of two different probit models?

In "Learning and Opinion Change, Not Priming: Reconsidering the Priming Hypothesis", Lenz (2009) runs two probit models and compares the difference in the coefficients of the independent variable.

• Dependent variable: Support for Labour $$Y_t$$
• Independent variable: Attitude towards European Integration $$x_t$$
• Control variables: various demographic controls, $$\phi_t$$

The regression he runs takes the form $$Y_{t} = \beta_0' x_{t} + \phi_{t}$$, but this is a probit model, so something like

$$Pr(Y_{1994} | X_{1994}) = \Phi(X_{1994}^T \beta_0)$$ $$Pr(Y_{1997} | X_{1997}) = \Phi(X_{1997}^T \beta_0')$$

He then compares $$\beta_0' - \beta_0$$ and checks if the difference is significant (see "Diff." column), and then concludes that only in the "Learned from" case is there a significant difference. I've reproduced the results here (see third row)

My question here is whether we can meaningfully compare coefficients of two different probit models. As Mood (2010) in "Logistic Regression: Why We Cannot Do What We Think We Can Do, and What We Can Do About It" points out, odds ratios and log-odds ratios are not comparable across groups or across time "without assuming that the unobserved heterogeneity is the same across compared points in time".

For example, in a model where we want to study the difference in the effect of one variable between different ethnic groups, between men and women, or between different years, the estimate will depend on the extent to which the model predicts the outcome differently in the different categories. For example, we might be interested in how the effect of school grades on the probability of transition to university, controlling for the student’s sex, varies over time or between countries. A weakening of LnOR or OR for grades between two points in time can mean that the effect of grades is diminishing over time, but it can also mean that the importance of sex for educational choice has decreased over time.

In this case, while it's true that the importance of the independent variable could have increased, couldn't it also be the case that the importance of those various demographic controls have increased? Is Lenz justified in claiming that the independent variable becomes more explanatory in the "Learned from" case? Furthermore, do omitted variables also make a difference (say if Lenz fails to control completely for all confounders)?

Thanks very much for reading this and please drop a comment if this is unclear in any way.

In principle, the practice is wrong for the same reasons Mood laid out. In practice, it may be justifiable, especially when model quality is similar between the two models.

But if some of the variables become more important over time, and the same variables are in both models, then model quality will be different over time making log odds incomparable.

However, this issue does not affect p-values within a single model. So one can note that a variable has a large p-value in one model, then a small one in another. This is only trivially instructive though since the difference between a value that is statistically significant and one that is not statistically significant is not necessarily statistically significant (Gelman & Stern, 2006).

In summary, one can easily say that Lenz should have estimated the difference as an interaction within a single model.

Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60(4), 328–331. doi:10.1198/000313006X152649

EDIT

Lenz can stack both datasets then estimate the following right hand side:

$$$$\Phi \big( \gamma_0 + \gamma_1 d_{97} + \gamma_2 x + Z\beta + d_{97} \times (\gamma_3 x_t + Z\delta) \big)$$$$

where $$d_{97} = \mathbf{1}(\mathrm{year} = 97)$$ and $$Z$$ is the matrix of covariates.

In this setup, $$\gamma_3$$ is of interest since it captures the differential effect of $$x$$ in 1997 from the effect of $$x$$ in 1994 $$(\gamma_2)$$.

I do not know the source data but if there are repeated cases in 1994 and 1997, then stacking the data will violate the independence assumption of the probit regression model. So the investigator might need GEE or cluster robust or multilevel approach to account for the independence violation.

• @Kodiologist thanks for the citation, didn't know one existed. Although I feel it is an obvious point without need for a reference, my experience with researchers reminds me that I'm wrong. – Heteroskedastic Jim Dec 6 '18 at 17:53
• Thanks Jim for the answer! Unfortunately, please let me ask more questions to clarify exactly what you meant. 1. "This issue does not affect p-values within a single model.... in one model, then a small one in another." If you only have a single model, why would there be "one model" and then "another"? 2. If the difference between a value that is s.s. and one that isn't is not necessarily s.s., doesn't this mean Lenz's methodology doesn't work? 3. Exactly what sort of model/interaction term did you envisage---could I trouble you to write out the regression equation, please? – Lieu Zheng Hong Dec 6 '18 at 18:06
• @LieuZhengHong I'll provide the equation in an edited answer in some time. I'll clarify the point about Lenz here. The p-values within each of Lenz's models are fine. So if something is s.s., Mood does not provide a reason to doubt it. But you cannot eyeball compare s.s. across models. Normally, the correct approach would be what Lenz does: statistically compare using the coefficients and standard errors. But Mood's work suggests this is problematic. That was my point. Hope this clarifies this question. – Heteroskedastic Jim Dec 6 '18 at 18:33
• Dear @Heteroskedastic Jim, thank you for the equation. Before I accept your answer I would like to ask one final question. You reference Gelman and Stern (2006) and say that the difference between value that is s.s. and one that isn't is not necessarily s.s. But then that is exactly what Lenz does, isn't it? Why do you then say it is the correct approach in your previous comment? – Lieu Zheng Hong Dec 12 '18 at 2:00
• @LieuZhengHong Lenz conducted a statistical test to claim differences. This would be fine but for Mood's problem. The problem Gelman and Stern talk about is based on eyeball comparisons. – Heteroskedastic Jim Dec 12 '18 at 2:38