Comparing coefficients of two different probit models---is this "bad statistics"? Apologies for any stupid mistakes, or if the answer to this question is trivial: I have no formal statistical training. Long story short: can we meaningfully compare coefficients of two different probit models?
In "Learning and Opinion Change, Not Priming: Reconsidering the Priming Hypothesis", Lenz (2009) runs two probit models and compares the difference in the coefficients of the independent variable.


*

*Dependent variable: Support for Labour $Y_t$

*Independent variable: Attitude towards European Integration $x_t$

*Control variables: various demographic controls, $\phi_t$
The regression he runs takes the form $Y_{t} = \beta_0' x_{t} + \phi_{t} $, but this is a probit model, so something like
$$Pr(Y_{1994} | X_{1994}) = \Phi(X_{1994}^T \beta_0) $$
$$Pr(Y_{1997} | X_{1997}) = \Phi(X_{1997}^T \beta_0') $$
He then compares $\beta_0' - \beta_0$ and checks if the difference is significant (see "Diff." column), and then concludes that only in the "Learned from" case is there a significant difference. I've reproduced the results here (see third row)

My question here is whether we can meaningfully compare coefficients of two different probit models. As Mood (2010) in "Logistic Regression: Why We Cannot Do What We Think We Can Do, and What We Can Do About It" points out, odds ratios and log-odds ratios are not comparable across groups or across time "without assuming that the unobserved heterogeneity is the same across compared points in time".

For example, in a model where we want to study the difference in the effect of one variable between different ethnic groups, between men and women, or between different years, the estimate will depend on the extent to which the model predicts the outcome differently in the different categories. For example, we might be interested in how the effect of school grades on the probability of transition to university, controlling for the student’s sex, varies over time or between countries. A weakening of LnOR or OR for grades between two points in time can mean that the effect of grades is diminishing over time, but it can also mean that the importance of sex for educational choice has decreased over time.

In this case, while it's true that the importance of the independent variable could have increased, couldn't it also be the case that the importance of those various demographic controls have increased? Is Lenz justified in claiming that the independent variable becomes more explanatory in the "Learned from" case? Furthermore, do omitted variables also make a difference (say if Lenz fails to control completely for all confounders)?
Thanks very much for reading this and please drop a comment if this is unclear in any way.
 A: In principle, the practice is wrong for the same reasons Mood laid out. In practice, it may be justifiable, especially when model quality is similar between the two models.
But if some of the variables become more important over time, and the same variables are in both models, then model quality will be different over time making log odds incomparable.
However, this issue does not affect p-values within a single model. So one can note that a variable has a large p-value in one model, then a small one in another. This is only trivially instructive though since the difference between a value that is statistically significant and one that is not statistically significant is not necessarily statistically significant (Gelman & Stern, 2006).
In summary, one can easily say that Lenz should have estimated the difference as an interaction within a single model.
Gelman, A., & Stern, H. (2006). The difference between "significant" and "not significant" is not itself statistically significant. The American Statistician, 60(4), 328–331. doi:10.1198/000313006X152649

EDIT
Lenz can stack both datasets then estimate the following right hand side:
\begin{equation}
\Phi \big( \gamma_0 + \gamma_1 d_{97} + \gamma_2 x + Z\beta + d_{97} \times (\gamma_3 x_t + Z\delta) \big)
\end{equation}
where $d_{97} = \mathbf{1}(\mathrm{year} = 97)$ and $Z$ is the matrix of covariates.
In this setup, $\gamma_3$ is of interest since it captures the differential effect of $x$ in 1997 from the effect of $x$ in 1994 $(\gamma_2)$.
I do not know the source data but if there are repeated cases in 1994 and 1997, then stacking the data will violate the independence assumption of the probit regression model. So the investigator might need GEE or cluster robust or multilevel approach to account for the independence violation.
