I've run two different linear mixed effects models on the same data and got two different estimates for the gradient of the longitudinal variable. e.g.

model 1 has estimate 30 with standard error 5. model 2 has estimate 40 with standard error 4.

I'm interested in calculating a p value for the probability that the models are different, from the estimate and standard error. How do I do this? I'm aware that checking for overlap in the 95% confidence intervals is a bad idea, and that overlapping 83% CIs are a better test, but would like to be able to quantify this with a p value.

  • $\begingroup$ AIC might be more appropriate for comparing the two models. $\endgroup$ – Dedekind Cuts Jul 8 '20 at 14:48
  • $\begingroup$ AIC makes a very different kind of comparison to the kind of comparison I'm looking to make. Whether a test is appropriate is dependant on the question you're actually trying to answer with the test. I'm looking for a way to compare how different the estimates are, rather than compare model quality. $\endgroup$ – E. Rei Jul 8 '20 at 15:03
  • 1
    $\begingroup$ To clarify what you're attempting: When you say you've run two different models, do you mean that you have fitted two models with different terms, e.g. as a simple example for response variable y, and explanatory variables x, z and v, did you fit models y = a.x + b.z + c as opposed to y = l.x + m.v + e, and now you want to compare coefficients a and l? Or have you parameterised a model twice with the same terms (e.g. y = a.x + b.z + c but presumably something more complicated) using different fitting processes each time, and you now want to compare your two estimates of a? $\endgroup$ – Izy Jul 9 '20 at 10:25
  • 3
    $\begingroup$ I suggest you try to formulate what null hypothesis you're actually trying to test. $\endgroup$ – Dedekind Cuts Jul 9 '20 at 17:28
  • 1
    $\begingroup$ your model estimates are different, you know that. what do you want to test? $\endgroup$ – carlo Jul 14 '20 at 9:27

It is unclear from your question how these models relate to one another. However, the way you perform a statistical test for equivalence of models is to pose them as nested models and then perform a statistical test to see whether the additional terms in the more complex model are "zero". For example, suppose you have two linear models like this:

$$\begin{matrix} \text{Simple Model} & & y_i = \beta_0 + \beta_1 x_{1,i} + \cdots \beta_k x_{k,i} + \epsilon_i, \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \\[6pt] \text{Complex Model} & & y_i = \beta_0 + \beta_1 x_{1,i} + \cdots \beta_k x_{k,i} + \beta_{k+1} x_{k+1,i} + \cdots \beta_{k+m} x_{k+m,i} + \varepsilon_i. \\[6pt] \end{matrix}$$

This is an example where your simple model is "nested" within your more complex model. You can test the equivalence by testing the hypotheses:

$$H_0: \beta_{k+1} = \cdots = \beta_{k+m} = 0 \quad \quad \quad \quad \quad H_A: H_0 \text{ is false}.$$

The null hypothesis here reduces the complex model to the simple model, which is equivalent to saying that the models are really the same. Using a classical hypothesis test we can see if there is evidence to reject the null hypothesis that both models are the same.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.