Mixed-effect models for repeated measures - problems with the "Significance level"

Question

I am trying to fit a linear mixed-effect model to my dataset to see the relationship between a self-reported questionnaire and some physiological data.

I've created a first model including all the features as fixed variables, a random inercept that varies with subject, and the questionnaire responses as response variable. I've verified the significance of each one of the variables by looking at the p-values.

In a second attempt, I created another model, this time leaving aside all the variables that were not significant (p<0.05) in the first model. Now, all the p-values have changed, and some of those who seemed to be significant in the first model, seem not to be anymore.

I don't understand why this happens, neither how can I choose significant variables to create a good model.

Could you help me, please? Thanks!!

Answer 1

It certainly ought to change. When you remove a variable from a model, you are no longer controlling for that variable. Unless all the variables are strictly orthogonal, that will change everything about the model.

Building a model is part art, part science. There is no automated way to do it well. However, using a method such as LASSO or LAR is better than most others, if you insist on automating the process