Does leaving out an important predictor in a mixed linear model violate the independence assumption?

Question

I have data from an experiment with 3 groups, measured at 4 time points, where each subject performed a task where 2 factors are manipulated: valence (3 levels) and predictability (2 levels).

I know that valence has a strong effect in the outcome variable. I want to understand the effect of the other variables (group, time point, predictability) and their interactions.

It has been suggested to me that I can build a mixed model with group * timepoint * predictability, leaving out valence. I am worried, however, that that would violate the independence assumption used for the tests (e.g. ANOVA) that I would run on the model. That is, if I have two samples of the same valence, I can predict that they will be more similar than two samples with different values. Is that correct or am I mixing things up?

Sorry if I am mixing up concepts or terminology, I am still new to analyzing data. I also did not design the experiment.

Answer 1

We shouldn't leave out any variables that have a (significant) effect if we are interested in the causal effect and the design is not orthogonal.

If a variable is left out and this variable is correlated with an included variable, then the coefficient of the included variable includes part of the effect of the left out variable. This is standard missing confounder, omitted variable problem with typical text book case in Simpson's paradox.

"That is, if I have two samples of the same valence, I can predict that they will be more similar than two samples with different values."

This is the idea behind matching estimator, like for example propensity score matching, in that we want to remove the effect of left out confounders by comparing only similar individuals.

On the other hand, if the design is orthogonal with respect to the left out variables, i.e. those are not correlated with the included variables, then there is no distorting effect on the included coefficients.

Answer 2

First of all independence has nothing to do with the fact of dropping or adding a variable; remember that there is not a such test to prove the independence assumption, since it is related with your experimental design. Imagine this silly (but illustrative) example: “A political scientist is interested on people’s opinion about candidates for the next election in her county. For this study 100 participants were chosen randomly and they were interviewed and recorded during at most 5 minutes individually in a meeting room. During the study, the participants were asked to arrive at different times because the waiting room has only a capacity for 15 people. Also, it was possible for the participants on the waiting room to listen the opinion of the person who was being recorded at that moment in the meeting room.” On this case the independence assumption was totally violated since a participant’s opinion could be clearly influenced by the previous one. As you can see the data analysis has not even started and the independence assumption is already breached. So, remember independence has to do with your experimental design and not with your variables.

On the other hand, if you want to drop out the variable Valence you can do it and you will not violate the independence assumption, however it is possible that you will lose valuable information. Suppose that you find a model “A” that uses all the variables: valence, group, timepoint, predictability and some of their interactions (all significant). Suppose too that you obtain another model “B”, which only drops valence, you also observed that the coefficients for effects and interactions are significant. In theory both models would be a valid way to explain the variability in your response variable. Which one would be better? You would have to analyze how much variability is being explained by your independent variables in “A” and then repeat the same for “B”, if the difference is noticeable then valence would explain a great proportion of the total variability.

Personally, I would test more models using stepwise regression. At the end if you have several models you can also check for goodness of fit metrics like AIC or R² in order to decide which one is better for you.

I hope this could be useful for you.

Good luck.

Answer 3

You can still see the effects of group + timepoint + predictability and their interactions even when valence is left in the model. Keep in mind that interpreting interaction terms of continuous variables is quite difficult to interpret.

The condition of independence is that we are assuming that our data is a random sample from the population of interest. As a contrast to this condition, suppose we are interested in the number of pieces in a jigsaw puzzle and the time it takes to complete it. If all our data come from one person (e.g., multiple puzzles), who happens to be very good at jigsaw puzzles. Then our estimate of the line will be much lower than it should be, because this person will finish all the puzzles quickly, i.e. small values for $y_i$. However, had our data been independent, then we have the chance of also getting someone who is very bad at jigsaw puzzles and things even out in some way.

So what you need to ask yourself is, does leaving out valence violate the possibility of getting a random sample from the population of interest? (The answer is no.)

If your goal here is to build a linear model, I would suggest just performing some type of subset model selection. antoniom suggested stepwise regression, this would work...

library(MASS)
fit <- lm(y~(x1+x2+x3)^2 + x4, data = mydata)
step <- stepAIC(fit, direction="both")
step$anova # display results

Other options could be nested F-tests backward selection...

lm1 <- lm(y~(x1+x2+x3)^2 + x4, data = mydata)
drop1(lm1, test = "F")

If your goal is to predict, you can do CV with best subset...

library(leaps)
regfit.best=regsubsets(y~(x1+x2+x3)^2 + x4,data=Hitters[train ,], nvmax=)
test.mat=model.matrix(Salary∼.,data=Hitters [test ,])
val.errors =rep(NA ,nvmax)
for(i in 1:nvmax){coefi=coef(regfit.best ,id=i)
pred=test.mat[,names(coefi)]%*%coefi
val.errors[i]=mean(( Hitters$Salary[test]-pred)^2)
}
which.min(val.errors)

this gives best subset model's number of coefficients, then to find the model...

coef(regfit.best ,which.min(val.errors))

All of these methods will tell you valuable information about the relationships between variables without having to remove valence in the beginning.

Answer 4

No, leaving out a dependent variable does not violate any independence assumption.

Consider a very simple example where I'm examining the effect of height, weight, baseball batting average on points scored per basketball game. We know that height is a great predictor of basketball skills, but maybe we want to see how batting average and weight do as a predictor. There is no prohibition against dropping any number of variables in order to build a model that examines the relationship between two variables.

Think of it like this: there are millions of data out there that predict things like my income, height, weight and country of birth, however, it's possible to build a highly accurate model in which you predict any of these with very minimal data or with loads of data. There could be a "valence" out there that predicts my height or country of birth perfectly!

The thing to remember about modeling/prediction in general is that your goal is to make predictions better than you would by random. If you're doing that part correctly (i.e. cross validating, testing on "wild" data) then you've met the goals of the task.