Understanding Mixed Effect Models visually with some questions on P-value calculations

Question

I've been tackling the concept of Mixed Effects Models on and off for the last 9 months. Every time I would give up and come back later to try and understand it again with my basic statistics knowledge (I come from a medicine background, so I have no high-level mathematics knowledge involving calculus and derivations for proofs, just intuitive understandings) in a never ending cycle. I'd see all these words about random and fixed effects and while on a superficial level they would make sense, my understanding would always break down when I'd try to dig deeper and understand how the P-values ARE CALCULATED. That is until recently where it has kind of started to click more than usual. The reason being VISUALISATION of how data in a mixed effects model works. I feel like I am almost there and have some questions. The post will be long with a lot of pictures(for visualization) but the questions will most likely be very simple for any of you with a formal training in statistics. I will start off with the basics and build my way up to questions which I consider more and more difficult.

I will be using the example from the following video: https://www.youtube.com/watch?v=4bGG02Jsjyc which helped me immensely in developing my intuition of mixed effects models through visualization (the video specifically discusses linear mixed effect models)

Let us begin with the data that's presented in the video example which involves 4 subjects following the same diet (Fig.1)

Fig.1 (Subjects and their weights in Repeated Measurements)

Let us now proceed and make a Simple Linear Regression for these data points (Fig.2) (Yes, I understand this is incorrect as our example violates the rule of complete independence between all data points but it is purely for explanation purposes just as was done in the video)

Fig.2 (A simple linear regression for all the data points with a P-value for Hypothesis testing)

We then proceed with hypothesis testing whereby the Null Hypothesis indicates that the diet actually has no effect on weight loss (which is just a different way of saying the Slope Coefficient = 0 and not actually -3.25 ). We get a p-value=0.372 which shows that the slope is indeed not statistically significant(<0.05) and that this slope coefficient we got from the sample could have likely happened due to random chance and not due to an actual effect of the diet.

The way we calculated this was through calculating a t-statistic for a one sample t-test for the regression line(Fig.3)

Fig.3 (Calculating a T-statistic for Hypothesis testing of Diet Effect in Simple Linear Regression)

However, what's weird is that every single person actually did demonstrate a weight loss effect in this experiment which makes the claim that the effect of the diet(the slope) is not statistically significant a bit dubious. This is indeed where we realize that a Simple Linear Regression is the wrong technique to use. The 2 reasons being that:

1. Simple Linear Regression requires that all data points are independent which is not the case with our Repeated measures data on each of the 4 subjects.

2. Because Simple Regression assumes that all data points are independent, in our case the Standard Error becomes very large because the data points are so far away from the Simple Regression Line against which their error/residual has to be measured. This is according to the Standard Error equation in Fig 3, specifically the (yi-ŷi)^2 part. You can also see this visually in Fig.4

Fig.4 (Large Standard Error due to data points being quite far away from Regression Line)

The large standard error gives us a result of the diet effect(slope) not being statistically significant and yet as we said, there is a visible effect of weight loss on every individual. So how do we account for this? This is where a Linear Mixed Effects Model kicks in.

Looking at our data points we can see that each subject did indeed experience a decline in their weight over time. The 2 variations between each of the subjects are

1. The weight they started this whole experiment at

2. The rate at which they lost weight (some people lost weight faster while others lost it a bit slower)

There are 2 ways of handling this in a Linear Mixed Effects Model:

1. Random Intercept + Fixed Slope

2. Random Intercept + Random Slope

Now, my first question is what is the Random Effect and what is the Fixed Effect in both of the scenarios above?

In the case of Random Intercept + Fixed Slope it makes sense that the random effect is the differing starting weights and the Fixed effect is the Fixed Slope(which is the effect of the diet)

But what about in the case of the Random Intercept + Random Slope? Is the random effect both the starting weight and the inherent randomness of the fixed effect within each individual? And the fixed effect is just the fact that weight is being lost?

Moving on, let us proceed with the Random Intercept + Fixed Slope model for our example (Fig.5)

Fig.5 (Random Intercept + Fixed Slope Model where each subject gets their own regression line which has the same slope taken from the overall sample regression line we calculated in the Simple Regression model)

If we proceed to do some hypothesis testing against the null hypothesis with our new model, we will get a p-value of <0.001 . Which shows us that the effect of the diet(slope) is indeed statistically significant unlike what the Simple Regression model told us. The reasoning behind this is that the Standard Error is significantly reduced when comparing each subjects data points to their own respective regression line and hence accounting for their inherent randomnesses. (Fig.6)

Fig.6 (Data points are much closer to their respective regression lines leading to a much smaller standard error which in turn results in a significantly lower p-value.)

My second question is: for this Random Intercept + Fixed Slope model, are we using the same formulas in Fig.3 to calculate the T-Statistic and the P-value as we did for the Simple Linear Regression model? That is, does calculating the p-value follow the same reasoning on a Random Intercept + Fixed Slope model as it does in the Simple Linear Regression model?

Third question: If we were to do a Random Intercept + Random Slope model, since each subject would have their own Slope Coefficient(Thus a total of 4 different slope coefficients) which Slope Coefficient would we be comparing with the zero slope in the Null Hypothesis? Or are we still comparing the overall sample slope coefficient (-3.125) with the zero slope? That is, we're just using the random intercept + random slope to derive the standard error but in the end we're using the -3.125 slope in our T-statistic calculation. So basically, T-score = -3.125 / (S.E of Random Intercept + Random Slope model)

Fourth question: We can see in our example that the random intercepts account for the random effect. But what is the calculation for proving that the random effects are statistically significant? How would we calculate the P-value to prove the significance of the random effects?

Finally(we're almost there I promise), moving on to the last example which is from the second part of the video series: https://www.youtube.com/watch?v=oI1_SV1Rpfc

This is the new data(Fig.7). Where now we divide 4 subjects equally into 2 DIET GROUPS

Fig.7 (Subjects following different Diets and their weights over repeated measurements)

We will proceed to make a random intercept + fixed slope model where we include the type of diet group as an effect as well (Fig.8)

Fig.8 (Random Intercept + Fixed Slope model where Diet Type is included as an effect)

We can see 3 P-values in Fig.8. The "Slope" row indicates that the Slope is statistically significant and thus the null hypothesis is rejected.

The "Diet B" row indicates that the mean weight difference of Diet Group B from A is statistically significant (P=0.003). My fifth question is how was this P-value calculated?

My final and 6th question is the "Intercept" Row uses the Diet A Group's mean weight as a baseline for the Intercept(97.962). But what does the P-value indicate here and how is it calculated? This intercept is statistically significant relative to what exactly? A null hypothesis? I can't see how that makes sense

If you've made it this far, I highly appreciate you taking the time to read my post and trying to help me out. All the best.

Answer 1

First, I don't think it's helpful to get hung up on how p values are calculated in these different models. The calculation of p values is fairly straightforward from a given coefficient and its standard error. The more important issue is what each coefficient and standard error represent in these models. Also, the p value associated with ANY intercept is almost always useless: it is telling you how confident you can be that the intercept is different from zero, regardless of whether that is a plausible or meaningful value. Stats programs tend to report p values for intercepts because they can, but they rarely tell you anything useful.

Second, I think you are getting tripped up by the admittedly confusing terminology surrounding the terms "random" and "fixed." Try thinking about it like this:

In a linear regression model with one independent variable (say "weeks") there are two key parameters: the intercept, and the slope coefficient. To say that we are treating one of these parameters as "fixed" means that we use the data to estimate a specific value for the entire sample. This is what we would do to both the intercept and slope coefficient in a single level OLS model: calculate a single intercept and a single slope coefficient for the whole model.

Now, as you note, this doesn't necessarily make sense in longitudinal data, where observations are "nested" within individuals. In particular, the idea that there would be just one intercept for the entire dataset seems wrong and might bias our estimates of both coefficients and standard errors (and thus p values). It seems more likely that we would want to allow each individual to have their "own" intercept. There are two ways of dealing with this. The first, is just to add a series of dummy variables to the model, one for each value of the person-level identifier variable (omitting one as the reference category). Basically we are treating the person ID variable the same way we would treat a categorical variables like race in a single level model), in effect, giving each individual their own intercept, which we estimate directly, just like any other coefficient. In the context of multilevel modeling this approach is called a "fixed effects model" because each intercept is estimated directly. Here you get a separate coefficient (and p value) for each intercept, but they are not very useful because they just tell you how far that intercept is away from the arbitrary reference category. The coefficient and p value for any independent variables in this model tells you the "effect" of that variable controlling for ALL person-level differences (basically, you are using each person's earlier values as their own "control"). One problem with these models though is that they don't allow you to control or any person level variables, because those variables will be perfectly colinear with the individual level intercepts. Also note that estimating all of these intercepts directly takes a lot of degrees of freedom, and thus increases standard errors.

But there is an alternative. Instead of estimating the person-level intercepts individually, one by one, we could make the assumption that, in the population at large, these intercepts actually vary "randomly" (that is, according to a normal distribution) around some overall value. (this is the same assumption we make about the *error term in an OLS model - that individual errors vary normally around the regression line). We still estimate the overall value, but we don't actually try to compute a separate intercept for each person. Instead we use the data we have to compute the variance of the normal curve that we assume that these intercepts follow. These "random intercepts" (one per person) are what you could call the "random effects" in this model, but we don't estimate them directly, so we don't get coefficients (or p values) for them. As before we also get a coefficient (and p value) for any independent variables in the model. In this "random intercept" model, the random intercepts do account for the fact that some individuals have different starting points than others just due to random chance, but they do not account for systematic differences between individuals due to confounding variables (like SES or gender). But unlike in the fixed effects model we can account for those differences by including those variables as controls. So the coefficient we get for a particular independent variable (like weeks) may be biased due to omitted person-level confounders, although the estimate for the standard error around that coefficient does account for the nesting of observations within individuals, and does so more efficiently than the fixed effects model.

To decide which of these three approaches is best for a given model is a complex task, too long to discuss here (there are tradeoffs between bias and efficiency). Google "hausman test" and "intraclass correlation coefficient" for more.

Now we can also apply this same question to the coefficient of an independent variable in the model. Let's assume we've already decided to treat the intercept as a random effect. We could still just estimate a single slope coefficient for the entire sample like we normally do. Or we could allow the slope itself to also vary normally around an overall value. The assumption here is that there is an overall "average" effect for "weeks" but we assume that each individual's own slope varies according to a normal distribution around that average. Again, we don't estimate these person-specific slopes directly (so you won't get coefficients or p values for them), we just use the data we have to estimate the variance of that normal curve. Is this a good idea? This is actually somewhat tricky to figure out. The approach I use is to just run two versions of the same model, one with a random slope and one without, and calculate a likelihood ratio test to see if one has significantly better fit than the other.