I am trying to understand/visualize it in my head how fixed/random effects models work. Can someone explain how can I infer something about the population from which I drew the sample with random effects model? Perhaps on a simple example.

  • 1
    $\begingroup$ Is this one helpful: stats.stackexchange.com/questions/4700/… ? $\endgroup$ – Tim May 11 '15 at 13:48
  • $\begingroup$ I have read it all but still don't understand it completely. $\endgroup$ – Quirik May 11 '15 at 13:55
  • $\begingroup$ OK, so maybe describe in greater detail what you do understand, and what you don't, so you can get an answer that is aimed at the things that are not clear for you. $\endgroup$ – Tim May 11 '15 at 13:56
  • $\begingroup$ Suppose a model with a grade as dependent and study time as independent variable. With fixed model I can compare average grade differences between students in the sample by introducing dummy variable. If we assume that random-effects assumptions hold, what can I infer with running random-effects model? Does running random effects estimates intercept for every student? $\endgroup$ – Quirik May 11 '15 at 14:13

Suppose that we own a factory that makes cardboard boxes and we want to assess our box making machines. We have access to an unlimited amount of perfectly homogeneous input cardboard and the machines are totally automatic so that there is no operator effect.

Experiment 1: We have 10 box-making machines and make a box with each one.

Experiment 2: We have 100 machines and take a sample of 10 machines. Each of these machines produces 1 box.

For experiment 1, we are looking at every machine that we have. We care about these particular machines and want to see how they deviate from each other. This reasonably could be modeled by a fixed effects model $$ Y_{ij} = \mu + \alpha_i + \varepsilon_{ij} $$ where $\mu$ is the overall mean, $\alpha_i$ is the fixed deviation from $\mu$ for machine $i$, and $\varepsilon_{ij} \sim \mathcal N(0, \sigma^2)$ is the random error. We then could run a typical one-way ANOVA to see if the $\alpha_i$ term matters, after which perhaps we could use Tukey's HSD to see which specific machines differ.

Now for experiment 2, we don't care so much about these particular machines but rather we care about the population of machines. What we really want to investigate is the variation of the machines, not the specific mean values of the 10 machines that we happened to get. This means we're looking at a population-level parameter (population variability) rather than a machine-level parameter (specific machine mean deviation).

For this our model is again $$ Y_{ij} = \mu + \alpha_i + \varepsilon_{ij} $$ and everything is the same except we now assume $\alpha_i \sim \mathcal N(0, \sigma^2_\alpha)$, i.e. $\alpha_i$ is a random effect.

Our goal now is not to get good estimates $\hat \alpha_i$ but rather to look at $\sigma^2_\alpha$ and in particular to test if it equals 0. This is because, as I've said, we don't care about these specific machines (i.e. the $\hat \alpha_i$) but rather care about the variation of the entire machine population (i.e. $\sigma^2_\alpha$).

In general, a good way to decide if something should be random or fixed is to think about repeatability. If you could redo the experiment with the exact same things, i.e. with the exact same drug dosage, then that can be a fixed effect. But if you've got something that you can't repeat, like which mice you gave the drug to (you can give the drug to 10 mice of the same species, but those exact 10 mice are probably long gone) then the effect of that probably ought to be random.

  • $\begingroup$ Now it makes a little clearer to me but still have few questions. Why are we interested to investigate variation at the population-level? Do you have maybe a link with simple example how to estimate random-effects model? $\endgroup$ – Quirik May 11 '15 at 16:11
  • 1
    $\begingroup$ This link describes the two main ways of estimating the parameters (ML and REML). $\endgroup$ – jld May 11 '15 at 17:03
  • 1
    $\begingroup$ As for why we want to investigate the population variance, it's because our random effects are draws from a random population. They are random variables, not fixed parameters, so as with any other inferential problem we try to learn about the generating process, not the random variables themselves. $\endgroup$ – jld May 11 '15 at 17:05
  • $\begingroup$ What do you mean by "generating process"? $\endgroup$ – Quirik May 14 '15 at 9:41
  • $\begingroup$ A typical statistics problem is "if $X_1, \dots, X_n \sim \mathcal N (\mu, \sigma^2)$ then estimate $\mu$ and $\sigma^2$." In this case the "generating process" is $\mathcal N (\mu, \sigma^2)$ and estimating the parameters is how we learn about this process. It's the process (distribution) that gives rise to the data. $\endgroup$ – jld May 14 '15 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.