My intuitive understanding is that if $x$ and $z$ are categorical factors, then each observation $y_i$ is given a mean value which is equal to the mean value given to $y_j$ if $y_i$ and $y_j$ belong to the same $(x,z)$ product group.

However, when I run the command in R and look at the model matrix, the parametrisation is clearly not the same. R seems to give each observation a mean value equal to a sum of a bunch of parameters. Why? What does it mean?

My only clue right now is that R maybe considers $x$ and $z$ to be numerical predictors, and thus the parametrisation is a regression?


closed as off-topic by Andy, Tim, Christoph Hanck, Greenparker, mpiktas May 10 '16 at 11:29

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Andy, Tim, Christoph Hanck, Greenparker, mpiktas
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ In an R formula x * y = x + y + x:y. Where x:y is x times y, as in normal algebra i.e. an interaction between between x and y. Also the question is off topic, and I have flagged it as such $\endgroup$ – Repmat May 10 '16 at 9:06

I'm not sure I understand the question completely, so I'll try to debug and answer any potential problems in order.

Are you actually working with factors?

First things first: Have you made sure that X and Z actually enter the model as factors? Use a built-in R command to make sure that the variables are not interpreted by R as integers or other numeric format.

Understanding how R interprets the input

Now assuming that X and Z are factors: Let's say that X is a factor related to gender, and Z is a factor related to vocation. Let's pretend that $X = \{male, female\}$ and $Z = \{Carpenter, Personal Assistant, Data Scientist\} $. Let's also say that Y is the annual income per person in the data set. Then you're trying to predict the annual income based on gender and vocation.

When specifying a model in R, the syntax X*Z means: $X + Z + X:Z$. So you're looking at the effect of gender ($X$), the effect of vocation($Z$), and the effect of specific gender within a specific vocation($X:Z$).

Let's pretend that men and woman earn the same (to within your chosen level of significance) except that male carpenters earn more than female carpenters. The p-values for all parameters would then be above your chosen threshold (meaning there is no added effect on income from the factor), except for the one interaction male:carpenter, which would be significant, with a positive parameter value.

That was (roughly) the $X*Z$ model. What if you did the $X+Z$ model? Then you get parameters for genders and you get parameters for vocation. But no interaction.

Finally you could model $X:Z$ and get only the interactions.

Why are some parameters missing from the output

You need to know why some parameters will appear to be 'missing' from the output of the call to lm. This is due to the way factors enter the design matrix; if you have an intercept (first column is all ones), then you can't also have a column of 1's where the test subject is male, while you also have a column of 1's where the test subject is Female, since the design matrix would become ill-posed (due to rank-deficiency). Therefore the design matrix can only have a column of all 1's (the intercept) and a column where the test subject if Female. These two columns in the design matrix then gives you two parameters: the intercept and the added effect of being female. The effect of Male is included in the intercept.

You should look into Helmert, Sum-coding, and Treatment-coding to see how factors can be included in the design matrix. Doing a few hand calculations on small systems gives a nice idea of how to implement and interpret factors in a linear model.

So if you we're looking at $y\text{~} x$ then you'd get an output with an intercept and a single parameter value that would say something like: genderFemale, but you won't see genderMale in the output. The reason for this is that you can't find an intercept, an added effect from Male, and an added effect from Female at the same time. But you can find an intercept and an added effect from Female.

Sums of a bunch of Parameters

Factors enter models as sums of 'added effects' because you can only ask whether a test subject is Male/Female or Carpenter/Personal Assistant/Data Scientist. You're asking an either/or question of the test subject, not 'how much'. Either/Or questions can only predict a very specific amount of variation in your output. Think of this: if you asked someone if their primary vocation is carpenter and the person said: Yeah, at about 0.3. Then you'd definitely consider the answer nonsense. You can't 'scale' factors. A factor is a yes/no thing. If you asked the same test subject how many hours he/she worked as a carpenter per week and you got an answer in hours, then you'd get a slope on the annual income because a number makes sense in this regard.

  • $\begingroup$ The part I don't understand is the $X \cdot Z$-model and why it is the way it is. What I've been told (and, again, my problem is that R doesn't agree with what I've been told) is that the $X \cdot Z$-model is the one where $\xi_i = \xi_j$ (for two observations $i$ and $j$) if $i, j \in (x,k)$ where $(x,k)$ is one group out of $|X| \cdot |Z|$ groups. So, we'd have one mean for all observations belonging to (Male, Carpenter), one mean belonging to (Male, Personal Assistant), and so on .... Is this completely wrong? Or is this just a different parametrisation of the same model? $\endgroup$ – Vici May 10 '16 at 10:30
  • $\begingroup$ What I expected to see would be a model matrix which in each row only has 1 non-zero number (and that number is 1), in the position corresponding to the group to which that row-observation belongs to. So, if it's the first row, then that may correspond to a Male Carpenter, and if that's our "first" parameter, we'd have a $[1,0,0,....,0]$ in the first row. And so on. $\endgroup$ – Vici May 10 '16 at 10:34
  • $\begingroup$ It sounds to me like you ONLY want the interactions. Have you tried: Y~-1+X:Z and looked at the design from that? $\endgroup$ – Beyer May 10 '16 at 10:37
  • $\begingroup$ Yeah, that seems to be it. Question: Is it the same model? I tried to compare the fitted values of both these two models, and they seem similar. If so, I guess what I was told was correct, except it didn't mention that R parametrises it differently, i..e the parameters have different interpretations? $\endgroup$ – Vici May 10 '16 at 11:01
  • 1
    $\begingroup$ "The Same Model"? I have to ask: The same model as what? The y~-1+X:Z is not the same as y~X*Y. The difference in parameter estimates might be small from one model to the other if the effect size is not very large, but the models are not the same. The interaction-only model cannot say anything about the general paycheck-advantage of being a man, compared to being a woman, say. It can only say something about men/vs woman within a specific vocation. $\endgroup$ – Beyer May 10 '16 at 11:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.