Why is it necessary to "ignore" a level when applying sum contrasts?

Question

I am confused about how sum contrasts are set up. As I understand, if I have some $K$-leveled factor, I can use sum contrasts to compare each level to the grand mean ($M_G$), effectively testing multiple hypotheses each predicting that some group $i$ of $K$ is different from $M_G$.

In practice (in R), when I create a contrast matrix for a 4-leveled factor by calling contr.sum(4), I get:

contr.sum(4)

 1   0   0
 0   1   0
 0   0   1
-1  -1  -1

then when I invert it using ginv from MASS (adding an intercept column) to get a hypothesis table of mean weights, I get this:

ginv(
  cbind(1, contr.sum(4))
)

 0.25    0.25    0.25    0.25
 0.75   -0.25   -0.25   -0.25
-0.25    0.75   -0.25   -0.25
-0.25   -0.25    0.75   -0.25

which gives me 3 ($K - 1$) hypotheses, excluding the null. So, the first row (hypothesis) says that the weighted average of all 4 groups ($M_G$) is 0. Then, the second row says that $\frac{3}{4}M_1 - \frac{1}{4}M_2 - \frac{1}{4}M_3 - \frac{1}{4}M_4 = 0$, or equivalently that $M_1 = \frac{M_1+M_2+M_3+M_4}{4} = M_G$ and so on ($M_i$ is the mean of group $i$).

However, there is no hypothesis comparing the mean of the 4th group, or in general the $K$th group. Why is that?. What if I wanted to specifically compare each group mean to the grand mean in my analysis? I feel like I am missing something obvious here, as most resources I see on that simply mention that there are $K-1$ sum contrasts for a factor with $K$ levels.

Answer 1

Similar questions have been asked a lot here ... but mostly in the context of dummy encoding, see Dropping one of the columns when using one-hot encoding and also Removing intercept from GLM for multiple factorial predictors only works for first factor in model The underlying issue is the same when using contr.sum as it is in those posts where contr.treatment where used. Using your example, first, you can get a full contrast matrix with

test_full <- cbind(1, contr.sum(4, contrasts=FALSE))
 test_full
    1 2 3 4
1 1 1 0 0 0
2 1 0 1 0 0
3 1 0 0 1 0
4 1 0 0 0 1
 Matrix::rankMatrix(test_full)
[1] 4
attr(,"method")
[1] "tolNorm2"
attr(,"useGrad")
[1] FALSE
attr(,"tol")
[1] 1.110223e-15 

(and, if you want you can trick R into using such a contrast matrix with lm, but then you get non-identifiable parameters). Continuing with your example,

test <- cbind(1, contr.sum(4))
myC <- MASS::ginv(test)
myC
      [,1]  [,2]  [,3]  [,4]
[1,]  0.25  0.25  0.25  0.25
[2,]  0.75 -0.25 -0.25 -0.25
[3,] -0.25  0.75 -0.25 -0.25
[4,] -0.25 -0.25  0.75 -0.25 

solve(t(myC), c(-1, -1, -1, 3)/4)
[1]  0 -1 -1 -1

that way, you get your missing contrast for the fourth group as a linear combination of the four rows of myC:

-(myC[2, ]+myC[3, ]+myC[4, ])
[1] -0.25 -0.25 -0.25  0.75