Multifactor ANOVA- What is the connection between sample size and orthogonality?

Question

It is commonly stated that "balanced" designs in multifactor ANOVA (equal sample sizes in each group) have "orthogonal" factors. In other regression contexts, I totally get the connection between the geometry (things at right angles to each other) and the statistics. Like how a residual vector is orthogonal to a least-squares solution because the latter is the projection of the response variable vector onto the column space of the design matrix. That makes perfect sense to me.

But for ANOVA, I'm at a loss. I can't figure out how factors in ANOVA correspond to some kind of geometric objects that are orthogonal. Even more perplexing is how this can only depend on the sample size in each group.

What am I missing? Can someone explain what geometric objects are supposed to correspond to "orthogonal" factors and why their orthogonality is a function only of the sample size in each group?

Answer 1

Let's take a simple 2x2 ANOVA, with two factors, x1 and x2, each of which can take one of two values, 0 and 1. There are therefore four cells in the design: [x1=0,x2=0], [0,1], [1,0], and [1,1].

In a balanced design, there are equal numbers of cases in each cell.

As a result, the two factors, x1 and x2 are uncorrelated (orthogonal), $r = 0$.

In an unbalanced design, there are more cases in some cells than others.

As a result, the factors are not uncorrelated, and so not orthogonal: in this example, cases where x1=1 are more likely to have x2=1, $r \approx .25$.

Demo code

library(tidyverse)
plot_design = function(df){
  long = df %>% 
    mutate(.index=1:n()) %>%
    pivot_longer(-.index)
  ggplot(long, aes(name, -.index, fill = factor(value))) +
    geom_tile(color = 'white') +
    scale_fill_manual(values = c('grey', 'black')) +
    labs(x = 'Column', y = 'Row', fill = 'Value') +
    scale_y_continuous(labels = function(x) -x) +
    theme_minimal() + coord_fixed(ratio = .1)
}

balanced_ns = c(20, 20, 20, 20)
df_balanced = data.frame(
  x1 = rep(c(0, 0, 1,1), times = balanced_ns),
  x2 = rep(c(0, 1, 0, 1), times = balanced_ns)
)
plot_design(df_balanced) + labs(title = 'Balanced design') # First figure, above
cor(df_balanced)
##    x1 x2
## x1  1  0
## x2  0  1


unbalanced_ns = c(20, 20, 10, 30)
df_unbalanced = data.frame(
  x1 = rep(c(0, 0, 1, 1), times = unbalanced_ns),
  x2 = rep(c(0, 1, 0, 1), times = unbalanced_ns)
)
plot_design(df_unbalanced) + labs(title = 'Unbalanced design') # Second figure
cor(df_unbalanced)
##           x1        x2
## x1 1.0000000 0.2581989
## x2 0.2581989 1.0000000
```

Answer 2

I'll take a stab at this one, but first from a non-geometric perspective. In probability, we estimate the joint probability of two random events using the following equation:

$p(A\cap B) = p(A)\times p(B)$

We can use this equation to also calculate the deviations from independence:

$\text{observed} - \text{expected under independence} = P(A\cap B) - P(A)\times P(B)$

where $P$ represents the observed probability. (Sorry if my notation is terrible...it's not my forte).

This is exactly what the traditional chi-square test of independence uses, but it replaces probabilities with counts:

expected = row sum $\times$ col sums / table sum

So, with an ANOVA, if your sample sizes are equal in each cell, they will be independent (orthogonal) because row sums $\times$ colsums / table sums will always exactly equal the expected probability as with a chi-square test.

Or, a different way to think about it is this: assuming a random process, it would be highly unusual to have a sample where the number of males/females over 6 feet tall are equal. Why would that be unusual? Because height is NOT independent of gender.

Likewise, for an ANOVA, if we have uncorrelated variables, they should have equal sample sizes. If they're correlated, they should not.

Make sense?

Geometric Explanation

This may be better to "see" than to explain. Let's say we have two factors (a and b) where the sample sizes are the same. The R code below generates the data and plots it. If we plot a against b (or vice versa), the best fitted regression line is exactly flat:

a = c(rep(c(1,2), times=6))
b = c(rep(1, times=6), rep(2, times=6))
d = data.frame(a=a,b=b)
table(a,b) #verify orthogonality
require(ggplot2)
ggplot(data=d, aes(x=a, y=b)) +
  geom_jitter(width=.1, height = .1) +
  geom_smooth(method="lm") # flat line

If we now allow different sample sizes in each group, there's now a positive slope:

a = c(rep(c(1,2), times=6))
b = c(rep(1, times=3), rep(2, times=9))
d = data.frame(a=a,b=b)
table(a,b) #verify nonorthogonality
ggplot(data=d, aes(x=a, y=b)) +
  geom_jitter(width=.1, height = .1) +
  geom_smooth(method="lm") # nonflat line