Factorial experiment design with incompatible levels in IVs

Question

Is it safe to run a Factorial ANOVA if you cannot collect data for some cells?

Scenario: There are two independent variables, and one dependent variable.

All independent variables are nominal:

• D (data structure) has 4 levels: $d_1$ to $d_4$.

• F (distance function) has 6 levels: $f_1$ to $f_6$.

The dependent variable is interval: Y (outcome) has domain $[0, 1] \subset \mathbb{R}$.

Data looks like this:

(subject, D, F, Y) Example: (user001, $d_1$, $f_1$, 0.3).

Goal: To determine the effect of D and F on Y.

Problem: there is incompatibility between some levels of D and F.

Example: $d_1$ is incompatible with $f_6$. This means that Y cannot be measure in this cell.

There are other incompatibilities:

• $d_2$ is incompatible with $f_6$.

• $d_3$ is incompatible with $f_3$ to $f_6$.

• $d_4$ is incompatible with $f_1$ to $f_5$. This means that two objects represented as data structures of type $d_4$ can only be compared by using the function $f_6$.

Answer 1

There is no problem doing this ... although it will depend on your goals. Eliminating from the design the impossible combinations, you will get a non-regular fraction, but that in itself is not a problem. Main effects can still be estimated. If you are interested in interactions, interaction parameters corresponding to the impossible combinations cannot be estimated, but as they are not meaningful that is not a problem.

You can investigate doing some simulations in R:

D <- factor(paste0("D", 1:4, sep=""))
F <- factor(paste0("F", 1:6, sep=""))
dat0 <- expand.grid(D=D, F=F)
incompat <- c(21, 22, 4, 8, 12, 16, 20) 
dat <- dat0[-incompat, ]  

N <- NROW(dat)
# Simulate some data:
Y <- rnorm(N, 10, 2)  
mod <- lm(Y ~ D + F, data=dat) # Main effects model 

summary(mod)   

Call:
lm(formula = Y ~ D + F, data = dat)

Residuals:
    Min      1Q  Median      3Q     Max 
-2.4055 -1.0877  0.2836  0.9961  1.8258 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  9.71458    1.29380   7.509 6.87e-05 ***
DD2          0.08264    1.19783   0.069   0.9467    
DD3          1.37563    1.19783   1.148   0.2840    
DD4          5.46891    2.93407   1.864   0.0993 .  
FF2          0.90613    1.54639   0.586   0.5741    
FF3          0.29150    1.54639   0.189   0.8552    
FF4          0.42037    1.54639   0.272   0.7926    
FF5         -2.00301    1.54639  -1.295   0.2313    
FF6         -2.22783    2.29367  -0.971   0.3598    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.894 on 8 degrees of freedom
Multiple R-squared:  0.5162,    Adjusted R-squared:  0.03233 
F-statistic: 1.067 on 8 and 8 DF,  p-value: 0.4647

If you are interested in interaction, try

mod2 <- lm(Y ~ D*F, data=dat)  

and you will se what said above. In addition, there will be no degrees of freedom for error, so you will need to replicate the design in this case.