Data analysis of unbalanced study - Interaction techniques in virtual reality

Question

I'm planning a study on interaction techniques in virtual reality. That means I want to compare the performance of the participants on different interaction forms (e.g. selecting objects with a ray or by grabbing) in different scenarios (e.g. different distances and object sizes). My study design is rather complex and I'm not that familiar with data analysis so I hope you can help me with the following two questions:

1. I have multiple Independent variables. For simplification let's have a look on three of them. A is the interaction technique, B is the distance and C is the object size. Not all Interaction technique work for all distances. That means for some interaction techniques there will be missing data: \begin{array} {|r|r|}\hline & B_1 & B_2 & C_1 & C_2 \\ \hline A_1 & x & & x & x \\ \hline A_2 & x & x & x & x \\ \hline A_3 & x & x & x & x \\ \hline A_4 & x & & x & x \\ \hline ... & & & & \\ \hline \end{array} $X_1$ and $X_2$ indicate the different conditions of a variable $X$. After the study I want to analyse the data for $B_1$ and $B_2$ separately. If a technique does not support $B_2$ it will simple not considered in the analysis. Is this possible? I'm afraid that the two conditions are somehow connected. I think the cleanest way would be to do two studies investigate $B_1$ and $B_2$ separately but that would be more time consuming ...

2. I want to test 12 different interaction techniques. That's a lot and it is impossible to let all participants test all techniques. Therefore each participant will test 3 random interaction techniques. That means I have some kind of a mix of within-subject and between-subject design. Unfortunately there are no fixed groups because of the randomly assigned techniques. Therefore I cannot use a split plot ANOVA for example. Are there any other models I can use? Or is it possible to assume that each technique was used by a different person even if one person used multiple techniques? Then if would be possible to use a split plot ANOVA.

Answer 1

You have factors (predictors) A (technique), B (distance), C (object size) and a random effect (Participant), and a response variable that I now take as measured (maybe a time?, you didn't specify.) Without knowing much context I can propose a simple linear mixed model $$ Y_{ijkl}=\mu + \alpha_j + \beta_k + \gamma_l + \pi_i + \epsilon_{ijkl} $$ where $i$ indexes participant, $j$ technique, $k$ distance and $l$ object size, and not all combinations of indexes occur (maybe also some interaction terms are necessary.) $\pi_i \sim \mathcal{N}(0,\sigma^2_\pi)$ is a participant random effect, $\epsilon_{ij} \sim \mathcal{N}(0, \sigma^2)$ is experimental error.

Modern linear mixed model software like lme4 can handle this model without all index combinations being present (those non-present simply being left out from the data file.) What you should not do is hiding the non-independence created by each participant contributing three observations! The one participant random effect here represents that dependence.

The model written above should only be taken as a point of departure, there might be aspect you didn't tell us about ...

About the experimental design: The layout should not be chosen haphazardly, but systematically, maybe to optimize some criterion like D-optimality. But I am not sure about software for D-optimality (or similar) for linear mixed models.