Two-way ANOVA with interaction term / orthogonal design

Question

The two-way ANOVA model with interaction for some continuous variable $y$ can be expressed as $$y = X\mu + \varepsilon,$$ where $X$ is the design matrix (the first column of $X$ contains the constant, the second and third columns the two main effects, and the fourth column the interaction of the two main effects), $\mu$ is the vector of effects, and $\varepsilon$ is a normally distributed error term. The estimator $$\hat\mu = (X'X)^{-1}(X'y)$$ is the ordinary least squares estimator for $\mu$. In the case of a balanced design with block length $n$, the factors can be rendered uncorrelated with unit variance by post-multiplying the design matrix with $\operatorname{chol}((X'X)^{-1})$. Here $\operatorname{chol}$ denotes the Cholesky operator that assigns a positive definite square matrix to the upper triangular matrix of its Cholesky decomposition. It turns out that this is the same as dividing the first column of $X$ by $\sqrt{2n}$, transforming the second and third column (i.e., the main effects) by the linear-affine transformation $T(u) = \frac{2u - 1}{\sqrt{2n}}$, and the fourth columns, i.e., the interaction effect can be obtained by multiplying the second and third columns.

Edit: Transforming $X$ by post-multipliying it by $\operatorname{chol}((X'X)^{-1})$, i.e., $X\operatorname{chol}((X'X)^{-1})$, results in the following matrix $$Z = \begin{pmatrix} \frac{2x_{11} - 1}{\sqrt{2n}} & \frac{2x_{12} - 1}{\sqrt{2n}} & \frac{2x_{13} - 1}{\sqrt{2n}} & \frac{2x_{12} - 1}{\sqrt{2n}} \cdot \frac{2x_{13} - 1}{\sqrt{2n}} \\ \frac{2x_{21} - 1}{\sqrt{2n}} & \frac{2x_{22} - 1}{\sqrt{2n}} & \frac{2x_{23} - 1}{\sqrt{2n}} & \frac{2x_{22} - 1}{\sqrt{2n}} \cdot \frac{2x_{23} - 1}{\sqrt{2n}} \\ \vdots & \vdots & \vdots &\vdots \\ \frac{2x_{2n1} - 1}{\sqrt{2n}} & \frac{2x_{2n2} - 1}{\sqrt{2n}} & \frac{2x_{2n3} - 1}{\sqrt{2n}} & \frac{2x_{2n2} - 1}{\sqrt{2n}} \cdot \frac{2x_{2n3} - 1}{\sqrt{2n}},\end{pmatrix}.$$ where $x_{ij}$ denotes the row $i$ column $j$ element of $X$.

I am now interested in interpreting the transformed columns, i.e., I want to estimate the model $$y = Z\theta + u,$$ where $Z$ is the matrix we obtain if we transform $X$ according to the aforementioned recipe. Since $Z'Z$ equals the identity, the ordinary least squares estimator for $\theta$ is given by $$\hat\theta = Z'y.$$ I often read that in an orthogonal design, "each factor can be evaluated independently of all the other factors". I can pre-multiply model equation by some elimination matrix $E$ that removes, say, the interaction effect, and the estimates of the other effects will not change (this mainly follows from the fact that matrix multiplication is associative). My question is whether this is also true for interpreting the effect. That is, does the interpretation of, say, the second column of the first (transformed) main effect independently from the interaction effect? In other words, if we looked at the ceteris paribus effect of $\theta_2$ is it the same with or without the interaction term? Looking at the estimated model equation, I can hardly believe that this is true; so the quote refers to the numerical estimate of $\hat\theta$ only.

Answer 1

tl;dr answer: it depends on the data itself.

longer answer:

Eventually, no matter how many linear manipulations you make, your data will always contain the columns $x_2,x_3$ which may or may not have some collinearity. The answer to the question "are the correlated?" which is very much similar to the question "is the coefficient of the interaction term a zero?", depends on the data and not solely on the design.

True, an orthogonal design promises individual evaluation, but this is when the data columns are independent by nature - for example, consider data of math/sports/history grades at 3 different schools. we take $n$ samples for each subgroup (overall $3n$ per school and $9n$ in the whole dataset) but by nature these are independent factors.

One manipulation you might want to consider in order to obtain orthogonal columns is PCA, but then it is up to you to assign the proper explanation per column (even before estimating the model coefficients).