# Is it possible to determine how many effects I can estimate in a least squares problem just by looking at the correlation matrix?

I currently have a model matrix $X$ with $6$ columns, which is being used for a factorial design problem, with each column associated with an effect. The ultimate goal is to be able to estimate as many of the effects, $A,B,C,D,E,F$, as we can. This is to be done by using the least squares equation: $\hat{\beta} = (X^{T}X)^{-1}X^{T}Y$, where $Y$ is a vector of responses which I didn't include as it is not need here.

$$X = \begin{bmatrix} A & B & C & D & E & F \\ \hline -1 & 0 & -1 & -1 & 0 & -1 \\ -1 & 0 & -1 & 1 & 1 & 0 \\ -1 & 0 & 1 & -1 & 0 & 1 \\ -1 & 0 & 1 & 1 & -1 & 0 \\ 1 & -1 & 0 & -1 & 0 & 1 \\ 1 & -1 & 0 & 1 & -1 & 0 \\ 1 & 1 & 0 & -1 & 0 & -1 \\ 1 & 1 & 0 & 1 & 1 & 0 \\ \end{bmatrix}$$

My correlation structure, $Corr(X)$, where the correlation is computed as the correlation between the 6 columns of $X$, looks like:

$$Corr(X) = \begin{bmatrix} 1.0 & 0.0 & 0.0 & 0.0 & 0.0 & 0.0 \\ 0.0 & 1.0 & 0.0 & 0.0 & 0.5 & -0.5 \\ 0.0 & 0.0 & 1.0 & 0.0 & -0.5 & 0.5 \\ 0.0 & 0.0 & 0.0 & 1.0 & 0.0 & 0.0 \\ 0.0 & 0.5 & -0.5 & 0.0 & 1.0 & 0.0 \\ 0.0 & -0.5 & 0.5 & 0.0 & 0.0 & 1.0 \\ \end{bmatrix}$$

My question is, how many of the $6$ original effects can be estimated and how do I go about determining this? I saw in a paper that looking at the correlation matrix, we can see that five of the six effects can be estimated, and this is because $dim(space) = 5$, or the dimension of the correlation matrix spanned by six columns is $5$.

I am not exactly sure why this is true. Could anyone shed some insight how one can determined which effects may be estimated just by looking at the correlation matrix? I know in general that for OLS to work, the columns of the regressors $X$ must be linearly independent. However, what does looking at the linear independence of a correlation matrix have anything to do with this? Thanks!!

The rank of the $X$ matrix is 5. Do not focus on the correlation matrix; it is a red herring at this point. As you correctly assess having a rank of 5 mean that despite $X$ having 6 columns it spans a 5-dimensional space. That is because one of the $X$ matrix's columns in a linear combination of your previous five. For the matrix $X$ presented this is the 6th column. The sixth column can be written as the sum of the third and fifth column minus the second column. (So $F = C + E - B$.) This clearly violates the linear independence condition between the columns of $X$.
Given that the rank is 5, instead of 6, we know we look for a single "problematic" column. As such if I took a column out of $X$ (call that thinner $X$, $X^5$) and got the rank of it to be 5 the column removed could be expressed as the linear combination of other columns. I tried that and I immediately got that column $F$ is not independent of the others. Let me stress that there definitely not a single column you can remove. That is because if $F = C + E - B \Leftrightarrow B = C + E - F \Leftrightarrow C = \dots$, etc. etc. So OK, I found that $F$ is a problematic column, how I get what linear combination of the other columns reconstructs it? To do this, I solve the corresponding linear system $X^5$ over $F$. I used MATLAB so I literally typed X(:,1:5)\ X(:,6) but in R I would have typed qr.solve(X[:,1:5], X[:,6]) to get the solution as 0.0000, -1.0000, 1.0000, 0.0000, 1.0000. So you have it $F = 0A -1B + 1C + 0D + 1E \Rightarrow F = C + E - B$. Easy right?
• Side comment: The correlation matrix reported appears wrong. For example the correlations between columns $B$ and $C$ has to be zero. There is not correlation whatsoever while in the matrix provided they are anti-correlated ($\rho = -0.5$). – usεr11852 Feb 1 '16 at 19:49
• I am glad I could help. You do not mention any papers in your question so I cannot comment on that. What I said for $X$ applies for the correlation matrix too. Eg. take the 4th row: $0 + 0 - 0 = 0$ or the 5th row: $-0.5 + 1 -0.5 = 0$, etc. etc. I will describe the heuristic I used in the question. – usεr11852 Feb 1 '16 at 21:30
• Thanks so much, when you write "eg. take the 4th row: $0+0-0=0$ or 5th row" I am not quite sure how you get the two equations and how it relates to why I can do the same thing with a correlation matrix. This might be a dumb question but do you mind if I ask what these equations are? – user1398057 Feb 1 '16 at 22:01
• I refer at the $i$-th row of the matrix you denote $Corr(X)$. So let's say you look at the second row: $C_{col} = 0$, $E_{col} = 0.5$ and $B_{col} = 1$ so you have $0 + 0.5 -1 = -0.5$ which actually your value in $F_{col}$. In the case of $i =1 \& 4$ everything equates to zero so the equality is trivial. For for $i=5$ $B_{col} = 0.5$, $C_{col} = -0.5$ and $E_{col} = 1$. So you have $-0.5 +1 -0.5 = 0$ which again equates to the value you have in column $F_{col}$ at row $i=5$. In general as mentioned the correlation matrix is a bit unnecessary here all the information need can be found in $X$. – usεr11852 Feb 1 '16 at 22:10