2
$\begingroup$

In linear regression suppose we parition the regressors X (with k variables and n observations) into two sets X1 (with k1 variables) and X2 (with k2 variables) where k1 and k2 sum to k. I found some resource online that said:

"Columns of X1 are orthogonal to the columns of X2 such that X1'X2 = 0. This is the same as assuming that the empirical correlation between variables in X1 and variables in X2 is zero".

I don't quite understand the statement. First are they taking the dimension of X1 to be 1xk1 or nxk1? I am not able to understand what the columns of X1 and X2 are and how they have definied orthoginality. Also what exactly is empirical correlation and how does orthogonality imply no correlation?

$\endgroup$
5
$\begingroup$

If we view each regressor as a random variable that we observe, then the $n$ observations are simply realizations of these random variables. So, whenever we refer to something "empirical", we're really speaking with regards to a sample. Our sample is our $n$ observations. This is in contrast to talking about the random variables themselves.

To address your question, say we have regressors $X_i$ and $X_j$. The correlation between random variable $X_i$ and $X_j$ is

$$\frac{Cov(X_i, X_j)}{\sqrt{Var(X_i)Var(X_j)}}$$

as you probably know. If we replace the covariance and variances with sample covariance and sample variances we get the empirical correlation of $X_i$ and $X_j$: empirical = strictly based off the data.

The dimensions of $X_1$ are $n \times k_1$. We can tell because the excerpt says that the columns of $X_1$ are orthogonal to the columns of $X_2$. Clearly, this means $X_2$ is $n \times k_2$.

What the author means by orthogonal is just that, for any column $i$, $X_i^{(1)}$ of $X_1$ and column $j$, $X_j^{(2)}$ of $X_2$, we have

$$\sum_{l=1}^n X_{l,i}^{(1)} X_{l,j}^{(2)} = 0.$$

In other words if we take the elementwise product of two columns, where one column is in the $X_1$ partition, and the other column is in the $X_2$ partition, then that elementwise product is guaranteed to be zero.

There is a caveat to what the author says, however. Notice that the empirical correlation is zero if and only if the empirical covariance is zero. It is the definition of empirical covariance, which we'll represent as $c_{X_i, X_j}$ (standing for the empirical covariance between the regressors $X_i$ and $X_j$) that

$$c_{X_i, X_j} = \frac{1}{n}\sum_{l=1}^n (X_{l,i} - \bar{X}_{i})(X_{l,j} - \bar{X}_j)$$

where $\bar{X}_i$ and $\bar{X}_j$ are the sample means of the $i$th and $j$th column of our data matrix, respectively. Indeed, it does not matter if the columns of our data matrix are orthogonal at all if they are not centered!

Thus, the correct statement to make is that orthogonality of centered (meaning that every column must have mean zero) columns between $X_1$ and $X_2$ implies that the empirical correlation of any column from $X_1$ and any column from $X_2$ is zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.