0
$\begingroup$

In econometrics, we often have n observations (in a column vector $y$) which we want to explain with k$<$n regressors (the observations are in an nxk matrix $X$). In this case we use least squares estimation and we can uniquely write the observation as a sum of a fitted value $\hat{y}\in C(x)$ and a residual $\hat{\epsilon} \notin C(X)$, where $C(X)$ is the column span of X: $y = \hat{y} +\hat{\epsilon}$

In this case we have $$X^T\hat{\epsilon} = 0$$ which expresses "othogonality". My question is, Where does the transpose on $X$ come from? The above form is very reminiscent of that of the orthogonality of two column vectors defined by the inner product: $<x,y> = x^Ty = 0$, but I have never seen "the inner product" of matrices defined and when we multiply matrices we never need to take the transpose.

$\endgroup$
0
$\begingroup$

The equation $X^\top \epsilon = 0$ is simply matrix shorthand for the system of equations $$ \begin{align} x_1^\top \epsilon & = 0 \\ & \vdots \\ x_n^\top \epsilon & = 0 \\ \end{align} $$ where $x_i$ is a vector in $\mathbb{R}^n$ representing the $i$th column of the design matrix $X$. This system encodes the constraint that each feature vector be orthogonal to the residual vector.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.