# Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression

The closed form of w in Linear regression can be written as

$\hat{w}=(X^TX)^{-1}X^Ty$

How can we intuitively explain the role of $(X^TX)^{-1}$ in this equation?

• Could you elaborate on what you mean by "intuitively"? For instance, there is a wonderfully intuitive explanation in terms of inner-product spaces presented in Christensen's Plane Answers to Complex Questions, but not everybody will appreciate that approach. As another example, there's a geometric explanation in my answer at stats.stackexchange.com/a/62147/919, but not everybody views geometrical relations as "intuitive."
– whuber
Commented Aug 29, 2018 at 17:02
• Intuitively is like what does $(X^TX)^{-1} mean? Is it some kind of distance calculation or something, I don't understand it. Commented Aug 29, 2018 at 17:09 • That's fully explained in the answer I linked to. – whuber Commented Aug 29, 2018 at 17:27 • This question already exists here although possibly not with a satisfying answer math.stackexchange.com/questions/2624986/… Commented Aug 29, 2018 at 17:30 • – Tim Commented Jun 14, 2022 at 16:41 ## 3 Answers I found these posts particularly helpful: How to derive the least square estimator for multiple linear regression? Relationship between SVD and PCA. How to use SVD to perform PCA? http://www.math.miami.edu/~armstrong/210sp13/HW7notes.pdf If$X$is an$n \times p$matrix then the matrix$X(X^TX)^{-1}X^T$defines a projection onto the column space of$X$. Intuitively, you have an overdetermined system of equations, but still want to use it to define a linear map$\mathbb{R}^p \rightarrow \mathbb{R}$that will map rows$x_i$of$X$to something close to values$y_i$,$i\in \{1,\dots,n\}$. So we settle for sending$X$to the closest thing to$y$that can be expressed as a linear combination of your features (the columns of$X$). As far as an interpretation of$(X^TX)^{-1}$, I don't have an amazing answer yet. I know you can think of$(X^TX)$as basically being the covariance matrix of the dataset. •$(X^T X)$is sometimes referred to as a "scatter matrix" and is just a scaled up version of the covariance matrix Commented Mar 19, 2019 at 2:43 ## Geometric viewpoint A geometric viewpoint can be like the n-dimensional vectors $$y$$ and $$X\beta$$ being points in n-dimensional-space $$V$$. Where $$X\beta$$ is also in the subspace $$W$$ spanned by the vectors $$x_1, x_2, \cdots, x_m$$. ### Two types of coordinates For this subspace $$W$$ we can imagine two different types of coordinates: • The $$\boldsymbol{\beta}$$ are like coordinates for a regular coordinate space. The vector $$z$$ in the space $$W$$ are the linear combination of the vectors $$\mathbf{x_i}$$ $$z = \boldsymbol{\beta_1} \mathbf{x_1} + \boldsymbol{\beta_2} \mathbf{x_1} + .... \boldsymbol{\beta_m} \mathbf{x_m}$$ • The $$\boldsymbol{\alpha}$$ are not coordinates in the regular sense, but they do define a point in the subspace $$W$$. Each $$\alpha_i$$ relates to the perpendicular projections onto the vectors $$x_i$$. If we use unit vectors $$x_i$$ (for simplicity) then the "coordinates" $$\alpha_i$$ for a vector $$z$$ can be expressed as: $$\alpha_i = \mathbf{x_i^T} \mathbf{z}$$ and the set of all coordinates as: $$\boldsymbol{\alpha} = \mathbf{X^T} \mathbf{z}$$ ### Mapping between coordinates $$\boldsymbol{\alpha}$$ and $$\boldsymbol{\beta}$$ for $$\mathbf{z} = \mathbf{X}\boldsymbol{\beta}$$ the expression of "coordinates" $$\alpha$$ becomes a conversion from coordinates $$\beta$$ to "coordinates" $$\alpha$$ $$\boldsymbol{\alpha} = \mathbf{X^T} \mathbf{X}\boldsymbol{\beta}$$ You could see $$(\mathbf{X^T} \mathbf{X})_{ij}$$ as expressing how much each $$x_i$$ projects onto the other $$x_j$$ Then the geometric interpretation of $$(\mathbf{X^T} \mathbf{X})^{-1}$$ can be seen as the map from vector projection "coordinates" $$\boldsymbol{\alpha}$$ to linear coordinates $$\boldsymbol{\beta}$$. $$\boldsymbol{\beta} = (\mathbf{X^T} \mathbf{X})^{-1}\boldsymbol{\alpha}$$ The expression $$\mathbf{X^Ty}$$ gives the projection "coordinates" of $$\mathbf{y}$$ and $$(\mathbf{X^T} \mathbf{X})^{-1}$$ turns them into $$\boldsymbol{\beta}$$. Note: the projection "coordinates" of $$\mathbf{y}$$ are the same as projection "coordinates" of $$\mathbf{\hat{y}}$$ since $$(\mathbf{y-\hat{y}}) \perp \mathbf{X}$$. • A very similar account of the topic stats.stackexchange.com/a/124892/3277. Commented Aug 30, 2018 at 12:49 • Indeed very similar. To me this view is very new and I had to take a night to think about it. I did always view least squares regression in terms of a projection but in this viewpoint I have never tried to realize an intuitive meaning to the part$(X^TX)^{-1}$or I always saw it in the more indirect expression$X^T y = X^TX\beta$. Commented Aug 30, 2018 at 12:54 • How should we understand$X(X^TX)^{-1}X^T$in this perspective?$X^Ty$is the "$\alpha$-coordinates" of$y$, and so$(X^TX)^{-1}X^Ty$is the "$\beta$-coordinates" of$y$. But...doesn't left multiplying that$\beta$-coordinate by the coordinate bases$X$restore$y$? Commented Jun 14, 2022 at 14:20 • @whoknows it restores$\hat{y}$, the projection of$y$into the column space of$X$. Commented Jun 14, 2022 at 15:38 Assuming you're familiar with the simple linear regression: $$y_i=\alpha+\beta x_i+\varepsilon_i$$ and its solution: $$\beta=\frac{\mathrm{cov}[x_i,y_i]}{\mathrm{var}[x_i]}$$ It's easy to see how$X'y$corresponds to numerator above and$X'X$maps to denominator. Since we're dealing with matrices the order matters.$X'X$is KxK matrix, and$X'y$is Kx1 vector. Hence, the order is:$(X'X)^{-1}X'y\$

• But that analogy itself doesn't tell you if pre- or postmultiply with the inverse. Commented Aug 29, 2018 at 18:17
• @kjetilbhalvorsen, I put the order of operations Commented Aug 29, 2018 at 18:21