In my opinion, Flach’s statement regarding the role of the term $({X^TX})^{-1}$ is fuzzy and should be restated for clarity. It seems a reference to the Mahalanobis’ distance, as @doubled properly pointed out, but I was unable to follow his reasoning, as I remarked in the comments section of his answer.
I never had a formal training in mathematics, which maybe explains the trouble I am experiencing to grasp @doubled’s answer, but after a while I have got the rationale behind Flach’s statement. Now it is clear to me that Flach resorts to the reader’s intuition to introduce subjects that actually require some math background, which is essentially good but has a side effect: those who have more than zero background in Mathematics struggle to understand his reasoning given that it is based almost exclusively on inspection instead of a formal mathematical derivation. In a nutshell, I concluded that it was not I that failed to get Flach's point, but he that failed to demonstrate it clearly. Anyway, that is a minor issue in his book, which is sound and comprehensive. Given that a few people up voted my question, I felt it was convenient post my conclusions here, as follows.
Statement of the Problem
Claim: $({X^TX})^{-1}$ acts as a whitening transformation that decorrelates, centres and normalises the features $X$
Analysis: such a claim seems related to the definition of the Mahalanobis distance $M_D$, which employs the term $({X^TX})^{-1}$ in its formulation to normalize the values of distances calculated in a space characterized by non-spherical distributions. Centering consists in subtracting $\mathbb{E}X$ from $X$, which is not what $(X^TX)^{-1}$ does in the Mahalanobis’ formula. Decorrelating and normalizing a random variable is an algebraic procedure known as whitening and no whitening procedure I am aware of uses the term $(X^TX)^{-1}$. The so called Mahalanobis whitening is defined by the term $(X^TX)^{-\frac{1}{2}}$, which can be derived from $M_D$.
Proof: The reasoning proceeds in four steps: (1) the whitening procedures is succinctly described, (2) some remarks and assumptions are made, (3) the Mahalanobis’ is scrutinized, and (4) it is shown that the normal equations lead to a certain “hat matrix” that implicitly refer to a whitening procedure known as Mahalanobis whitening. With that, I show what Flach really meant and put his (bold) statement in perspective: no, $({X^TX})^{-1}$ is not a whitening transformation.
STEP (1) – Whitening
Whitening is a linear transformation intended to both normalize and decorrelate a given random variable $X$. In multivariate analysis, $X$ is a matrix whose rows $x_i$ are the realizations (observations) of some random process characterized by some features (the columns of $X$). As decorrelated multivariate data show a circular pattern when plotted, this transformation is also known as sphering.
By definition, $W$ is a whitening transformation (a sphering matrix) if $Z=XW$ and the covariance of the random variable $Z$, $\Sigma_Z$, is diagonal and unitary; i.e., $\Sigma_Z = \mathbb{I}$. It is trivial to show that $\Sigma_Z = \mathbb{I} \implies W W^T=\Sigma^{-1}_x$.
STEP (2) – Assumptions
- Assumption 0 (whitening transformation): let $X$ a random variable with a non-diagonal covariance matrix $\Sigma_x$. If we define a whitening matrix $W$ such that $Z=X W$ and $\Sigma_Z = \mathbb{I}$, then it is trivial to show that $W W^T=\Sigma^{-1}_x$
- Assumption 1 (Sample covariance): $\mathbb{E} X^TX = \Sigma_x=\frac{1}{N} X^TX $ if, and only if, $\mathbb{E} X$ = 0
- Assumption 2 (definition of the matrix square root): Differently from real numbers, a matrix can have several square roots. By definition, a matrix $A_{sqrt}$ is said to be a square root of $B$ if the matrix product $A_{sqrt} A_{sqrt} = B$. Equivalently, $\Sigma_x = {\Sigma^\frac{1}{2}_x} {\Sigma^\frac{1}{2}_x}$
- Assumption 3 (the square root of $\Sigma_x$ is symmetric): $\Sigma^\frac{1}{2}_x = {(\Sigma^\frac{1}{2}_x})^T$
- Assumption 4 (squaring and inversion are commutative): ${\Sigma^{-\frac{1}{2}}_x} = {(\Sigma^\frac{1}{2}_x)}^{-1}$
- Assumption 5 (covariance of a linear transformation): $\mathbb{E} X^TX = \Sigma_x$ implies that the covariance of a linear transformation $AX$ is the covariance of $A(X-\mathbb{E} X)$ which is $A\Sigma_X A^T$
- Assumption 6 (normal equations): given an unknown multivariate function $y=f(x)$, the estimated coefficients of the corresponding linear regression analysis are collected in a vector $\hat{\beta}$ such that $\hat{\beta}= (X^TX)^{-1} X^T Y $
STEP (3) – Mahalanobi’s distance
The Mahalanobis’ distance $D_M$ gives the dissimilarity degree between two random vectors $u$ and $v$ in a features space characterized by a distribution $f(x)$ whose covariance matrix is $\Sigma_x$. It may be thought as a generalized form for the Euclidean distance given that it weights the Euclidean distance by $\Sigma_x^{-1}$, as given by the formula $D_M=\sqrt{u^T \Sigma_x^{-1} v}$.
By weighting the Euclidean distance with the inverse covariance matrix of the underlying distribution of $X$, the Mahalanobis’ distance considers how data points spread out around their mean in the Cartesian space, something ignored by its Euclidean counterpart. As a matter of fact, if the spread is symmetrical (spherical) the covariance matrix will be diagonal and both Euclidean and Mahalanobis distance will be equivalent in the sense that the loci defined by a constant distance will be a sphere. If the spread is not symmetrical, a constant Mahalanobi’s distance will still define a sphere due the weighting factor $\Sigma_x^{-1}$, but the Euclidean one will define a ellipsoid (here).
Often, it is convenient to consider the Mahalanobis’ distance as a multivariate generalization of of the univariate standardization procedure (z-scores), in which the distance between $u$ and $v$ is measured in standard deviations.
Consider the problem of computing the weighted distance between the points $x$ and $\mu = \mathbb{E} X$, under Assumptions 2 and 3. For convenience, we will deal with the squared Mahalanobis’ distance, as follows:
$
D^2_M = (x-\mu)^T \Sigma_x^{-1} (x-\mu) \\
D^2_M = (x-\mu)^T (\Sigma_x^{-\frac{1}{2}} \Sigma_x^{-\frac{1}{2}})(x-\mu) \\
D^2_M = ((x-\mu)^T \Sigma_x^{-\frac{1}{2}}) (\Sigma_x^{-\frac{1}{2}}(x-\mu)) \\
D^2_M = (\Sigma_x^{-\frac{1}{2}} (x-\mu))^T (\Sigma_x^{-\frac{1}{2}}(x-\mu))
$
If we define $z \triangleq \Sigma_x^{-\frac{1}{2}} (x-\mu) $, then
$
D^2_M = z^T z = ||z||
$
We note that $z$ is the result of a linear transformation given by $z=\Sigma^{-\frac{1}{2}}_x (x-\mu)$.By assumptions 2 and 5, the covariance of $z$ can be computed as
$
\Sigma_z = \textrm{cov}(Z)= \textrm{cov} (\Sigma^{-\frac{1}{2}}_x (X-\mu)) = \textrm{cov} (\Sigma^{-\frac{1}{2}}_x X) \\
\textrm{cov} (\Sigma^{-\frac{1}{2}}_x X) = \Sigma^{-\frac{1}{2}}_x \Sigma_x (\Sigma^{-\frac{1}{2}}_x)^T =
\Sigma^{-\frac{1}{2}}_x (\Sigma^{\frac{1}{2}}_x \Sigma^{\frac{1}{2}}_x) \Sigma^{-\frac{1}{2}}_x =
(\Sigma^{-\frac{1}{2}}_x \Sigma^{\frac{1}{2}}_x) (\Sigma^{\frac{1}{2}}_x \Sigma^{-\frac{1}{2}}_x) = \mathbb{I}
$
So, we conclude that the transformation $Z=\Sigma^{-\frac{1}{2}}_x X$ is a whitening transformation with $W=\Sigma^{-\frac{1}{2}}_x $. In fact, this kind of whitening is called ZCA whitening (where ZCA stands for “zero-phase components analysis”) or Mahalanobis whitening (here).
STEP (4) – The Hat Matrix
From the multivariate regression analysis, the estimates $\hat{Y}$ are given in function of a set of estimated parameters $\hat{\beta}$; i.e.,
$
\hat{Y}=X \hat{\beta} \\
\hat{\beta}= (X^TX)^{-1} X^T Y \\
\therefore \hat{Y}= X (X^TX)^{-1} X^T Y \\
$
Using this result and Assumption 6, we can define the so-called hat matrix $H$ and define $\hat{Y}$ in terms of $H$:
$H \triangleq X (X^TX)^{-1} X^T \implies \hat{Y}=HY$
, where $\hat{Y}=HY$ justifies the mnemonic "the hat matrix puts a hat on y". Now, let us pay a closer attention to the hat matrix $H= X (X^TX)^{-1} X^T$ and factor it as appropriate, using Assumptions 0, 1 and 3 and, furthermore, assuming that $X$ is zero-centered:
$
H = X (X^TX)^{-1} X^T = \\
H = N X \Sigma_x^{-1} X^T = \\
H = N X (\Sigma_x^{-\frac{1}{2}} \Sigma_x^{-\frac{1}{2}}) X^T = \\
H = N (X \Sigma_x^{-\frac{1}{2}}) ( \Sigma_x^{-\frac{1}{2}} X^T) = \\
H = N (X \Sigma_x^{-\frac{1}{2}}) (X \Sigma_x^{-\frac{1}{2}}) ^T \\
\therefore \hat{Y} = N (X \Sigma_x^{-\frac{1}{2}}) (X \Sigma_x^{-\frac{1}{2}}) ^T Y
$
Now we have all we need to establish if $\Sigma_x^{-1}$ effectively decorrelates, centres and normalizes the features $X$ as stated by Flach. By factoring the hat matrix definition as above and defining $Z\triangleq X \Sigma_x^{-\frac{1}{2}}$ then we have
$
\hat{Y} = N Z Z^T Y
$
Hence, the linear regression actually decorrelates $X$ via the aforementioned Mahalanobis whitening, something we represented by $Z= X \Sigma_x^{-\frac{1}{2}}$ in the results above. Right afterwards, this result is squared ($Z Z^T$) and then multiplied by $Y$ (and $N$). So, yes, to “put a hat in Y” amounts to whitening $X$ as an intermediary step – but that does not mean that $\Sigma_x^{-1}$ “decorrelates the features”. Flach probably meant something like
“the term $\Sigma_x^{-1}$ appears in the hat matrix multiplied by $X$ on both sides. Given that
$\Sigma_x^{-1}=\Sigma_x^{-\frac{1}{2}} \Sigma_x^{-\frac{1}{2}}$
, it is clear that $\hat{Y}$ is a function of a decorrelated version of $X$”.
A huge difference from the original statement – specially considering Flach didn’t defined the hat matrix $H$. It is pretty hard to follow his reasoning without an extensive derivation as I did here.
In a nutshell: it is inaccurate to state that $(X^TX)^{-1}$ acts as a transformation that decorrelates, centres and normalises the features. It is, at best, a reference to another context (Mahalanobis’ formula) and, therefore, should be put in perspective in Flach’s book.