Orthogonality of columns of the augmented design matrix for ridge regression

Question

In the question: How to derive the ridge regression solution? there is a solution by whuber, which describes how the columns of the augmented matrix approach pairwise orthogonality as the regularization strength increases. However, I am not able to reproduce this argument in the following example. Can someone explain what is incorrect or missing?

Suppose the original design matrix is $$A = \begin{pmatrix}1 & 2 \\ 1 & 2 \end{pmatrix},$$ so $rank(A) = 1.$ Further, suppose the augmented design matrix is $$B = \begin{pmatrix}1 & 2 \\ 1 & 2 \\ \nu & 0 \\ 0 & \nu \end{pmatrix},$$ where $\nu^{2} = \lambda$ is the regularization strength, so $rank(B) = 2.$ Then, the columns of $B$ are linearly independent, whereas the columns of $A$ are linearly dependent.

Now, the inner product is the standard inner product on $\mathbb{R}^{4},$ so we may take the transpose of the first column of $B$ times the second column of $B$ to obtain: $$\begin{pmatrix} 1 & 1 & \nu & 0\end{pmatrix} \begin{pmatrix} 2 \\ 2 \\ 0 \\ \nu \end{pmatrix} = 4.$$ However, this inner product is nonzero, so the columns are not pairwise orthogonal.

Answer 1

Take $A^TA$:

$$A^TA = \begin{pmatrix}1 & 1 \\ 2 & 2 \end{pmatrix}\begin{pmatrix}1 & 2 \\ 1 & 2 \end{pmatrix}=\begin{pmatrix}2 & 4 \\ 4 & 8 \end{pmatrix},$$

and compare with $B^TB$:

$$B^TB = \begin{pmatrix}1 & 2 \\ 1 & 2 \\ \nu & 0 \\ 0 & \nu \end{pmatrix}^T\begin{pmatrix}1 & 2 \\ 1 & 2 \\ \nu & 0 \\ 0 & \nu \end{pmatrix}\\= \begin{pmatrix} 1 & 1 & \nu & 0 \\2 & 2 & 0 & \nu \end{pmatrix}\begin{pmatrix}1 & 2 \\ 1 & 2 \\ \nu & 0 \\ 0 & \nu \end{pmatrix} =\\ \begin{pmatrix}2 + \nu^2 & 4 \\ 4 & 8 + \nu^2 \end{pmatrix} $$

The bigger $\nu$ is, the more $B^TB$ resembles an (scaled) identity matrix.

Answer 2

The claim in whuber's answer that the vectors are becoming "more orthogonal" is ambiguous. I would take it to mean that the correlation is getting closer to $0$ as $\nu$ gets bigger. In $A$, the correlation of the columns is $1$. In $B$, the centered correlation is given by $$\frac{-\nu^2/4 - 3\nu/2 + 2}{\sqrt{(3\nu^2 / 4 - \nu + 1)(3\nu^2/4 - 2\nu + 4)}},$$ which approaches $-1/3$ from below as $\nu \rightarrow \infty$.

Using the definition that the orthogonality of columns $c_1, c_2$ is $$\frac{c_1 \cdot c_2}{\sqrt{|c_1|^2 |c_2|^2}},$$ we have that the orthogonality of the columns of $A$ is 1, and the orthogonality of the columns of $B$ is $$\frac{4}{\sqrt{(1^2 + 1^2 + \nu^2)(2^2 + 2^2 + \nu^2)}} = \frac{4}{\sqrt{(2 + \nu^2)(8 + \nu^2)}},$$ which decreases from $1$ to $0$ as $\nu$ increases from $0$ to $\infty$.