# On using an orthogonal series to estimate a regression function

Suppose I have a function $g\in L_2(\mathbb{R})$, and we observe variables two -vectors $(Y_i,X_i)$ such that $Y_i = g(X_i) + U_i$ for some IID error terms $U_i$. If I want to estimate $g$, I want to use an orthonormal basis of $\mathbb{R}$, which we will call the set $\{e_i\}_{i\geq 0}$. Then we could write $g(x)$ as \begin{align*} g(x) \sim \sum_{k=0}^{\infty} \alpha_k e_k(x), \end{align*} and in the spirit of having a finite sample amount of data, we have truncate the series to a sum from 0 to $p_n$ such that $p_n \rightarrow \infty$, and $p_n = o(n)$. Then using a least squares algorithm, we obtain estimates \begin{align*} \boldsymbol{\alpha}_p = (\boldsymbol{A}^\text{T}\boldsymbol{A})^{-1}\boldsymbol{A}^\text{T}\boldsymbol{Y} \end{align*}

where $\boldsymbol{\alpha}_p = (\alpha_0,...,\alpha_{p_n})^{-1}.$ My question has to do with the invertibility of the matrix $\boldsymbol A^\text{T}\boldsymbol A$.

$\boldsymbol{QUESTION}$: How do I guarantee that $\boldsymbol A^\text{T}\boldsymbol A$ is invertible? Or more importantly, is it stochastically invertible? I know that the nonzero eigenvalues of $\boldsymbol A^\text{T}\boldsymbol A$ are the same the as the nonzero eigenvalues of $\boldsymbol{A}\boldsymbol{A}^\text{T}$, but the size of the latter matrix is smaller than the latter of the former matrix, which implies that some of the eigenvalues of $\boldsymbol{A}^\text{T}\boldsymbol{A}$ must be zero, however when I run simulations, I seem to get decent estimators using the above formula for $\boldsymbol{\alpha}_p$. So I am not sure how to justify the invertibility of $\boldsymbol{A}^\text{T}\boldsymbol{A}$. Any help would be appreciated.

## 1 Answer

The classical thing to do here is to $$\text{replace } (A^\intercal A)^{-1} \text{ with } (A^\intercal A + \lambda I)^{-1}$$ for some $\lambda > 0$. Notice that $A^\intercal A$ is always positive semi-definite, so $A^\intercal A + \lambda I$ must be invertible.

This regularization trick has many names, including ridge regression and Tikhonov regularization, and underlies much of the literature on smoothing splines and kernel regression.