0
$\begingroup$

Suppose I have a function $g\in L_2(\mathbb{R})$, and we observe variables two -vectors $(Y_i,X_i)$ such that $Y_i = g(X_i) + U_i$ for some IID error terms $U_i$. If I want to estimate $g$, I want to use an orthonormal basis of $\mathbb{R}$, which we will call the set $\{e_i\}_{i\geq 0}$. Then we could write $g(x)$ as $$ \begin{align*} g(x) \sim \sum_{k=0}^{\infty} \alpha_k e_k(x), \end{align*} $$ and in the spirit of having a finite sample amount of data, we have truncate the series to a sum from 0 to $p_n$ such that $p_n \rightarrow \infty$, and $p_n = o(n)$. Then using a least squares algorithm, we obtain estimates $$ \begin{align*} \boldsymbol{\alpha}_p = (\boldsymbol{A}^\text{T}\boldsymbol{A})^{-1}\boldsymbol{A}^\text{T}\boldsymbol{Y} \end{align*} $$

where $\boldsymbol{\alpha}_p = (\alpha_0,...,\alpha_{p_n})^{-1}.$ My question has to do with the invertibility of the matrix $\boldsymbol A^\text{T}\boldsymbol A$.

$\boldsymbol{QUESTION}$: How do I guarantee that $\boldsymbol A^\text{T}\boldsymbol A$ is invertible? Or more importantly, is it stochastically invertible? I know that the nonzero eigenvalues of $\boldsymbol A^\text{T}\boldsymbol A$ are the same the as the nonzero eigenvalues of $\boldsymbol{A}\boldsymbol{A}^\text{T}$, but the size of the latter matrix is smaller than the latter of the former matrix, which implies that some of the eigenvalues of $\boldsymbol{A}^\text{T}\boldsymbol{A}$ must be zero, however when I run simulations, I seem to get decent estimators using the above formula for $\boldsymbol{\alpha}_p$. So I am not sure how to justify the invertibility of $\boldsymbol{A}^\text{T}\boldsymbol{A}$. Any help would be appreciated.

$\endgroup$
0
$\begingroup$

The classical thing to do here is to $$\text{replace } (A^\intercal A)^{-1} \text{ with } (A^\intercal A + \lambda I)^{-1} $$ for some $\lambda > 0$. Notice that $A^\intercal A$ is always positive semi-definite, so $A^\intercal A + \lambda I$ must be invertible.

This regularization trick has many names, including ridge regression and Tikhonov regularization, and underlies much of the literature on smoothing splines and kernel regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.