Timeline for How does addition of a regularization term ensures that the matrix is nonsingular? ( least squares )
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 10, 2022 at 21:20 | answer | added | Aksakal | timeline score: 1 | |
| Jan 10, 2022 at 14:33 | answer | added | Sycorax♦ | timeline score: 6 | |
| Jan 10, 2022 at 9:36 | vote | accept | Mas A | ||
| Jan 10, 2022 at 7:55 | answer | added | Eric Johnson | timeline score: 3 | |
| Jan 7, 2022 at 20:48 | comment | added | Mas A | Thank you @whuber for your comments and the url that you shared! | |
| Jan 7, 2022 at 20:46 | comment | added | Mas A | How could I not think of that, of course, it makes sense! Thank you so much for your explanations @Sycorax | |
| Jan 7, 2022 at 18:59 | comment | added | whuber♦ | stats.stackexchange.com/questions/69205 is closely related: it discusses Ridge Regression, which is an example of this form of regularization. | |
| Jan 7, 2022 at 18:31 | comment | added | Sycorax♦ | @whuber That’s a much better demonstration! | |
| Jan 7, 2022 at 18:26 | comment | added | whuber♦ | @Sycorax That machinery is overkill. In the context $A=\Phi^\prime T\Phi$ must be square and positive semi-definite. (Otherwise the conclusion is incorrect.) Now, $\lambda+A$ is singular for a positive real number $\lambda$ if and only if there exists nonzero $x$ for which $(\lambda + A)x=0$ (that's the definition of "singular"). That's obviously equivalent to the eigenvalue equation $Ax=-\lambda x,$ yet we know (by supposition) that $A$ has no negative eigenvalues, QED. | |
| Jan 7, 2022 at 15:40 | comment | added | Sycorax♦ | Here's a sketch: Any $M=\Phi^T \Phi$ (for real $\Phi$) must be positive semi-definite (sums of squares must be non-negative). Adding $\lambda I$ (for $\lambda > 0$) guarantees that the singular values must be positive (sum of positive number and non-negative number must be positive). You can demonstrate this by writing down the SVD of $\Phi$ and working through the algebra of $M + \lambda I$. | |
| Jan 7, 2022 at 15:27 | history | asked | Mas A | CC BY-SA 4.0 |