# Making sure that the design matrix is positive (semi-) definite

In Bayesian linear regression, how do I make sure that the design matrix produced by a neural network $$\Phi$$ is positive definite?

Computing the covariance matrix on the weight requires inverting --- i.e., $$(\Phi^\text{T}\Phi)^{-1}$$. For reference, the posterior on the weights follows: $$\theta \sim \mathcal{N}(m_N,S_N)$$ and prior $$\theta \sim \mathcal{N}(0,\alpha^{-1}I)$$

$$S_N^{-1} = \beta \cdot (\Phi^T\Phi)^{-1} + \alpha \cdot I$$ where $$\beta$$ is the noise precision on output.

$$m_N = \beta \cdot S_N \cdot \Phi^\text{T} \cdot t$$ where $$t$$ is the target on seen data.

• When $\Phi$ is any matrix, $\Phi^\prime\Phi$ is always positive semidefinite. (Proof: for any vector $x,$ $x^\prime(\Phi^\prime\Phi)x = ||\Phi x||^2$ is the square of a real number.) In practice, accumulation of floating point roundoff errors may sometimes make it appear not to be PSD by introducing extremely tiny negative eigenvalues: those can be treated as zeros. Note that semi-definite matrices are not necessarily invertible, anyway.
– whuber
Sep 9, 2021 at 12:25
• Thank you for your reply. Shouldn't it in this case require that $\Phi$ to have full rank? math.utah.edu/~zwick/Classes/Fall2012_2270/Lectures/… (Pg. 4) Sep 9, 2021 at 16:36
• Full rank is unnecessary: consider the case where $\Phi$ is the zero matrix, for instance. It's still positive-semidefinite (obviously!). Your reference refers to positive-definite matrices, not positive-semidefinite ones (as specified in the title to your question).
– whuber
Sep 9, 2021 at 20:09
• I am using a wide shallow network on low dimensional input to build this design matrix. I could solve the issue by reducing the number of output neurons from 100 to 80. Sep 19, 2021 at 16:04
• the premise of this question is mistaken: $S_N^{-1} = \beta\Psi^\top\Psi + \alpha \mathbf{I}$, not $\beta(\Psi^\top\Psi)^{-1} + \alpha \mathbf{I}$, i.e. the Bayesian prior precision is added to $\beta\Psi^\top\Psi$ before inversion, not after. The posterior thus has a full rank covariance even if the data don't (so long as the prior does). (Wikipedia's article has a derivation of this). Jul 31, 2023 at 21:09

If you want $$(\Phi^T\Phi)^{-1}$$ to be invertible (where $$\Phi \in R^{N \times K})$$ you need to ensure $$rk(\Phi) = K$$ (i.e. full rank). As mentioned in the comments, if $$\Phi$$ is rank deficient then $$\Phi^T\Phi$$ is positive semidefinite and thus not always invertible.