# Making sure that the design matrix is positive (-semi) definite

In bayesian linear regression, how to make sure that the design matrix produced by a neural network $$\Phi$$ is positive definite? Because to computing the covariance matrix on the weight requires inverting .i.e. $$(\Phi^T\Phi)^{-1}$$

For reference, the posterior on the weights follows: $$\theta \sim \mathcal{N}(m_N,S_N).$$ and prior $$\theta \sim \mathcal{N}(0,\alpha^{-1}I)$$

$$S_N^{-1} = \beta.(\Phi^T\Phi)^{-1} + \alpha.I$$ where $$\beta$$ is the noise precision on output.

$$m_N = \beta.S_N.\Phi^T.t$$ where $$t$$ is the target on seen data.

• When $\Phi$ is any matrix, $\Phi^\prime\Phi$ is always positive semidefinite. (Proof: for any vector $x,$ $x^\prime(\Phi^\prime\Phi)x = ||\Phi x||^2$ is the square of a real number.) In practice, accumulation of floating point roundoff errors may sometimes make it appear not to be PSD by introducing extremely tiny negative eigenvalues: those can be treated as zeros. Note that semi-definite matrices are not necessarily invertible, anyway.
– whuber
Sep 9 at 12:25
• Thank you for your reply. Shouldn't it in this case require that $\Phi$ to have full rank? math.utah.edu/~zwick/Classes/Fall2012_2270/Lectures/… (Pg. 4) Sep 9 at 16:36
• Full rank is unnecessary: consider the case where $\Phi$ is the zero matrix, for instance. It's still positive-semidefinite (obviously!). Your reference refers to positive-definite matrices, not positive-semidefinite ones (as specified in the title to your question).
– whuber
Sep 9 at 20:09
• I am using a wide shallow network on low dimensional input to build this design matrix. I could solve the issue by reducing the number of output neurons from 100 to 80. Sep 19 at 16:04

If you want $$(\Phi^T\Phi)^{-1}$$ to be invertible (where $$\Phi \in R^{N \times K})$$ you need to ensure $$rk(\Phi) = K$$ (i.e. full rank). As mentioned in the comments, if $$\Phi$$ is rank deficient then $$\Phi^T\Phi$$ is positive semidefinite and thus not always invertible.