0
$\begingroup$

In bayesian linear regression, how to make sure that the design matrix produced by a neural network $ \Phi$ is positive definite? Because to computing the covariance matrix on the weight requires inverting .i.e. $(\Phi^T\Phi)^{-1}$

For reference, the posterior on the weights follows: $\theta \sim \mathcal{N}(m_N,S_N).$ and prior $\theta \sim \mathcal{N}(0,\alpha^{-1}I)$

$S_N^{-1} = \beta.(\Phi^T\Phi)^{-1} + \alpha.I$ where $\beta$ is the noise precision on output.

$m_N = \beta.S_N.\Phi^T.t$ where $t$ is the target on seen data.

$\endgroup$
4
  • 3
    $\begingroup$ When $\Phi$ is any matrix, $\Phi^\prime\Phi$ is always positive semidefinite. (Proof: for any vector $x,$ $x^\prime(\Phi^\prime\Phi)x = ||\Phi x||^2$ is the square of a real number.) In practice, accumulation of floating point roundoff errors may sometimes make it appear not to be PSD by introducing extremely tiny negative eigenvalues: those can be treated as zeros. Note that semi-definite matrices are not necessarily invertible, anyway. $\endgroup$
    – whuber
    Sep 9 at 12:25
  • $\begingroup$ Thank you for your reply. Shouldn't it in this case require that $\Phi$ to have full rank? math.utah.edu/~zwick/Classes/Fall2012_2270/Lectures/… (Pg. 4) $\endgroup$
    – justcurios
    Sep 9 at 16:36
  • $\begingroup$ Full rank is unnecessary: consider the case where $\Phi$ is the zero matrix, for instance. It's still positive-semidefinite (obviously!). Your reference refers to positive-definite matrices, not positive-semidefinite ones (as specified in the title to your question). $\endgroup$
    – whuber
    Sep 9 at 20:09
  • $\begingroup$ I am using a wide shallow network on low dimensional input to build this design matrix. I could solve the issue by reducing the number of output neurons from 100 to 80. $\endgroup$
    – justcurios
    Sep 19 at 16:04
0
$\begingroup$

If you want $(\Phi^T\Phi)^{-1}$ to be invertible (where $\Phi \in R^{N \times K})$ you need to ensure $rk(\Phi) = K$ (i.e. full rank). As mentioned in the comments, if $\Phi$ is rank deficient then $\Phi^T\Phi$ is positive semidefinite and thus not always invertible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.