1
$\begingroup$

In Bayesian linear regression, how do I make sure that the design matrix produced by a neural network $ \Phi$ is positive definite?

Computing the covariance matrix on the weight requires inverting --- i.e., $(\Phi^\text{T}\Phi)^{-1}$. For reference, the posterior on the weights follows: $\theta \sim \mathcal{N}(m_N,S_N)$ and prior $\theta \sim \mathcal{N}(0,\alpha^{-1}I)$

$S_N^{-1} = \beta \cdot (\Phi^T\Phi)^{-1} + \alpha \cdot I$ where $\beta$ is the noise precision on output.

$m_N = \beta \cdot S_N \cdot \Phi^\text{T} \cdot t$ where $t$ is the target on seen data.

Improve this question
$\endgroup$
4
  • 5
    $\begingroup$ When $\Phi$ is any matrix, $\Phi^\prime\Phi$ is always positive semidefinite. (Proof: for any vector $x,$ $x^\prime(\Phi^\prime\Phi)x = ||\Phi x||^2$ is the square of a real number.) In practice, accumulation of floating point roundoff errors may sometimes make it appear not to be PSD by introducing extremely tiny negative eigenvalues: those can be treated as zeros. Note that semi-definite matrices are not necessarily invertible, anyway. $\endgroup$
    – whuber ♦
    Sep 9, 2021 at 12:25
  • $\begingroup$ Thank you for your reply. Shouldn't it in this case require that $\Phi$ to have full rank? math.utah.edu/~zwick/Classes/Fall2012_2270/Lectures/… (Pg. 4) $\endgroup$
    – justcurios
    Sep 9, 2021 at 16:36
  • 1
    $\begingroup$ Full rank is unnecessary: consider the case where $\Phi$ is the zero matrix, for instance. It's still positive-semidefinite (obviously!). Your reference refers to positive-definite matrices, not positive-semidefinite ones (as specified in the title to your question). $\endgroup$
    – whuber ♦
    Sep 9, 2021 at 20:09
  • $\begingroup$ I am using a wide shallow network on low dimensional input to build this design matrix. I could solve the issue by reducing the number of output neurons from 100 to 80. $\endgroup$
    – justcurios
    Sep 19, 2021 at 16:04

1 Answer 1

Reset to default
0
$\begingroup$

If you want $(\Phi^T\Phi)^{-1}$ to be invertible (where $\Phi \in R^{N \times K})$ you need to ensure $rk(\Phi) = K$ (i.e. full rank). As mentioned in the comments, if $\Phi$ is rank deficient then $\Phi^T\Phi$ is positive semidefinite and thus not always invertible.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.