1
$\begingroup$

In Bayesian linear regression, how do I make sure that the design matrix produced by a neural network $ \Phi$ is positive definite?

Computing the covariance matrix on the weight requires inverting --- i.e., $(\Phi^\text{T}\Phi)^{-1}$. For reference, the posterior on the weights follows: $\theta \sim \mathcal{N}(m_N,S_N)$ and prior $\theta \sim \mathcal{N}(0,\alpha^{-1}I)$

$S_N^{-1} = \beta \cdot (\Phi^T\Phi)^{-1} + \alpha \cdot I$ where $\beta$ is the noise precision on output.

$m_N = \beta \cdot S_N \cdot \Phi^\text{T} \cdot t$ where $t$ is the target on seen data.

$\endgroup$
5
  • 5
    $\begingroup$ When $\Phi$ is any matrix, $\Phi^\prime\Phi$ is always positive semidefinite. (Proof: for any vector $x,$ $x^\prime(\Phi^\prime\Phi)x = ||\Phi x||^2$ is the square of a real number.) In practice, accumulation of floating point roundoff errors may sometimes make it appear not to be PSD by introducing extremely tiny negative eigenvalues: those can be treated as zeros. Note that semi-definite matrices are not necessarily invertible, anyway. $\endgroup$
    – whuber
    Sep 9, 2021 at 12:25
  • $\begingroup$ Thank you for your reply. Shouldn't it in this case require that $\Phi$ to have full rank? math.utah.edu/~zwick/Classes/Fall2012_2270/Lectures/… (Pg. 4) $\endgroup$
    – justcurios
    Sep 9, 2021 at 16:36
  • 1
    $\begingroup$ Full rank is unnecessary: consider the case where $\Phi$ is the zero matrix, for instance. It's still positive-semidefinite (obviously!). Your reference refers to positive-definite matrices, not positive-semidefinite ones (as specified in the title to your question). $\endgroup$
    – whuber
    Sep 9, 2021 at 20:09
  • $\begingroup$ I am using a wide shallow network on low dimensional input to build this design matrix. I could solve the issue by reducing the number of output neurons from 100 to 80. $\endgroup$
    – justcurios
    Sep 19, 2021 at 16:04
  • $\begingroup$ the premise of this question is mistaken: $S_N^{-1} = \beta\Psi^\top\Psi + \alpha \mathbf{I}$, not $\beta(\Psi^\top\Psi)^{-1} + \alpha \mathbf{I}$, i.e. the Bayesian prior precision is added to $\beta\Psi^\top\Psi$ before inversion, not after. The posterior thus has a full rank covariance even if the data don't (so long as the prior does). (Wikipedia's article has a derivation of this). $\endgroup$ Jul 31, 2023 at 21:09

1 Answer 1

1
$\begingroup$

If you want $(\Phi^T\Phi)^{-1}$ to be invertible (where $\Phi \in R^{N \times K})$ you need to ensure $rk(\Phi) = K$ (i.e. full rank). As mentioned in the comments, if $\Phi$ is rank deficient then $\Phi^T\Phi$ is positive semidefinite and thus not always invertible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.