My prof. said in the class that for Bayes decision rule, the likelihood is Gaussian and in practice, we will almost always work with a diagonal $\Sigma$. Why is that? I know that a diagonal $\Sigma$ means, the features are independent, but why make this assumption?

  • 4
    $\begingroup$ It's an assumption that's generally false but it simplifies things a great deal. It may be useful to read up on naive Bayes: en.wikipedia.org/wiki/Naive_Bayes_classifier $\endgroup$
    – dsaxton
    Jul 16, 2015 at 19:35
  • $\begingroup$ Perhaps the most important thing that could be said is that the simplification is best when the off-diagonal elements are very small -- the larger these off-diagonal elements are in reality, the worse this simplification will be in practice. Whether or not a diagonal $\Sigma$ is appropriate for a particular problem depends on what, precisely, you want to do. $\endgroup$
    – Sycorax
    Jul 16, 2015 at 19:50
  • 4
    $\begingroup$ In addition to what comments above have said, one reason for assuming $\Sigma$ is diagonal in practice is that this reduces the number of parameters from $O(D^2)$ to $O(D)$. Obviously, it is easier to estimate $O(D)$ parameters, so using a diagonal $\Sigma$ can be an advantage from a bias-variance tradeoff perspective. This is in addition to the computational benefits of a diagonal $\Sigma$. $\endgroup$
    – guy
    Jul 16, 2015 at 20:32
  • $\begingroup$ Interesting, does the fact using diagonal covariance matrix implies feature independence assumption or this is not related? $\endgroup$ Jul 16, 2015 at 22:04
  • 2
    $\begingroup$ @xeon One can justify diagonal covariances through models which assume independence. However, there is nothing stopping us from using a method which "assumes" independence even when we know independence fails. We can still ask the question "if independence fails, does that mean that estimates based on diagonal covariance matrices will do poorly?" And, as noted by Fisher, the benefit of using fewer parameters may outweigh the drawback of independence failing. This is the bias-variance tradeoff - we introduce bias by using diagonal $\Sigma$, but we gain more in variance reduction. $\endgroup$
    – guy
    Jul 18, 2015 at 16:22


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.