1
$\begingroup$

I am trying to learn about Bayesian networks and try to make them work in the context of a simple prediction problem. But my question is more theoretical:

For argument's sake, assume we have a dataset with three free and one dependent variable, all of them categorical.

First, build a Naive Bayes classifier. Second, learn the network structure of a Bayes network by some optimisation procedure.

My main question: In this context, is it ever possible for the Bayes Network to outperform Naive Bayes?

enter image description here

Here is my attempt at answering this question: Naive Bayes always uses 100% of the available information. However, a Bayes Network may have some nodes (like $x_2$ in the figure) that don't have an edge to the dependent variable. Both $x_1$ and $x_3$ are instantiated, so the information in $x_2$ is not used for classification. So, it is unlikely that the general Bayes network are better than Naive Bayes.

I only see two specific situations when the Bayes Network might outperform Naive Bayes.

  • Naive Bayes is more vulnerable to overfitting since it assumes its connections. Bayes Network learns a more "general" structure, which could make it less vulnerable.

  • Both models perform well if we have missing data (say, the value of $x_3$ is missing). However, in the general network we can predict $x_3$ from $x_2$, which may make the prediction more accurate compared to Naive Bayes.

Please let me know if you agree with my reasoning. Also, if you have concrete examples (projects, papers, datasets) where you have found useful application of Bayes Networks and if you are willing to share them, I would be very grateful!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Naive Bayes algorithm makes the simplistic assumption that all the features are independent of each other, hence the name "naive". What follows, any algorithm that makes more realistic assumptions about data can possibly outperform it. Whether it happens, it strongly depends on properties of your data and sometimes simple algorithms are more robust. Because of its simplicity, naive Bayes works quite well on smaller datasets. To estimate joint probabilities, it would need more data and it gets away with multiplying marginal probabilities.

Improve this answer
$\endgroup$
4
  • $\begingroup$ To put this into my example: Naive Bayes assumes that $x_1$ and $x_3$ are independent, while the network has learned that $x_3$ depends on $x_1$. But if we already know $x_3$, then we don't care that $x_1$ influences it, do we? $\endgroup$
    – Rafael Bankosegger
    Commented Apr 13, 2019 at 16:53
  • $\begingroup$ @RafaelBankosegger the answer depends on what is actual relationship between the variables etc. If Bayes Net would incorrectly learn irrelevant relationship, this wouldn't help with anything. $\endgroup$
    – Tim
    Commented Apr 13, 2019 at 17:15
  • $\begingroup$ Okay. But if it's a correctly learned relationship, how would it help? $\endgroup$
    – Rafael Bankosegger
    Commented Apr 13, 2019 at 17:42
  • $\begingroup$ @RafaelBankosegger you seem to ask is model that is closer to reality would be better, then the one that is more unrealistic. Yes, it should. Notice however that many other factors influence performance, e.g. quality of the data, so if the data is noisy and the model overfitts to the noise, then it wouldn't help; moreover naive Bayes is much simpler model, so it possibly would work better if your sample is small, etc. $\endgroup$
    – Tim
    Commented Apr 13, 2019 at 17:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.