I am trying to learn about Bayesian networks and try to make them work in the context of a simple prediction problem. But my question is more theoretical:
For argument's sake, assume we have a dataset with three free and one dependent variable, all of them categorical.
First, build a Naive Bayes classifier. Second, learn the network structure of a Bayes network by some optimisation procedure.
My main question: In this context, is it ever possible for the Bayes Network to outperform Naive Bayes?
Here is my attempt at answering this question: Naive Bayes always uses 100% of the available information. However, a Bayes Network may have some nodes (like $x_2$ in the figure) that don't have an edge to the dependent variable. Both $x_1$ and $x_3$ are instantiated, so the information in $x_2$ is not used for classification. So, it is unlikely that the general Bayes network are better than Naive Bayes.
I only see two specific situations when the Bayes Network might outperform Naive Bayes.
Naive Bayes is more vulnerable to overfitting since it assumes its connections. Bayes Network learns a more "general" structure, which could make it less vulnerable.
Both models perform well if we have missing data (say, the value of $x_3$ is missing). However, in the general network we can predict $x_3$ from $x_2$, which may make the prediction more accurate compared to Naive Bayes.
Please let me know if you agree with my reasoning. Also, if you have concrete examples (projects, papers, datasets) where you have found useful application of Bayes Networks and if you are willing to share them, I would be very grateful!