What is the difference between neural network, Bayesian network, decision tree and Petri nets, even though they are all graphical models and visually depict cause-effect relationship.
Wow, what a big question! The short version of the answer is that just because you can represent two models using diagrammatically similar visual representations, doesn't mean they are even remotely related structurally, functionally, or philosophically. I'm not familiar with FCM or NF, but I can speak to the other ones a bit.
In a Bayesian network, the graph represents the conditional dependencies of different variables in the model. Each node represents a variable, and each directed edge represents a conditional relationship. Essentially, the graphical model is a visualization of the chain rule.
In a neural network, each node is a simulated "neuron". The neuron is essentially on or off, and its activation is determined by a linear combination of the values of each output in the preceding "layer" of the network.
Let's say we are using a decision tree for classification. The tree essentially provides us with a flowchart describing how we should classify an observation. We start at the root of the tree, and the leaf where we end up determines the classification we predict.
As you can see, these three models really have basically nothing at all to do with each other besides being representable with boxes and arrows.
It is easy to show (see Daphne Koller's course) that Logistic Regression is a restricted version of Conditional Random Fields, which are undirected graphical models, while Bayesian Networks are directed graphical models. Then, Logistic Regression could also be viewed as a single layer perceptron. This is the only link (which is very loose) that I think could be drawn between Bayesian Networks and Neural Networks.
I have yet to find a link between the other concepts you asked about.
First we attempt to state the nature of problem attempted to be solved by these methods. If a problem is straightforward, Polynomial or NP Complete we have ready to plug algorithms that could provide with a deterministic answer, by simple recombination of the axioms along logical rules. However, if that is not the case, we would have to rely on a method of reasoning, wherein, we attempt to treat the problem as being heterogeneous and plug it to a network, the nodes being evaluations and the edges being pathways between the components.
In any kind of network based reasoning, we do not reason deductively, by using abstract generalisations and combinations, according to logical rules in a linear flow, but rather work through the problem based on the propagation of reasoning in different directions, such that we solve the problem one node at a time, open to improvements on discovery of new facts concerning any node in the future. Now let us see how each of these techniques approach this method of problem solving in their own way.
Neural Network: The Neural network is a black box, where it is believed (never could be verified from outside the system) that connections among simpleton nodes are formed and emphasised by repeated external reinforcements. It approaches the problem in a Connectionsitic paradigm. The problem is likely solved, but there is little by way of explainability. The neural net now widely used because of its ability to produce quick results, if the problem with explainability is overlooked.
Bayesian Network: The Bayesian Network is a directed acyclic graph, which more like the flowchart, only that the flow chart can have cyclic loops. The Bayesian network unlike the flow chart can have multiple start points. It basically traces the propagation of events across multiple ambiguous points, where the event diverges probabilistically between pathways. Obviously, at any given point in the network, the probability of that node being visited is dependent on the joint probability of the preceding nodes. The Bayesian network is different from the Neural Network in that it is explicit reasoning, even though probabilistic and hence could have multiple stable states based on each step being revisited and modified within legal values, just like an algorithm. It is a robust way to reason probabilistically, but it involves encoding of probabilities, conjecturing the points where randomized actions can happen and hence need more heuristic effort to build.
Decision Trees: The Decision tree is again a network, which is more like a flow chart, which is closer to the Bayesian network than the neural net. Each node has more intelligence than the neural net and the branching can be decided by mathematical or probabilistic evaluations. The decisions are straightforward evaluations based on frequency distributions of likely events, where the decision is probabilistic. However, in Bayesian networks, the decision is based on the distribution of 'evidence' that points to an event having occurred, rather than the direct observation of the event itself.
An Example For instance, if we were to predict the movement of a man-eating tiger across some Himalayan villages that happens to be in the edge of some tiger reserve, we could model it on either approach as follows:
In a decision tree, we would rely on expert estimates whether a tiger would given the choice between open fields or rivers would choose on the latter. In a Bayesian network, We track the tiger by pug marks, but reason in a manner that acknowledges that these pug marks might have been those of some other similar sized tiger routinely patrolling its territory. If we are to use a neural net, we would have to train the model repeatedly using various behavioural peculiarities of the tiger in general, such as its preference to swim, preference of covered areas over open areas, its avoidance of human habitations in order to allow the network to generally reason over the course the tiger might take.
Excellent answer by @David Marx. I have been wondering, what is the difference between Classification/Regression tree and Bayesian network.One builds on entropy to classify an outcome into classes based on different predictors and the other builds a graphical network using conditional independence and probabilistic parameter estimates.
I feel that the methodology of building the Bayesian network is different compared to the Regression/Decision tree. The algorithm for structural learning, objectives for using the models as well as inferential ability of the models are different.
The score-based and constrained based approach can be understood with some parallels drawn with the information gain criteria in the decision tree families.
Regarding graphical models, Petri Net formalises a system behaviour; in that it sharply differs from the rest of the mentioned models, all of which relate to how a judgement is formed.
Worth to note that most of the cited names designate quite extensive AI concepts, which often coalesce: for example, you may use a Neural Network to build a decision tree, while the Neural Network itself, as an earlier post discussed, may depend on Bayesian inference.
Its a good question and I've been asking myself the same. There are more than two kinds of neural network, and it seems the previous answer addressed the competitive type, whereas the Bayesian network seems to have similarities to the feed-forward, back-propagation (FFBP) type, and not the competitive type. In fact, I would say the Bayesian network is a generalisation of the FFBP. So the FFBP is a type of Bayesian network and works in a similar fashion.