Is a Bayesian network doing feature selection?

Question

I have just started playing around and reading about Bayes Nets. Here is a snippet of code using the bnlearn package in R, which seems to be a fantastic tool.

install.packages("bnlearn")
library(bnlearn)

data <- data.frame(matrix(c("sunny","hot","high","weak","no",
                 "sunny","hot","high","strong","no",
                 "overcast","hot","high","weak","yes",
                 "rain","mild","high","weak","yes",
                 "rain","cool","normal","weak","yes",
                 "rain","cool","normal","strong","no",
                 "overcast","cool","normal","strong","yes",
                 "sunny","mild","high","weak","no",
                 "sunny","cool","normal","weak","yes",
                 "rain","mild","normal","weak","yes",
                 "sunny","mild","normal","strong","yes",
                 "overcast","mild","high","strong","yes",
                 "overcast","hot","normal","weak","yes",
                 "rain","mild","high","strong","no"), byrow = TRUE,
               dimnames = list(day = c(),
                 condition = c("outlook","temperature",
                   "humidity","wind","playtennis")), nrow=14, ncol=5))

res = hc(data)
plot(res)

This graph that is fit using the hill climbing algorithm has two nodes that are not directed: Outlook and wind. Is it fair to say that these random variables are unrelated to the others and a sort of feature selection is occurring?

Answer 1

Usually it means models with directed edges on these two nodes are industinguishable, given the training data. So the search algorithm leaves them as undirected. I don't think we can say the variables are unrelated based on the result of the learning algorithm. To determine if the nodes were truly unneeded, build the model, set the edge directions manually, and test against a network without them. Does the performance improve?

Answer 2

In Bayesian Belief Network (BBN) structure learning, you are trying to learn the directed acyclic graph (DAG). If you learn a partially directed acyclic graph (PDAG), or if a PDAG is the output of your structure learning algorithm, then you need to orient the undirected arcs in some way (e.g. manually) so as to form a DAG.

When you see an undirected arc between two nodes, X1 and X2, it means there is a dependency (not as you stated, which is that they are unrelated), however, whatever test (statistical or not) which have been applied to orient that undirected arc has failed to distinguish between

• X1 -> X2
• X1 <- X2

In general, learning a BBN DAG is a type of feature selection, and many feature selection algorithms have referenced BBN structure learning papers. When you learn the DAG of a BBN, with respect to each node, you are learning its Markov blanket (directly or indirectly); those variables in the Markov blanket are the "features" relevant to the variable of concern.