How to specify a prior on the network structure while learning a Bayesian network structure? I don't have experience with Bayesian networks. Bayesian network learning libraries like BANJO and bnlearn can learn the structure and fit the parameters of Bayesian networks on data. I see that there are various options for the search algorithm (annealing etc.) and for scoring (Gaussian priors on the parameters, lossfunctions for categorical data etc.), but I don't understand how to specify priors/regularization on the network structure. For example, I want to impose a cost on an arc between variables A and B because expert knowledge suggests such a causality is unlikely. Or, for example, I want to constrain that the parents of variable X can only be from the set {A,B,D,F}.
Can these kind of costs/constraints be implemented? I could write a function that computes a cost based on the network structure alone, and I would want this cost to be included to the data fit whenever a candidate network is stored during learning.
Update: I was able to mine the code for bnlearn and there is an option of providing the prior probabilities for individual arcs using the Castelo score formulation. For example,
## BDe with a prior.
beta = data.frame(from = c("A", "D"), to = c("B", "F"),
         prob = c(0.2, 0.5), stringsAsFactors = FALSE)
score(res, learning.test, type = "bde", prior = "cs", beta = beta)

Not sure of the exact probabilities. Does this mean A -> B has probability 0.2 and D -> F is 0.5? What about other arcs? i.e., is A -> B penalized or rewarded compared to an arc C -> B?
The example is given at https://github.com/cran/bnlearn/blob/0e4d3af6ad579b79bc9959e77385ec6b825ca6fc/man/score.Rd#L144
 A: “For example, I want to impose a cost on an arc between variables A and B because expert knowledge suggests such a causality is unlikely.”
This can be done by giving it a low prior probability. Further, if there is a known direction of effect, you could deselect the other direction.
“Or, for example, I want to constrain that the parents of variable X can only be from the set {A,B,D,F}.”
This can be done by disallowing all other parents of that random variable (vertex).
The two paragraphs below are from
https://cran.r-project.org/web/packages/bnlearn/bnlearn.pdf
“cs (the Castelo & Siebes prior, puts an independent prior probability on each arc and direction). If prior is cs, beta is a data frame with columns from, to and prob specifying the prior probability for a set of arcs. A uniform probability distribution is assumed for the remaining arcs.”
“When using the Castelo & Siebes prior in structure learning, the prior probabilities associated with an arc are bound away from zero and one by shrinking them towards the uniform distribution as per Hausser and Strimmer (2009) with a lambda equal to 3 * sqrt(.Machine$double.eps). This dramatically improves structure learning, which is less likely to get stuck when starting from an empty graph. As an alternative to prior probabilities, a blacklist can be used to prevent arcs from being included in the network, and a whitelist can be used to force the inclusion of particular arcs.”
So in your example you are correct that the prior probabilities are 0.2 for A -> B and 0.5 for D -> F. The other arcs are uniform.
If the probability A -> B is specified by the user as 0.2 then it is penalized compared to an arc C -> B which the user does not provided any prior knowledge. This is because if there is not any prior information given regarding an arc (in any direction and not saying that there is definitely not a link) then $pr(C \rightarrow B | \zeta) = pr(B \rightarrow C | \zeta) = pr(C \cdots B | \zeta) = 1 / 3 $.
The last probability (three dots) is that of no arc between C and B. More information is given in  Priors on Network Structures: Biasing the Search for Bayesian Networks by R. Castelo, Arno Siebes.
Some functions of bnlearn, including “score”, have a debug argument, setting this can help understand the selection process. Other learning algorithms are listed in the “constraint-based algorithms” section of the manual.
