While most books and papers on Probability Graphical Models (PGMs) describe a nice representational method for Bayesian Networks BNs (and/or Dynamic Bayesian Networks DBNs), where a Joint Probability Distribution/Table (JPD)/(JPT) computation is factored into individual Conditional Probability Distributions/Tables (CPDs)/(CPTs), they almost take it for granted to elaborate on the methods for learning these CPDs/CPTs to effectively implement these techniques.

Example: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1963499/ specifies "Classification/regression models can be used to learn the parameters for each node in the network."

The way I understand so far CPDs/CPTs can either come from domain knowledge of the system or from the data about the system. What I do wonder is how can CPDs/CPTs be learned from data? Such as:

  1. What if the data does not have all states equally represented or have missing values for certain states?
  2. What if the data is extremely sparse when there are more states possible which have been expressed very rarely or not at all?
  3. Example, let's say we want to find P(Vote|State) and have a dataset with multiple features. In this case would counts of known Votes in known States be sufficient for specifying a CPT or are there specific methods to calculate this with greater accuracy?

Elaborate literature references are welcome.


1 Answer 1

  1. If reasonable, one can make the assumption that the data is missing at random or missing completely at random. If so, then for example Expectation Maximization or Variational Bayes can be used to maximize the likelihood and assess parameters.

  2. There is great collection of literature on Mike Tipping's website on the topic of sparse Bayesian modeling and the "relevance vector machine" principle.

  3. I think a good reference for your question is David Barber's book on Bayesian Reasoning. It has lots of examples that might help.

  • $\begingroup$ Thanks for the answer, trying to understand this comment "If so, then for example Expectation Maximization or Variational Bayes can be used to maximize the likelihood and assess parameters." What is your interpretation of a parameter? Is it a multivariate probability distribution? If yes how does it connect to CPD? Also what utility function you are maximizing the likelihood for? $\endgroup$
    – D.S.
    Dec 28, 2016 at 0:33
  • $\begingroup$ Could you also elaborate on what concepts in sparse Bayesian modeling or "relevance vector principle" relate to this situation when you have data on hand and you are trying to compute a CPD? or what chapters in the Bayesian Reasoning book relate to the question? $\endgroup$
    – D.S.
    Dec 28, 2016 at 0:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.