37

Suppose we have a lightbulb controlled by two switches. Let $S_1$ and $S_2$ denote the state of the switches, which can be either 0 or 1. Let $L$ denote the state of the lighbulb, which can be either 0 (off) or 1 (on). We set up the circuit such that the lighbulb is on when the two switches are in different states, and off when they're in the same state. So, ...


37

So if that's the case, does statistical independence automatically mean lack of causation? No, and here's a simple counter example with a multivariate normal, set.seed(100) n <- 1e6 a <- 0.2 b <- 0.1 c <- 0.5 z <- rnorm(n) x <- a*z + sqrt(1-a^2)*rnorm(n) y <- b*x - c*z + sqrt(1- b^2 - c^2 +2*a*b*c)*rnorm(n) cor(x, y) With ...


27

Improper scoring rules such as proportion classified correctly, sensitivity, and specificity are not only arbitrary (in choice of threshold) but are improper, i.e., they have the property that maximizing them leads to a bogus model, inaccurate predictions, and selecting the wrong features. It is good that they disagree with proper scoring (log-likelihood; ...


27

Wow, what a big question! The short version of the answer is that just because you can represent two models using diagrammatically similar visual representations, doesn't mean they are even remotely related structurally, functionally, or philosophically. I'm not familiar with FCM or NF, but I can speak to the other ones a bit. Bayesian Network In a ...


21

A probabilistic graphical model (PGM) is a graph formalism for compactly modeling joint probability distributions and (in)dependence relations over a set of random variables. A PGM is called a Bayesian network when the underlying graph is directed, and a Markov network/Markov random field when the underlying graph is undirected. Generally speaking, you use ...


17

First a few words about Markov Processes. There are four distinct flavours of that beast, depending on the state space (discrete/continuous) and time variable (discrete/ continuous). The general idea of any Markov Process is that "given the present, future is independent of the past". The simplest Markov Process, is discrete and finite space, and discrete ...


16

Why is the AUC for A better than B, when B "seems" to outperform A with respect to accuracy? Accuracy is computed at the threshold value of 0.5. While AUC is computed by adding all the "accuracies" computed for all the possible threshold values. ROC can be seen as an average (expected value) of those accuracies when are computed for all threshold values. ...


12

From my understanding, if a DAG G is said to be the I-Map of probability distribution P, then every independence we can observe from G is encoded in P. Let's consider a simple example: Suppose distribution $P_1$ has independence $\{(I\perp D)_p\}$, and distribution $P_2$ has no independence, or $\emptyset$. Now we define two DAGs: $G$ and $G'$ $G$ is I-Map ...


11

As far as I can tell, Bayesian Networks do not claim to be able to estimate causal effects in non-directed acyclic graphs, whereas SEM does. That's a generalization in favor of SEM... if you believe it. An example of this might be measuring cognitive decline among people where cognition is a latent effect estimated using a survey instrument like 3MSE, but ...


10

Yes, it's written as such and contains sample questions, for which you can request the answers here You might also want to have a look at Pattern Recognition and Machine Learning by Chris Bishop and Information Theory, Inference and Learning Algorithms by David MacKay, which can also be downloaded for free. Both of these cover some aspects of graphical ...


10

Yes, this is because the $\alpha, \beta$ node $d$-separates $m_2$ and $y_3$. See Probabilistic Reasoning in Intelligent Systems for an explanation of $d$-separation.


9

Your first derivation is correct! Because we haven't observed "Starts" or "Moves", "Ignition" is independent of "Gas". What you are writing here is just the factorisation of the joint distribution, not how to compute a the probability of a specific node given a set of observations. What the Markov Blanket says, is that all information about a random ...


9

It takes $3\times 2 \times 2 \times 3 = 36$ numbers to write down a probability distribution on all possible values of these variables. They are redundant, because they must sum to $1$. Therefore the number of (functionally independent) parameters is $35$. If you need more convincing (that was a rather hand-waving argument), read on. By definition, a ...


8

Many structure learning algorithms can only score competing structures up to their Markov equivalences and as a result it is impossible to learn a unique DAG for a Bayesian Network (BN) based solely on data, which makes the causality hypothesis questionable. Spirtes et al. term this issue as “statistical indistinguishability”, discussing it at length in ...


8

Bayesian neural nets are useful for solving problems in domains where data is scarce, as a way to prevent overfitting. They often beat all other methods in such situations. Example applications are molecular biology (for example this paper) and medical diagnosis (areas where data often come from costly and difficult expiremental work). Actually, Bayesian ...


7

As a whole the network represents the factorization of the multivariate joint distribution into a product of simpler factors: $p(c,r,s,w)=p(w|r,s)p(s|c)p(r|c)p(c)$ with $c \in C$ representing a particular state for cloudy variable (i.e. the variable cloudy can take on any state $c$ in the set $C$, $s \in S$ for the sprinkler variable and so on. I'm going ...


7

Since we are calculating the joint distribution, we'll assume that our initial sample is $ x = P(D=0,I=0,G=0,L=0,S=0) $ . To calculate the next sample, we'll need to sample each variable from the conditional distribution. $ P(D\mid G,I,S,L) $,from the conditional independencies in the Bayes net, simplifies to just sampling $ P(D)$. We sample and get the ...


7

Bayesian Networks (BN's) are generative models. Assume you have a set of inputs, $X$, and output $Y$. BN's allow you to learn the joint distribution $P(X,Y)$, as opposed to let's say logistic regression or Support Vector Machine, which model the conditional distribution $P(Y|X)$. Learning the joint probability distribution (generative model) of data is more ...


7

The remark is not referring to continuous-time--continuous-observation Kalman-Bucy filters, but to discrete-time Kalman filters. The confusion seems to be only due to the OP not knowing about the discrete-time version (which in my experience is most commonly meant when 'Kalman filter' is mentioned). See, for example, the Wikipedia article 'Kalman filter' or [...


6

Take a look at a post in Healthy Algorithm: http://healthyalgorithms.com/2011/11/23/causal-modeling-in-python-bayesian-networks-in-pymc/ also in PyMC's totorial: http://pymc-devs.github.io/pymc/tutorial.html Maybe you would try the following code clip (assuming you have imported pymc as mc): A = mc.Normal('A', mu_A, tau_A) B = mc.Normal('B', mu_B, tau_B) ...


6

I would prefer the book Graphical Models by Steffen L. Lauritzen, and his lecture at Oxford.


6

I spent a little while reading the first couple of chapters of Koller & Friedman, and I wasn't happy with it as an introductory text. On several occasions, the book gives a motivating example, but the example cannot be understood without background material later in the chapter. This kind of exposition works for me only if the example explicitly says ...


6

Is it not intuitive that you cannot reason from cause to unobserved effect to another cause? If the rain (B) and the sprinkler (D) are causes of the wet ground (C), then can you argue that seeing rain implies that the ground is probably wet, and continue to reason that the sprinkler must be on since the ground is wet?! Of course not. You argued that the ...


6

Why are you using table to compare the output? Using cbind to put the actual and predicted values side by side shows that the predictions are not the same as the actual, and you can compute standard accuracy metrics to quantify the degree to which they diverge. library(bnlearn) # Load the package in R library(forecast) data(gaussian....


6

Being Markov is a property of the distribution, not the graph (although it is only defined relative to a given graph). A graph can't be Markov or fail to be Markov, but a distribution can fail to be Markov relative to a given graph. Here is an example in terms of causal networks. Say we know that $X_1$ influences $X_2$ and $X_3$, but $X_2$ and $X_3$ don't ...


6

One advantage of the BNN over the NN is that you can automatically calculate an error associated with your predictions when dealing with data of unknown targets. With a BNN, we are now doing Bayesian inference. Let's define our BNN prediction as $\bar{f}(x′|x,t)=∫f(x′,ω)p(ω|x,t)dω$, where $f$ is the NN function, $x'$ are your inputs, $ω$ are the NN ...


6

A few things. The Chain Rule: $$ P(E, C | A) = P(E|C,A) \cdot P(C | A) $$ Conditional Independence: The notation of a Bayes Network implies $$ P(E|C,A) = P(E|C) $$ The idea is, if you know that $C$ did or did not happen, it doesn't matter whether or not $A$ happened. So at this point we have: $$ P(E,C|A) = P(E|C) \cdot P(C|A) $$ Marginalization: Removing ...


6

By definition of conditional probability* we have that: $$P(E=e|A=a)=\frac{P(E=e,A=a)}{P(A=a)}=\frac{\sum_{c}P(E=e,C=c,A=a)}{P(A=a)}$$ In the last step I used marginalization over $c$. Then, again using the definition of conditional probability, this is equal to: $$\sum_{c}P(E=e,C=c|A=a)$$. *Definition of conditional probability: $$P(x_1,...,x_n|y_1,...,...


5

$$\text{E}[X] = \frac{\int x \, P(y_i \mid x) \, dx}{\int P(y_i \mid x) \, dx}$$ is not a general statement, but only a first step in expectation propagation (EP). EP tries to approximate a posterior distribution $P(x \mid \mathcal{D})$ using a given factorization of the joint, $$P(x) \prod_i P(y_i \mid x).$$ To reduce clutter, the dependency on the data $...


5

I only draw a directed edge if I am happy to assume the relationship is causal. This assumption of course cannot be verified by observational data, but by formalizing a set of hypothesized causal relationships as a DAG, I can identify which variables to adjust for to make the best possible causal inferences about a given relationship in the graph. From my ...


Only top voted, non community-wiki answers of a minimum length are eligible