# Creating conditional probability distribution

I have a system which visits two locations and collects signals every 10 seconds. It is well known that A causes B to occur but not all the time. So, I am trying to find a conditional probability distribution of B given A.

The data looks like following:

A    B
----------
1    0
1    1
1    0
1    0
..........


there are about 10,000 rows

How do I use the above to create a conditional distribution: P(B|A) i.e. probability of B hapenning given A.

@Coronoe: Yes I mean the uncertainty in Arthurs calculation, how do I achieve that. Do we fit a 4 case binomial distribution here? The 4 cases are mentioned by Arthur.

How does this change is I have instead:

A    B    C
------------
1    0    1
1    1    0
1    0    1
1    0    1
............


Now I want to understand P(B,C|A). @Corone I really like the way you have described the math and it just intrigued my interest as to what happens now? Again I am pretty much of novice in this field.

So, my understanding now is that I would have 8 cases here like the following: P(B=1,C=1|A=1), P(B=1,C=1|A=0), P(B=1,C=0|A=1), etc and I modify the equation of frequentist approach to consider above 8 cases? But does that take care of any correlation between B and C as well i.e how B effects the happening of C in addition to A? Or should I start to think in terms of Dirichlet distribution here since it is a joint probability distribution.

• What is wrong with simpky counting up the occurrences in a frequency table? Are your data independent samples? Do you have any reason to fear selection biases? Is each visit independent or is there strong autocorrelation here? – Corone Feb 3 '13 at 22:26
• Each visit is independent but B depends on A due to way logic is built. – user1243255 Feb 3 '13 at 22:30
• So neither A nor B "stick" and hang around for a while? Your example shows a sequential string of A's - just coincidence? – Corone Feb 3 '13 at 22:51
• We do not know/care cause our signal gatherer just hits those locations every 10 seconds and sees whether location is alive or not and moves on but we do know that B is caused by A just by design of logic. – user1243255 Feb 3 '13 at 22:53
• Be careful what you do with the results then. For example, if it were the case that after B became 1, it hung around at 1 for 30 seconds, then the probability you get from a naive method will very biased if interpreted as the "probability that any randomly chosen location with A=1 is currently in B=1 at any given time". However, if your intention is only to apply this result to exactly the same data collection algorithm in the future, then it should be OK. – Corone Feb 3 '13 at 23:08

## 2 Answers

Picking up where Arther left off, assuming independence between samples, the completely saturated (most general) model for this type of data is to assign a different probability to each possible outcome $p_{00}, p_{10}, p_{01}$ and $p_{11}$ as defined by Arthur. Notice that the only restriction on what these values may take is

$p_{00}+p_{01}+p_{10}+p_{11}=1$

These probability could be estimated from your data simply by counting up the frequencies. Because of the constraint imposed, and the nature of the problem, the set of 4 probability estimates would be drawn from a Dirichlet distribution.

Now, in your case you are only interested in the two conditional probabilities $p_{1|1} = P(B|A=1)$ and $p_{1|0}=P(B|A=0)$. We can write down the likelihood function for these, given your data, as follows:

$P(p_{1|1} | \{n_{ij}\}) \propto p_{1|1}^{n_{11}} (1-p_{1|1})^{n_{10}}$

If we differentiate this with respect to $p_{1|1}$ then we find

$\frac{\partial L}{\partial p_{1|1}} = n_{11} p_{1|1}^{n_{11}-1} (1-p_{1|1})^{n_{10}} - n_{10} p_{1|1}^{n_{11}} (1-p_{1|1})^{n_{10}-1} = 0$

$n_{11} (1-p_{1|1}) = n_{10} p_{1|1}$

$p_{1|1} = \frac{n_{11}}{n_{11} + n_{10}}$

Which is the maximum likelihood expression given by Arthur

Assuming a non-informative prior then notice that the expression for $P(p_{1|1} | \{n_{ij}\})$ is a beta distribution (see http://en.wikipedia.org/wiki/Beta_distribution) with $\alpha = n_{11}+1$ and $\beta=n_{10}+1$

The mode of the beta distribution is the expression as given by Arthur, but you could put error bounds (strictly a credible interval) on this using percentiles of the beta distribution. The beta CDF is a not algebraically friendly, but R will calculate the credible interval like so:

c(qbeta(0.025, n11+1, n10+1), qbeta(0.975, n11+1, n10+1))


This also suggests that you could just as validly use the mean or median of the Beta distribution as your estimate, and in some situations that might be better.

Frequentist approach

The above is a little bit Bayesian, and so there is another approach.

The $n_{11}$ value is the number of time B=1 when A=1, while the $n_{10}$ is the number of time B=0 when A=1. Assuming independence, then the number of B=1 events, given A=1, should follow a binomial distribution with $n=n_{11}+n_{10}$ and $p=p_{1|1}$. We therefore need to estimate the $p$ parameter from the binomial distribution.

Using method of moments, we see that

$E[k] = np = (n_{11}+n_{10}) p_{1|1}$

$n_{11} = (n_{11}+n_{10}) p_{1|1}$

$p_{1|1} = \frac{n_{11}}{n_{11}+n_{10}}$

Recovering the same expression as before.

We could now construct a 95% confidence bound for $p_{1|1}$ by finding values of p. Since you have 10,000 rows, we can probably get away with the Normal approximation here, and so our symmetric approximate error bounds are

$\pm 1.96 \sqrt{\frac{p_{1|1}(1-p_{1|1})}{n_{01}+n_{11}}} = \pm 1.96 \sqrt{\frac{n_{11}n_{01}}{(n_{11}+n_{01})^3}}$

• What does 'non-informative prior' prior mean? Does it mean A doesn't cause B? – user1243255 Feb 4 '13 at 13:49
• Oh crikey... I dropped myself in it there :-) Non-informative prior has had tons written about it and what it "means". In this case I simply mean that our prior belief for the value $p_{1|1}$ is that it is equally likely to be any number between 0 and 1. It simply allows me to treat my likelihood function as a posteria probability distribution. – Corone Feb 4 '13 at 14:09
• "Because of the constraint imposed, and the nature of the problem, the set of 4 probability estimates would be drawn from a Dirichlet distribution." Where does the above statement fit in your math above? Also, another probably novice question: what does variance or confidence interval around probability mean. I thought if I have mean of something I would like to know the 95% confidence interval for mean to be bounded. Thanks for your time. – user1243255 Feb 4 '13 at 15:48
• There I was just discussing the general case, where you want the full joint distribution - not just the conditional distribution - i.e. $P(A,B)$. Beta distribution is a special case of Dirichlet. The probability that is estimated is only an estimate, and so we have some uncertainty about it. The confidence interval and credibility interval are two ways of characterising the plausible range of values that the parameter might take according to the data. – Corone Feb 4 '13 at 16:07
• thanks. You intrigued me through your explanation so I added a follow up question above. Or should I start a new thread? – user1243255 Feb 4 '13 at 16:38

For $i,j \in \{0,1\}$, let $n_{i,j}$ be the count of the number of rows in which $A=i$ and $B=j$. Define $p_{i,j}$ by

$p_{i,j}= \frac{n_{i,j}}{n_{i,0}+n_{i,1}} .$

Then $p_{i,j}$ is the maximum likelihood estimator for $P(B =j \mid A=i)$.

• Is there a distribution for above probability calculation. If not then why not. As you can see I am a novice in this subject, so please be patient with me. – user1243255 Feb 3 '13 at 22:55
• Also how is using logistic regression to get P(B|A) different from your method above? – user1243255 Feb 3 '13 at 23:00
• This is a very special minimalist case of logistic regression. You only have 1 explanatory variable and is is a 2 level factor, so the logistic is not needed. The logistic bit comes in useful for numerical explanatory variables that take more than 2 values. – Corone Feb 3 '13 at 23:11
• @Corone, thanks that makes sense but what about creating a probability distribution? – user1243255 Feb 3 '13 at 23:14
• What exactly do you mean? The probability distribution of (A,B) is simply a 2 bit Bernoulli distribution, parametrized by the four probabilities $p_{i,j}$ Arthur wrote. If you mean the probability distribution of the estimate - i.e. the uncertainty in Arthurs calculation, then you need to go back and edit the question - that is too much for the comments section. – Corone Feb 3 '13 at 23:19