Bernoulli trial - why do we multiply probabilities?

We assume Bernoulli trial is a series of $n$ consecutive independent experiments - each can end with success (with probability $p$) or failure (probability $q$ or $1-p$). I know it's the probability of getting $k$-successes and $n-k$ failures in $n$ experiments. Why do we multiply probabilities of individual outcomes to obtain the probability that the outcomes were identical? If I knew this, Bernoulli formula would be obvious for me: $p^k(1-p)^{n-k}$.

The probability theory is a bit fuzzy for me, it's not just as undeniable theory like linear algebra (matrices etc.). In linear algebra, theorems follow from sticking to precise definitions and using basic logic, here in probability and statistics in general, we try to model some problems and assume the mathematical formulation of the problem is correct and appropriate - it might often turn out to be wrong, sooner or later, no matter how confident we are of our model initially.

• Probability theory is just as "undeniable" as linear algebra. Applied statistics is where the math meets the data, and it will virtually always be the case that the mathematical formulation is incorrect. "All models are wrong; some models are useful." - G.E.P. Box. Apr 20 '15 at 21:17
• The independence assumption means that probabilities multiply. This is nothing other than the definition.
– whuber
Apr 20 '15 at 21:44
• I thought independence is defined for events in the same probability space. Let's say we toss a coin twice. Then $\Omega = \lbrace HH, HT, TH, TT\rbrace$. Then $P(H and T) = P(Head) P(Tail)$. But here in Bernoulli trials, the sample space is just $\Omega = \lbrace 1, 0 \rbrace$ (success or failure). We need to create a new sample space containing sequences of $1$'s and $0$'s. Apparently we need to prove that $P(1)$ and $P(0)$ is equal in both probability spaces (remember, we have the old sample space with two elementary events, and the new one). Apr 20 '15 at 21:55
• You have it basically right, but there's nothing to prove: the independence assumption asserts that the probabilities multiply.
– whuber
Apr 20 '15 at 22:07

Why do we multiply probabilities?

You have been given the short answer in the comments: because of the definition of independence. We may define independence as follows:

If for two events $A$ and $B$ it holds that $P(A \cap B)=P(A)P(B)$, then we say that $A$ and $B$ are independent. We say that two random variables $X$ and $Y$ are independent if for all (Borel) sets $C\subseteq \mathbb R$ and $D \subseteq \mathbb R$, $P(X\in C \cap Y \in D)=P(X \in C)P(Y \in D)$.

Although it may seem that probability is less rigorous than for example linear algebra, this need not be so. You are correct in worrying about the fact that independence is only defined for events and / or random variables on the same probability space. Luckily, someone else has already proven (I haven't found a nice link yet) that if you have set of random variables in mind with a certain distribution, a probability space carrying these random variables exists.

Illustration

As an illustration, let's construct a probability space for the coin tossing example you have in the comments. For simplicity, take 2 flips; the generalization to any finite number is straight forward. We need a sample space, $\Omega$, a $\sigma-$algebra, $\mathcal F$, a probability measure on this $\sigma-$ algebra, $P$, and two independent Bernoulli random variables with parameter $p$, $X_1,X_2$.

Let $\Omega = \{HH, HT, TH, TT\}$ (NB. The names of the sample points are chosen to agree with our intuition about coin tossing, it has no impact on the model), let $\mathcal F$ be the family of all subsets of $\Omega$, and define $P$ as follows:

$$P(HH)=p^2,P(HT)=P(TH)=p(1-p),P(TT)=(1-p)^2,$$ with $P(E)=\sum_{\omega \in E}P(\omega)$ for general subsets $E \in \mathcal F$. Now let $X_1(\omega) = I_{\{HH , HT\}}(\omega)$ and $X_2(\omega) = I_{\{HH , TH\}}(\omega)$. We may check that $P(X_1 = 1, X_2 = 1)=P(HH) = p^2$ and $P(X_1=1)=P(HH)+P(HT)=p^2+p(1-p)=p$ and $P(X_2=1)=P(HH)+P(TH)=p$ so that, indeed, $X_1,X_2$ are independent Bernoulli variables on the same space. It is important to note that the sample space cannot be $\Omega = \{0,1\}$. You need to have more sample points in order to construct independent random variables.

• It's been a long time, but: we have a sample space $S=\{H,T\}$ and the set of events (sigma algebra) defined on that sample space $\{\emptyset, \{H\},\{T\},\{HT\}\}$. In this case we can't say $H$ and $T$ are independent, because. there is no outcome in $S$ that would mean both $H$ and $T$ happened, and we need such an outcome to talk about the event of both happening $P(H\cap T)=P(H)P(T)$. Dec 11 '15 at 8:14
• So the independence as defined above works for the 2nd case described here: cut-the-knot.org/Probability/IndependentExperiments.shtml But by independent experiments we mean the 1st description (independent experiments) in that link. That article says it has combinatorial justification, ok. But what if the probability distribution isn't discrete? If we have continuous sample space, then there is no justification in terms of combinatorics, because the number of outcomes is uncountably infinite. That's just weird. Dec 11 '15 at 8:37
• The justification is measure theory. The combinatorics motivation is just a very limited special case. Dec 11 '15 at 11:26
• Ok. Is it right to state that every time I flip a coin, the strings of the form $HHTH...$ become longer by 1, that is a new sample space is created? After 2 flips $\Omega=\{HH, HT, TT, TH\}$, after 3 we have $\Omega=\{HHH, HHT, HTH, THH, TTH, THT, HTT, TTT\}$? Dec 11 '15 at 16:19

We multiply, because for independent events,say,$E_1,E_2,...,E_n$, $P( E_1\cap E_2\cap ... \cap E_n)=P(E_1)\times P(E_2)\times ... \times P(E_n)$

The formula will be $\binom{n}{k}p^k(1-p)^{n-k}$

$\binom{n}{k}$ is multiplied, because number of occurrence of k success may takes place differently in n events.(i.e. n choose k)

From @jbowman's comment,"All models are wrong; some models are useful." - G.E.P. Box.

Just on the confusion about "same probability space" - The two events to consider upon the right hand side are both in the same space \Omega = {0,1}, as is the Boolean expression on the left hand side.

• I find this hard to follow, would it be possible to clarify a little bit? In which example is $\Omega=\{0,1\}$? and what Boolean expression do you mean? Apr 20 '15 at 23:45
• I see, you are talking about sampling from a set. But $\{0,1\}$ can never be the sample space in a probabilistic model with two independent Bernoulli variables, there are just not enough outcomes. What you call the "joint space" is the only possible of those two. So, what I mean is, the events the OP needs to consider are not in $\{0,1\}$ Apr 21 '15 at 2:26
• Thanks for the correction. I may be a bit confused here. Call the sample space {H, T}. I was thinking H = False, T= True, for instance then "H and T" looks like a boolean - instead it is really the intersection of the events X_0 and X_1. Yes the subtlety is that the repeated throws sample space is {H, T}^n. Apr 21 '15 at 2:42

First, let's start with general probability theory. If you have two disjoint events, say $A$ and not-$A$, then the probability of either of those two occurring is a sum

$$P(A \cup A ^ \complement) = P(A) + P(A ^ \complement)$$

What is the probability of flipping a head or a tail? Those events are are "parts" of the same "whole" probability, so you can sum them. It gets more complicated with independent events. If you flip a coin and throw a dice at the same time, what is the probability of getting a head and roll six pips at the same time? Think of it: there are two possible outcomes of a coin and six possible outcomes of a dice, so there are $2 \times 6$ possibilities and if flipping a head has a probability of $1/2$ and rolling six pips has a probability of $1/6$, then the probability of getting both is simply $1/2 \times 1/6$, or

$$P(A \cap B) = P(A) \times P(B)$$

Now, Bernoulli distribution tells you what is the probability of a certain event $x \in \{0, 1\}$, such as single coin flip, from this distribution described by parameter $p$, where probability of flipping a head is $P(X = 1) = p$ and probability of getting a tail is $P(X = 0) = 1 - p$. The probability mass function can be described as

$$f(x; p) = p^x (1 - p) ^ {(1 - x)}$$

now notice that $x$ can be either $1$ or $0$, so if you put those values into the formula you get the same probabilities for $x = 0$ or $x = 1$.

From your question it seems however that you are talking about Binomial distribution, where there is $n$ "Bernoulli trials", i.e. $n$ coin flips, and so you can get $k$ successes (e.g. heads). The probability mass function is

$$f(k; n, p) = {n \choose k} p^k (1 - p) ^ {n - k}$$

You can read it as follows: probability of a single success in Bernoulli trial is $p$ so for $k$ independent successes it is $p \times p \times ...$ repeated $k$ times or simply $p ^ k$. Probability of failure is $1-p$ and it can occur only $n-k$ times since there is $n$ flips and $k$ is already "occupied" by successes, so $(1-p)^{(n-k)}$. Now the scary part of $n \choose k$ - it is simply a number of possible combinations on how you can draw $k$ elements out of $n$. So while there is $n \choose k$ possible ways on how you can flip heads $k$ times out of $n$ flips with probability $p$ and get $n-k$ times tails with $1-p$ probability in the same series of flips. If you think of it it makes perfect sens.