Bernoulli trial - why do we multiply probabilities?

Question

We assume Bernoulli trial is a series of $n$ consecutive independent experiments - each can end with success (with probability $p$) or failure (probability $q$ or $1-p$). I know it's the probability of getting $k$-successes and $n-k$ failures in $n$ experiments. Why do we multiply probabilities of individual outcomes to obtain the probability that the outcomes were identical? If I knew this, Bernoulli formula would be obvious for me: $p^k(1-p)^{n-k}$.

The probability theory is a bit fuzzy for me, it's not just as undeniable theory like linear algebra (matrices etc.). In linear algebra, theorems follow from sticking to precise definitions and using basic logic, here in probability and statistics in general, we try to model some problems and assume the mathematical formulation of the problem is correct and appropriate - it might often turn out to be wrong, sooner or later, no matter how confident we are of our model initially.

Answer 1

This answer is an attempt to address both the question in the headline and the concerns in the comments.

---

Why do we multiply probabilities?

You have been given the short answer in the comments: because of the definition of independence. We may define independence as follows:

If for two events $A$ and $B$ it holds that $P(A \cap B)=P(A)P(B)$, then we say that $A$ and $B$ are independent. We say that two random variables $X$ and $Y$ are independent if for all (Borel) sets $C\subseteq \mathbb R$ and $D \subseteq \mathbb R$, $P(X\in C \cap Y \in D)=P(X \in C)P(Y \in D)$.

What about rigor?

Although it may seem that probability is less rigorous than for example linear algebra, this need not be so. You are correct in worrying about the fact that independence is only defined for events and / or random variables on the same probability space. Luckily, someone else has already proven (I haven't found a nice link yet) that if you have set of random variables in mind with a certain distribution, a probability space carrying these random variables exists.

Illustration

As an illustration, let's construct a probability space for the coin tossing example you have in the comments. For simplicity, take 2 flips; the generalization to any finite number is straight forward. We need a sample space, $\Omega$, a $\sigma-$algebra, $\mathcal F$, a probability measure on this $\sigma-$ algebra, $P$, and two independent Bernoulli random variables with parameter $p$, $X_1,X_2$.

Let $\Omega = \{HH, HT, TH, TT\}$ (NB. The names of the sample points are chosen to agree with our intuition about coin tossing, it has no impact on the model), let $\mathcal F$ be the family of all subsets of $\Omega$, and define $P$ as follows:

$$ P(HH)=p^2,P(HT)=P(TH)=p(1-p),P(TT)=(1-p)^2, $$ with $P(E)=\sum_{\omega \in E}P(\omega)$ for general subsets $E \in \mathcal F$. Now let $X_1(\omega) = I_{\{HH , HT\}}(\omega) $ and $X_2(\omega) = I_{\{HH , TH\}}(\omega)$. We may check that $P(X_1 = 1, X_2 = 1)=P(HH) = p^2$ and $P(X_1=1)=P(HH)+P(HT)=p^2+p(1-p)=p$ and $P(X_2=1)=P(HH)+P(TH)=p$ so that, indeed, $X_1,X_2$ are independent Bernoulli variables on the same space. It is important to note that the sample space cannot be $\Omega = \{0,1\}$. You need to have more sample points in order to construct independent random variables.

Answer 2

We multiply, because for independent events,say,$E_1,E_2,...,E_n$, $P( E_1\cap E_2\cap ... \cap E_n)=P(E_1)\times P(E_2)\times ... \times P(E_n) $

The formula will be $\binom{n}{k}p^k(1-p)^{n-k}$

$\binom{n}{k}$ is multiplied, because number of occurrence of k success may takes place differently in n events.(i.e. n choose k)

From @jbowman's comment,"All models are wrong; some models are useful." - G.E.P. Box.

Answer 3

Just on the confusion about "same probability space" - The two events to consider upon the right hand side are both in the same space \Omega = {0,1}, as is the Boolean expression on the left hand side.

Answer 4

First, let's start with general probability theory. If you have two disjoint events, say $A$ and not-$A$, then the probability of either of those two occurring is a sum

$$ P(A \cup A ^ \complement) = P(A) + P(A ^ \complement) $$

What is the probability of flipping a head or a tail? Those events are are "parts" of the same "whole" probability, so you can sum them. It gets more complicated with independent events. If you flip a coin and throw a dice at the same time, what is the probability of getting a head and roll six pips at the same time? Think of it: there are two possible outcomes of a coin and six possible outcomes of a dice, so there are $2 \times 6$ possibilities and if flipping a head has a probability of $1/2$ and rolling six pips has a probability of $1/6$, then the probability of getting both is simply $1/2 \times 1/6$, or

$$ P(A \cap B) = P(A) \times P(B) $$

Now, Bernoulli distribution tells you what is the probability of a certain event $x \in \{0, 1\}$, such as single coin flip, from this distribution described by parameter $p$, where probability of flipping a head is $P(X = 1) = p$ and probability of getting a tail is $P(X = 0) = 1 - p$. The probability mass function can be described as

$$ f(x; p) = p^x (1 - p) ^ {(1 - x)} $$

now notice that $x$ can be either $1$ or $0$, so if you put those values into the formula you get the same probabilities for $x = 0$ or $x = 1$.

From your question it seems however that you are talking about Binomial distribution, where there is $n$ "Bernoulli trials", i.e. $n$ coin flips, and so you can get $k$ successes (e.g. heads). The probability mass function is

$$ f(k; n, p) = {n \choose k} p^k (1 - p) ^ {n - k} $$

You can read it as follows: probability of a single success in Bernoulli trial is $p$ so for $k$ independent successes it is $p \times p \times ...$ repeated $k$ times or simply $p ^ k$. Probability of failure is $1-p$ and it can occur only $n-k$ times since there is $n$ flips and $k$ is already "occupied" by successes, so $(1-p)^{(n-k)}$. Now the scary part of $n \choose k$ - it is simply a number of possible combinations on how you can draw $k$ elements out of $n$. So while there is $n \choose k$ possible ways on how you can flip heads $k$ times out of $n$ flips with probability $p$ and get $n-k$ times tails with $1-p$ probability in the same series of flips. If you think of it it makes perfect sens.