3
$\begingroup$

I am not very familiar with the Bernoulli distribution and am seeking to understand when to use it rather than other, more common binomial distributions in machine learning models.

I began by reading on Wikipedia, which gives the following definition:

The Bernoulli distribution is a special case of the two-point distribution, for which the two possible outcomes need not be 0 and 1. It is also a special case of the binomial distribution; the Bernoulli distribution is a binomial distribution where n=1.

It then proceeds to get more technical, but I don't see it clearly referring back to that statement later in the article. I'm just looking for a basic, high-level understanding but what confuses me about the definition above is that it says "where n = 1".

Which "n" is this referring to? It was not defined in the Wikipedia summary, but lacking another definition I usually take n to refer to sample size. Is that saying the Bernoulli distribution is for when you have a sample size of 1? That doesn't make any sense to me and seems to conflict with some empirical examples I've seen of the Bernoulli distribution being using in Generalized Boosting Models, thus I'm confused.

$\endgroup$
  • 2
    $\begingroup$ You know the Binomial has two parameters, $n$ and $p$, right? $\endgroup$ – Glen_b -Reinstate Monica Jun 29 '16 at 15:16
  • $\begingroup$ @Glen_b oh, that's right, it's one of those. It's been a long time since I learned about this back in school. The answers below brought it all back and answered the question very well. Thanks. $\endgroup$ – Hack-R Jun 29 '16 at 15:21
5
$\begingroup$

Bernoulli trials are experiments with binary outcomes: success or failure. One way to represent that is as 0 and 1. Or we could think about a coin toss: it's a head (H) or tails (T). The H/T representation is not numerical, but contains precisely the same amount of information about the experiment as the 0 and 1 encoding.

The choice of 0 and 1 is convenient because the mean of a set of 0 and 1-encoded data is the proportion of 1's. But one could conceivably use -1, 1 encoding, or 1, 2-encoding; in a somewhat obscure set of circumstances, these alternative encoding schemes might even be helpful.

The $n$ parameter is referring to the number of trials in a binomial distribution. The binomial distribution counts the number of experimental outcomes of one binary type (conventionally, it counts the successes). So if a sequence of coin tosses is recorded as HTTT, then we might say that there were 3 successes (if we define heads as successes; this is also a convention). We can parameterize the binomial distribution as a number of trials and a probability of success (when the trials are statistically independent, the probability is fixed at the same value for all trials, and the number of trials is finite). Clearly, if $n$ is 1, we have a Bernoulli distribution.

See also: Why 0 for failure and 1 for success in a Bernoulli distribution?

$\endgroup$
  • $\begingroup$ Thanks much, I will accept this as the answer. Having said that, could you help me understand why someone like this pareonline.net/getvn.asp?v=20&n=1 with data that has 445 binary trials would choose the Bernoulli distribution? Or am I misinterpreting that? The lalonde dataset they use has 445 observations on students. Those students were separated into 2 groups and a GBM was fitted with the Bernoulli distribution to estimate the prob of group assignment. Is that 445 trials or just 1? $\endgroup$ – Hack-R Jun 29 '16 at 15:24
  • $\begingroup$ My guess would be that it's because the authors are interested in making inferences about item difficulty and student cognition, under the assumption that not all items are equally difficult, nor are all students equally smart. But that's just based on skimming the first page. $\endgroup$ – Sycorax says Reinstate Monica Jun 29 '16 at 15:27
  • $\begingroup$ OK thanks, so basically they were not considering the number of trials as a factor in their choice then. It just confuses me because they have several hundred observations, so it seems like a logit distribution or something might be more appropriate. $\endgroup$ – Hack-R Jun 29 '16 at 15:29
  • $\begingroup$ @Hack-R I'd have to spend more time reading that paper to get a handle on it, and I don't have time at the moment. Perhaps this would be a good question it ask in its own right. $\endgroup$ – Sycorax says Reinstate Monica Jun 29 '16 at 15:30
5
$\begingroup$

The Binomial distribution models the number of "successes" $k$ in $n$ trials. Success can, of course, be defined in any suitable way, like the number of times your favorite animal (like Paul, the Octpus, pictured below) is going to correctly forecast the outcomes of $n$ different knock-out games (because there can be no ties) at the European championship (to take a current event as of the time of writing) given that it has a probability of $p$ of correctly forecasting any given game. (Many, but not all, people expect $p=0.5$ to be natural here...)

enter image description here

The Bernoulli then corresponds to the case in which just one trial has been made, like in the example of one coin being tossed - it either comes up head or not.

You are of course right that many samples can be taken from the Bernoulli (like from any other) distribution. For example, you may toss the coin many times.

$\endgroup$
2
$\begingroup$

Imagine you have a thumbtack with a $p$ probability of landing on its back when you flip it. (There are two possibilities: thumbtack lands on its back or on its side.)

enter image description here

  • Let $X$ denote the outcome of this flip of a thumbtack.
  • Let $X=1$ denote the outcome where the thumbtack lands on its back.
  • Let $X=0$ denote the outcome where the thumbtack lands on its side.

$X$ would a random variable following the Bernoulli distribution. The Bernoulli distribution is used to represent a binary outcome with $p \in [0, 1]$ probability of success.

Now let's imagine we flip the same thumbtack $n$ times, under the same conditions, with each flip denoted by $X_1$, $X_2$ etc.... Let $Y = \sum_{i=1}^n X_i$ be the number of successes, the number of times the thumbtack landed on its back. (We want the same conditions, same thumbtack, conditions such that $X_1$, $X_2$ etc... are independently and identically distributed with the same probability of success: the scalar value $p$.)

$Y$ would then follow the binomial distribution. Also observe that for $n=1$, for a single flip of a tack, $X_1$ (the result of a single flip) and $Y$ (the number of successful flips) are exactly the same thing.

$\endgroup$
  • $\begingroup$ For this to be a binomial distribution, the probability of back/side would have to be fixed for all trials, and the trials would have to be statistically independent. It seems plausible to assume these conditions in this scenario; however, it is not generally true, so I feel it is important to emphasize that point. $\endgroup$ – Sycorax says Reinstate Monica Jun 29 '16 at 15:29
  • $\begingroup$ @GeneralAbrial good point. I made a few edits. $\endgroup$ – Matthew Gunn Jun 29 '16 at 15:32
  • $\begingroup$ Note that the trials can be iid even if the probability of success is not fixed. Consider a beta-binomial distribution. These edits do not yet fully characterize the binomial distribution. $\endgroup$ – Sycorax says Reinstate Monica Jun 29 '16 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.