Revisions to What is the intuition behind beta distribution?

added 1 character in body

Source Link

edited Feb 23, 2023 at 4:36

10.3k
12
23
40

enter image description here

Let's assume a seller on some e-commerce web-site receives 500 ratings of which 400 are good and 100 are bad.

We think of this as the result of a Bernoulli experiment of length 500 which led to 400 successes (1 = good) while the underlying probability $p$ is unknown.

The naive quality in terms of ratings of the seller is 80% because 0.8 = 400 / 500. But the "true" quality in terms of ratings we don't know.

Theoretically also a seller with "true" quality of $p=77\%$ might have ended up with 400 good of 500 ratings.

The pointy bar plot in the picture represents the frequency of how often it happend in a simulation that for a given assumed "true" $p$ 400 of 500 ratings were good. The bar plot is the density of the histogram of the result of the simulation.

And as you can see - the density curve of the beta distribution for $\alpha=400+1$ and $\beta=100+1$ (orange) tightly surrounds the bar chart (the density of the histogram for the simulation).

So the beta distribution essentially defines the probability that a Bernoulli experiment's success probability is $p$ given the outcome of the experiment.

library(ggplot2)
 
# 90% positive of 10 ratings
o1 <- 9
o0 <- 1
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim1 <- data.frame(p=rep(0:M/M,v))
df_beta1 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
# 80% positive of 500 ratings
o1 <- 400
o0 <- 100
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim2 <- data.frame(p=rep(0:M/M,v))
df_beta2 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
ggplot(data=df_sim1,aes(p)) +
    scale_x_continuous(breaks=0:10/10) +
 
    geom_histogram(aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta1 ,aes(p,y),colour=I("red"),size=2,alpha=.5) +
 
    geom_histogram(data=df_sim2, aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta2,aes(p,y),colour=I("orange"),size=2,alpha=.5)

http://www.joyofdata.de/blog/an-intuitive-interpretation-of-the-beta-distribution/https://www.joyofdata.de/blog/an-intuitive-interpretation-of-the-beta-distribution/

enter image description here

Let's assume a seller on some e-commerce web-site receives 500 ratings of which 400 are good and 100 are bad.

We think of this as the result of a Bernoulli experiment of length 500 which led to 400 successes (1 = good) while the underlying probability $p$ is unknown.

The naive quality in terms of ratings of the seller is 80% because 0.8 = 400 / 500. But the "true" quality in terms of ratings we don't know.

Theoretically also a seller with "true" quality of $p=77\%$ might have ended up with 400 good of 500 ratings.

The pointy bar plot in the picture represents the frequency of how often it happend in a simulation that for a given assumed "true" $p$ 400 of 500 ratings were good. The bar plot is the density of the histogram of the result of the simulation.

And as you can see - the density curve of the beta distribution for $\alpha=400+1$ and $\beta=100+1$ (orange) tightly surrounds the bar chart (the density of the histogram for the simulation).

So the beta distribution essentially defines the probability that a Bernoulli experiment's success probability is $p$ given the outcome of the experiment.

library(ggplot2)
 
# 90% positive of 10 ratings
o1 <- 9
o0 <- 1
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim1 <- data.frame(p=rep(0:M/M,v))
df_beta1 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
# 80% positive of 500 ratings
o1 <- 400
o0 <- 100
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim2 <- data.frame(p=rep(0:M/M,v))
df_beta2 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
ggplot(data=df_sim1,aes(p)) +
    scale_x_continuous(breaks=0:10/10) +
 
    geom_histogram(aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta1 ,aes(p,y),colour=I("red"),size=2,alpha=.5) +
 
    geom_histogram(data=df_sim2, aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta2,aes(p,y),colour=I("orange"),size=2,alpha=.5)

http://www.joyofdata.de/blog/an-intuitive-interpretation-of-the-beta-distribution/

enter image description here

Let's assume a seller on some e-commerce web-site receives 500 ratings of which 400 are good and 100 are bad.

We think of this as the result of a Bernoulli experiment of length 500 which led to 400 successes (1 = good) while the underlying probability $p$ is unknown.

The naive quality in terms of ratings of the seller is 80% because 0.8 = 400 / 500. But the "true" quality in terms of ratings we don't know.

Theoretically also a seller with "true" quality of $p=77\%$ might have ended up with 400 good of 500 ratings.

The pointy bar plot in the picture represents the frequency of how often it happend in a simulation that for a given assumed "true" $p$ 400 of 500 ratings were good. The bar plot is the density of the histogram of the result of the simulation.

And as you can see - the density curve of the beta distribution for $\alpha=400+1$ and $\beta=100+1$ (orange) tightly surrounds the bar chart (the density of the histogram for the simulation).

So the beta distribution essentially defines the probability that a Bernoulli experiment's success probability is $p$ given the outcome of the experiment.

library(ggplot2)
 
# 90% positive of 10 ratings
o1 <- 9
o0 <- 1
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim1 <- data.frame(p=rep(0:M/M,v))
df_beta1 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
# 80% positive of 500 ratings
o1 <- 400
o0 <- 100
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim2 <- data.frame(p=rep(0:M/M,v))
df_beta2 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
ggplot(data=df_sim1,aes(p)) +
    scale_x_continuous(breaks=0:10/10) +
 
    geom_histogram(aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta1 ,aes(p,y),colour=I("red"),size=2,alpha=.5) +
 
    geom_histogram(data=df_sim2, aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta2,aes(p,y),colour=I("orange"),size=2,alpha=.5)

https://www.joyofdata.de/blog/an-intuitive-interpretation-of-the-beta-distribution/

trying to clarify what the bar chart represents

Source Link

edited Nov 15, 2013 at 21:37

Raffael

1.7k
3
20
30

The pointy histogrambar plot in the picture represents the frequency of how often it happend in a simulation that for a given assumed "true" $p$ 400 of 500 ratings were good. The bar plot is the density of the histogram of the result of the simulation.

And as you can see - the density curve of the beta distribution for $\alpha=400+1$ and $\beta=100+1$ (orange) tightly surrounds the histogrambar chart (which is resized to a density scalethe density of the histogram for the simulation).

library(ggplot2)
 
# 90% positive of 10 ratings
o1 <- 9
o0 <- 1
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim1 <- data.frame(p=rep(0:M/M,v))
df_beta1 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
# 80% positive of 500 ratings
o1 <- 400
o0 <- 100
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim2 <- data.frame(p=rep(0:M/M,v))
df_beta2 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
ggplot(data=df_sim1,aes(p)) +
    scale_x_continuous(breaks=0:10/10) +
 
    geom_histogram(aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta1 ,aes(p,y),colour=I("red"),size=2,alpha=.5) +
 
    geom_histogram(data=df_sim2, aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta2,aes(p,y),colour=I("orange"),size=2,alpha=.5)
- See more at: http://www.joyofdata.de/blog/an-intuitive-interpretation-of-the-beta-distribution/#more-1758

The pointy histogram in the picture represents the frequency of how often it happend in a simulation that for a given assumed "true" $p$ 400 of 500 ratings were good.

And as you can see - the density curve of the beta distribution for $\alpha=400+1$ and $\beta=100+1$ (orange) tightly surrounds the histogram (which is resized to a density scale).

library(ggplot2)
 
# 90% positive of 10 ratings
o1 <- 9
o0 <- 1
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim1 <- data.frame(p=rep(0:M/M,v))
df_beta1 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
# 80% positive of 500 ratings
o1 <- 400
o0 <- 100
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim2 <- data.frame(p=rep(0:M/M,v))
df_beta2 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
ggplot(data=df_sim1,aes(p)) +
    scale_x_continuous(breaks=0:10/10) +
 
    geom_histogram(aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta1 ,aes(p,y),colour=I("red"),size=2,alpha=.5) +
 
    geom_histogram(data=df_sim2, aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta2,aes(p,y),colour=I("orange"),size=2,alpha=.5)
- See more at: http://www.joyofdata.de/blog/an-intuitive-interpretation-of-the-beta-distribution/#more-1758

The pointy bar plot in the picture represents the frequency of how often it happend in a simulation that for a given assumed "true" $p$ 400 of 500 ratings were good. The bar plot is the density of the histogram of the result of the simulation.

And as you can see - the density curve of the beta distribution for $\alpha=400+1$ and $\beta=100+1$ (orange) tightly surrounds the bar chart (the density of the histogram for the simulation).

library(ggplot2)
 
# 90% positive of 10 ratings
o1 <- 9
o0 <- 1
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim1 <- data.frame(p=rep(0:M/M,v))
df_beta1 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
# 80% positive of 500 ratings
o1 <- 400
o0 <- 100
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim2 <- data.frame(p=rep(0:M/M,v))
df_beta2 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
ggplot(data=df_sim1,aes(p)) +
    scale_x_continuous(breaks=0:10/10) +
 
    geom_histogram(aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta1 ,aes(p,y),colour=I("red"),size=2,alpha=.5) +
 
    geom_histogram(data=df_sim2, aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta2,aes(p,y),colour=I("orange"),size=2,alpha=.5)

Source Link

created Nov 15, 2013 at 20:17

Raffael

1.7k
3
20
30

enter image description here

Let's assume a seller on some e-commerce web-site receives 500 ratings of which 400 are good and 100 are bad.

We think of this as the result of a Bernoulli experiment of length 500 which led to 400 successes (1 = good) while the underlying probability $p$ is unknown.

The naive quality in terms of ratings of the seller is 80% because 0.8 = 400 / 500. But the "true" quality in terms of ratings we don't know.

Theoretically also a seller with "true" quality of $p=77\%$ might have ended up with 400 good of 500 ratings.

The pointy histogram in the picture represents the frequency of how often it happend in a simulation that for a given assumed "true" $p$ 400 of 500 ratings were good.

And as you can see - the density curve of the beta distribution for $\alpha=400+1$ and $\beta=100+1$ (orange) tightly surrounds the histogram (which is resized to a density scale).

So the beta distribution essentially defines the probability that a Bernoulli experiment's success probability is $p$ given the outcome of the experiment.

library(ggplot2)
 
# 90% positive of 10 ratings
o1 <- 9
o0 <- 1
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim1 <- data.frame(p=rep(0:M/M,v))
df_beta1 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
# 80% positive of 500 ratings
o1 <- 400
o0 <- 100
M <- 100
N <- 100000
 
m <- sapply(0:M/M,function(prob)rbinom(N,o1+o0,prob))
v <- colSums(m==o1)
df_sim2 <- data.frame(p=rep(0:M/M,v))
df_beta2 <- data.frame(p=0:M/M, y=dbeta(0:M/M,o1+1,o0+1))
 
ggplot(data=df_sim1,aes(p)) +
    scale_x_continuous(breaks=0:10/10) +
 
    geom_histogram(aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta1 ,aes(p,y),colour=I("red"),size=2,alpha=.5) +
 
    geom_histogram(data=df_sim2, aes(y=..density..,fill=..density..),
        binwidth=0.01, origin=-.005, colour=I("gray")) +
    geom_line(data=df_beta2,aes(p,y),colour=I("orange"),size=2,alpha=.5)
- See more at: http://www.joyofdata.de/blog/an-intuitive-interpretation-of-the-beta-distribution/#more-1758

http://www.joyofdata.de/blog/an-intuitive-interpretation-of-the-beta-distribution/

Stack Exchange Network

Return to Answer