Working through a textbook on statistical inference and looking at the introductory example of estimating a probability from a proportion.

In the book it says that " the underlying statistical model assumes that a certain event happens with probability p".

Is this probability p obtained from the number of occurrences in a sample divided by the sample size ?

  • 3
    $\begingroup$ Your estimate of $p$ comes from the proportion of occurrences. The real value of $p$ is out there in nature, created by whatever it is that you believe about creation (God, Big Bang, etc). We then use our estimate of $p$ to say something about the real value of $p$. This is the inference. $\endgroup$
    – Dave
    Feb 11 '20 at 12:19

The answer to your question can be very simple or very deep, even with some philosophical idea involved (e.g., Bayesian and frequentist).

I will try to answer it using frequentist's point of view and with Maximize Likelihood Estimation.

We will start with the coin flip example. Let's assume each coin has its own attributes, may be this attributes is related to the physical mass distribution of the coin or the exact shape of the coin (may be the mass is not evenly distributed or not perfect round shape), but for one given coin, it has one parameter $\theta$ (probability of getting head), and this parameter has a "true value", we want to estimate.

Note that, this "true" value is "fixed" and unknown andWe can use experiment to estimate $\theta$.

Suppose we flip this coin $10$ times and we get $6$ head. How would we estimate the $\theta$? By intuition, we may say, we use the occurrence divided by the sample size. But why we have this intuition?

The answer is this is the Maximize the Likelihood Estimation (MLE). Note that, we can estimate $\theta$ by other estimator and the $\theta$ does not need to be 6/10. But it is very reasonable to use MLE. Here is why.

Assume independent samples, the probability of getting data is

$$ \theta^6(1-\theta)^4 $$

On the other hand, we know $\theta$ is between $0$ and $1$. If we plot the probability of getting data respect $\theta$ we get:

enter image description here

Note that, the probability of getting data (likelihood function) is maximized when we set $\theta$ to $0.6$. And this is why we use the empirical frequency to estimate the unknown parameter

  • $\begingroup$ $\frac{6}{10}$ in this example is also the naive frequentist estimate (abusing the law of large numbers) and an unbiased frequentist estimate, and so can be obtained without a maximum likelihood approach $\endgroup$
    – Henry
    Feb 11 '20 at 22:59
  • $\begingroup$ by "mass distribution" do you mean probability mass distribution or the actual physical mass distribution of the coin? $\endgroup$ Feb 29 '20 at 10:49
  • $\begingroup$ @stochasticmrfox physical mass。thanks for the comment My answer is revised $\endgroup$
    – Haitao Du
    Feb 29 '20 at 17:38

Since Haitao gave an explanation of a frequentist approach, I will supply a Bayesian one.

In a Bayesian setting, we still generally believe that there is a "true" $p$. We want to understand how probable different values of $p$ are given the data we have observed. In the coin flip example, $p$ is the probability of observing heads. Say we have a fair coin, and we flip it 100 times, and get 40 heads.

p <- 0.5
flips <- 100
heads <- 40

We can then use the binomial distribution to tell us how likely we would be to observe these results for different values of $p$.

s <- seq(0, 1, length.out = 1000)
  s, dbinom(heads, size = flips, prob = s), type="l",
  xlab = "p (probability of heads)",
  ylab = "Binomial likelihood"

In this case, a maximum likelihood estimator would give us a result of $p=0.4$.

However, imagine we have prior knowledge about how "fair" coins are in general. We could say that the distribution of $p$ (chance of heads) across all coins we've ever seen is described by a Beta distribution, say $\text{Beta}(50, 50)$

prior_alpha <- 50
prior_beta <- 50
  s, dbeta(s, prior_alpha, prior_beta), type="l",
  xlab = "p (probability of heads)",
  ylab = "Proportion of all coins"

We would like to combine this prior knowledge with our likelihood distribution. Formally, this is allowed by Bayes theorem: $$p(a|b) = \frac{p(b|a) p(a)}{\int p(b|a)p(a) da}$$

I'll skip the math here. Suffice to say that when we combine out beta and binomial distribution, we get a Beta distribution with updated parameters. This is because beta is a conjugate prior for the binomial distribution. In this case we take the $\alpha$ of our prior and add heads, and take the $\beta$ of our prior and add flips - heads.

    shape1 = prior_alpha + heads,
    shape2 = prior_beta + flips - heads
  type = "l",
  xlab = "p (probability of heads)",
  ylab = "Posterior density"

By using prior information like this, you can see that we've shrunk our guesses towards what we believe about coins in general.


This is saying that we're assuming there is some p such that the event happens with probability p. At this point, we are not making any assumptions about p other than it exists. We later make an estimate of p, but p itself is not obtained; it remains an unknown theoretical quantity.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.