2
$\begingroup$

Let's consider $M$ bits for which each of them has been flipped by a higher being to either $0$ or $1$. We don't know anything about the underlying random distribution, still the bits are fixed and thus probability $p$ for a bit being $1$ is fixed, too.

We can probe $n$ randomly chosen bits to estimate $p$. Results of the probe follow a Bernoulli distribution with maximum-likelihood estimate[1]: $p = \dfrac{1}{n}\sum_{i=1}^n x_i\\$.

Now let's consider we have just one probe $x_1$ and the result is $1$. This gives the estimate of $p=1$ which seems too much biased to $1$. Even if the higher being has flipped all bits to $1$ (e.g. because it's a natural law or just because it was throwing dices), we shouldn't estimate that for sure (i.e. $p=1$), because other bit combinations are at least possible.

Is there something wrong with this reasoning? If not, what's a more reasonable estimate for this specific example?

Update 1: added "e.g. because it's a natural law or just because it was throwing dices"

Update 2: two other examples of the same type:

  • $p$("stone falls down"), we only probe a single stone and it falls down: should we assume $p=1$?
  • $p$("flipped coin shows heads"), we only flip a single coin and it shows heads: should we assume $p=1$?
Improve this question
$\endgroup$
4
  • $\begingroup$ There is indeed an issue: "because other bit combinations are at least possible". What do you mean by possible? Do you mean that the probability of at least one of the bits being 0 is greater than zero? If you have that knowledge, then you contradict your statement that you don't know anything about the "underlying distribution" (a concept that seems problematic considering there is no randomness yet at that point). $\endgroup$
    – Oxonon
    Dec 18, 2020 at 23:41
  • $\begingroup$ @Oxonon , I have updated my question: we simply don't know whether all bits must actually be 1. We only know that a bit could be 0 or 1. $\endgroup$
    – mstrap
    Dec 18, 2020 at 23:54
  • $\begingroup$ Pretty sure my point stands. We don't know whether a bit could be 0. Stating that you know it could be zero is expressing a belief about the underlying distribution of the bits. So that contradicts your statement of not knowing anything about it. Any way in which you decide to 'estimate' is making assumptions about the underlying distribution. Those are always needed. You just happen to prefer the assumption that p = 0 is possible. $\endgroup$
    – Oxonon
    Dec 19, 2020 at 12:10
  • $\begingroup$ @Oxonon , we can forget about the "because other bit combinations are at least possible" but instead say that "because a bit has two valid states". $\endgroup$
    – mstrap
    Dec 19, 2020 at 12:52

1 Answer 1

Reset to default
3
$\begingroup$

Your reasoning is right. Maximum likelihood estimation is a tool which works well in many situations but isn't well suited to yours. The German tank problem is another situation where MLE gives a questionable result.

The rule of succession is one possible rule of thumb. It corresponds to a Beta(1, 1) prior distribution for p.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! The rule of succession gives a quite intuitive estimate for my example. $\endgroup$
    – mstrap
    Dec 19, 2020 at 1:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.