2
$\begingroup$

Suppose to count the drops of rain in a square meter in 15 seconds, producing 16 observations: 40, 20, 24, 15, 23, 12, 39, 26, 29, 33, 16, 36, 17, 32, 40, 15.

What is the probability of counting 28 drops in a following measure?

By simple computations we can find the mean and sigma of this dataset, namely $\mu = 26.06$ and $\sigma = 9.43$.

My issue is the following: from the definition of the problem, I would expect the data to be distributed according to a Poisson distribution, being it a count process. By plotting the dataset in a histogram, however, it is clear that the distribution is far from a Poisson distribution or even a Gaussian.

I could only come up with two possible solutions to this problem:

  1. I nonetheless suppose the dataset is Poissonian (and we would see a Poissonian appear with more measurements) and we can estimate $\lambda$ by taking the average value between $\mu$ and $\sigma$, since they should both converge to $\lambda$ for infinite observations. If this approach is true, the dataset is generated from

$$ P(x) = \frac{\lambda^x}{x!}e^{-\lambda} $$

with $\lambda = \frac{\mu+\sigma}{2} = 16.24$. This approach would give a negligible probability and I would say is clearly wrong.

  1. I forget all my assumptions about the process and only believe the data. In this case I would just find the probability from my histogram. by taking e.g. 6 bins with extremes (10-15|15-20|20-25|25-30|30-35|35-40) (right extreme included) I find the histogram (3, 3, 2, 2, 2, 4). The subset (26, 27, 28, 29, 30) appears with probability $\frac{obs_{(25, 30]}}{obs_{tot}} = \frac{2}{16} = 12.5\%$ and assuming all the numbers uniformly distributed in this subset, since there are 5 numbers, I get $P(x=28) = \frac{12.5\%}{5} = 2.5\%$.

The second result seems much more reasonable but is totally heuristic and obviously depends on the histogram chosen, although I am pretty sure we wouldn't get completely different results and this feels somewhat right.

Question

Is my second approach reasonable? What would be the "smart" way to infer this probability?

Improve this question
$\endgroup$
4
  • $\begingroup$ Perhaps you could set up a hierarchical Bayesian model with a wide prior on the Poisson, but whose mean would be 26.06 and generate umpteen posterior predictive observations and use that as the basis of your estimate. $\endgroup$
    – Avraham
    Commented Mar 13, 2022 at 19:10
  • $\begingroup$ Could you please edit the question to include the source of this example? If it's from homework or a textbook there probably is some associated point that the author is trying to make; knowing details of the source might help point you in the right direction. If this is a homework or self-study question, please replace one of your tags with self-study and read its tag info. $\endgroup$
    – EdM
    Commented Mar 13, 2022 at 19:56
  • $\begingroup$ The source is an old PhD admission test: phys.uniroma1.it/fisica/sites/default/files/allegati/29.pdf but it's in Italian. $\endgroup$
    – Andrea
    Commented Mar 13, 2022 at 19:58
  • $\begingroup$ Conventionally $\mu$ and $\sigma$ are population mean and standard deviation. The corresponding observed sample quantities are more commonly written as $\bar{x}$ and $s$ (or sometimes $s_n$ if the Bessel correction is not applied) $\endgroup$
    – Glen_b
    Commented Mar 13, 2022 at 23:08

1 Answer 1

Reset to default
1
$\begingroup$

The mean and the variance are equal for Poisson-distributed events, so these data aren't very likely to be Poisson--as you recognize. I suspect that's one point the question is getting at.

I don't know that there's a single "smart" approach to evaluating that probability. Your second method is certainly one method, although as you note it's depends on the histogram bins that you choose. You might generalize to average over all 5 width-5 bins that cover the value of 28 (24-28 through 28-32); by my count, that wouldn't be a much different result for these data.

I suspect that the question is expecting you to use a continuous normal approximation based on the observed mean and standard deviation, as the question to which you link does also ask to calculate those statistics. You would treat all continuous values between 27.5 and 28.5 as representing a count of 28. You would take the difference in the (cumulative) distribution function between 28.5 and 27.5 for a normal distribution having the mean and standard deviation you found.

For continuous empirical modeling (if you had appropriate software available during the test) you could generalize your empirical histogram bins into a smoothed density estimate, and use its cumulative distribution similarly to how you would use the normal distribution in the prior paragraph. As for your histogram approach that depends on specific choices, in this case the smoothing bandwidth and kernel that you use. Density plots suggest some bimodality in the values, with one mode in the teens and one in the upper 30s, and a fairly flat distribution in the upper 20s. It's easy to imagine things that aren't really there, however, when there are only 16 observations.

You could try to do discrete negative binomial modeling, one frequent way to model count data with variance higher than the mean. I'm skeptical in this case, as there seem to be a lot of very high values: out of 16, 2 at the highest value of 40, 1 at 39.

As there seems to be a reasonable amount of time allotted for each of the questions on that exam, I suspect that the examiners are interested in the quality of how one presents the thought process, including describing the assumptions involved, not just in the answer. That said, it's always dangerous to try to out-think those who design and grade a test.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.