0
$\begingroup$

I have a large but finite set of objects (phylogenetic trees), each of which is assigned an integer value 0 ≤ vx. x varies from set to set, but is small (≪ 100).

For a given set, I wish to estimate p(vn) for each n in 0 → x.

p(v = 0) can be calculated exactly. Because p(v = n) is close to 1/n when n ~ x, I'm able to obtain good estimates by sampling many (a million, say) objects; the standard error can be calculated using the binomial distribution. But p(v = n) is very small when n is small: even a large sample may contain zero objects with n = 1 or n = 2.
I wish to interpolate between my precise estimates for large n and the exact value known for p(n = 0) in order to obtain estimates for p(n ≤ 1, 2, …) in a way that reflects the increasing trend in p as n increases, and to calculate the uncertainty associated with these estimates.

My approach so far has been to attempt a non-linear regression, but I've not found a way to conduct this or other interpolation techniques that I've encountered in a manner that expressly accounts for the known uncertainty of the estimated values.

$\endgroup$

1 Answer 1

0
$\begingroup$

My understanding from what you wrote is that you want to adjust the estimates for v=1 or 2 because you observed 0 of them in the simulation and you know the probability has to be greater than 0.

When v=1 or 2 and you observe 0 of those out of 1 million, you can be pretty confident that the probability mass function at those two points is less than $3 \times 10^{-6}$ (3 out of a million). That's a 95% upper confidence limit for the probability mass at those points. If you know the true probability cannot be 0 and you want to force it to be something other than 0, you could replace it be something positive and less than $3 \times 10^{-6}$. If $P[v=0]$ is known exactly and you are estimating all the rest by simulation, you can do that and then re-weight the estimates for those $v$ greater than 0 so that all the mass adds to 1.

If you want to do something more sophisticated that allows you to use prior knowledge and the uncertainty from the simulated values directly, then you could try a Bayesian approach to density estimation. That's going to be alot of work and not worth the effort in my opinion. One other option using a Bayesian idea is to use Laplace rule of succession. If you have a uniform prior on P[v=1] and then you observed 0 occurrences of v=1 out of 1 million simulated objects, the posterior mean estimate of P[v=1] is 1/(1,000,002). You clearly cannot have Uniform(0,1) priors for all values of v so this approach doesn't really hold water without some modification, but this is just some idea of another estimate you could use instead of 0.

$\endgroup$
1
  • $\begingroup$ Thanks for these suggestions. I am hoping for an approach that assigns a higher probability to higher values of v. I need to approximate log(p(n <= 1)) - log(p(n = 0)) and log(p(n <= 2)) - log(p(n <= 1)), so the extra precision that a Bayesian approach might provide would be useful. $\endgroup$
    – ms609
    Feb 14, 2021 at 6:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.