5
$\begingroup$

Let's say I have a process which generates non-negative integers based on some unknown probability distribution, however those numbers can not be more than some threshold $N$ (by the design of the system). For example, let's say the threshold is $N=1000$. This system can generated a number from set $\{0,1,2,4,5,6,7,8,9,10,...,1000\}$ with unknown probability distribution.

Now, suppose I assume a statistical distribution to model the probability of above result which can take any value in positive domain --- e.g. the Poisson distribution, which theoretically can assume any non-negative integer.

Suppose I sample numbers from this process and get $ X_1, X_2,..., X_{50}$. Now to calculate the expectation, say I use these values and corresponding probabilities from the Poisson distribution. In this case, would I under-estimate or over-estimate the true mean of the process?

I guess I will be under-estimating the true mean because given that to represent probability I am assuming all positive values can occur with certain non zero probabilities, which is not the case for this system. Therefore I am assigning lesser probability to the corresponding number to estimate the expectation.

Is above thought process right? Will it be the case for any value of the threshold?

$\endgroup$
7
  • 1
    $\begingroup$ Could you please explain what you mean by "use ... to estimate"? Exactly how do you propose employing an assumed distribution in this context? You seem to be assuming something unstated about "distributions." For instance, there is no requirement that "all positive values can occur" in a particular distribution. This suggests the need for some clarification, which you can do by editing this post. Consider supplying a small specific example of the kind of processes and assumptions you have in mind. $\endgroup$
    – whuber
    Nov 28, 2021 at 17:14
  • 1
    $\begingroup$ @whuber I have edited the question with more clarity $\endgroup$ Nov 28, 2021 at 17:33
  • 1
    $\begingroup$ Thank you for the clarification. You might find it fruitful to consider what you mean by a "Poisson distribution," because there are infinitely many of them. For every positive number $\lambda$ there is a Poisson distribution whose expectation is $\lambda.$ They all assign positive probabilities to numbers above any specified finite threshold. Which Poisson distribution would you use, then? $\endgroup$
    – whuber
    Nov 28, 2021 at 17:33
  • $\begingroup$ Is it really important to explicitly specify? My question was some theoretical explanation of over/under estimation if I use full statistical distribution whereas the true process is somewhat truncated. Any way I added that information $\endgroup$ Nov 28, 2021 at 17:37
  • 1
    $\begingroup$ The Poisson example is more than adequate to address your general question. $\endgroup$
    – whuber
    Nov 28, 2021 at 17:45

2 Answers 2

3
$\begingroup$

It depends on the estimation method you use, which you haven't really specified clearly. From what you have written here, I take it that you are concerned with the case where you estimate an unbounded distribution first and then truncate the result, instead of estimating directly from a truncated distribution. In that case, you might over- or under-estimate the true mean of the distribution, depending on the particulars of the distribution and the estimation process.

$$\begin{matrix} \text{Method 1} & & \text{Estimate non-truncated distribution} & & \text{...now truncate}, \\[6pt] \text{Method 2} & & \text{Truncate the distribution} \quad \quad \quad \quad \ \ \ & & \text{...now estimate}. \\[6pt] \end{matrix}$$

You seem to be describing the first method above, where you take an initially non-truncated distribution and you estimate it first, prior to truncation. You are right that, on the one hand, your initial estimate will ascribe non-zero probabilities to values above the threshhold, which would tend towards over-estimating the true mean. However, you have not taken account of the fact that your data will only ever fall below the threshold, so this will operate in the other direction.

If you were to use method-of-moments estimation using the Poisson distribution then the first method will consistently under-estimate the mean (see below). However, if you were to use maximum-likelihood estimation then things are more ambiguous.


An example: To run with your example, suppose we use a Poisson distribution (which is not truncated) with rate parameter $\lambda$ and we estimate the non-truncated distribution using the method-of-moments. As a preliminary result, define the partial exponential function by:

$$E_N(\lambda) \equiv \sum_{k=0}^N \frac{\lambda^k}{k!}.$$

For a truncated Poisson distribution with upper-bound $N$ we have mean:

$$\begin{align} \mathbb{E}(K) &= \frac{\sum_{k=0}^N k \cdot \text{Pois}(k|\lambda)}{\sum_{k=0}^N \text{Pois}(k|\lambda)} \\[6pt] &= \frac{\sum_{k=0}^N k \cdot \frac{\lambda^k e^{-\lambda}}{k!}}{\sum_{k=0}^N \frac{\lambda^k e^{-\lambda}}{k!}} \\[6pt] &= \frac{\sum_{k=0}^N k \cdot \frac{\lambda^k}{k!}}{\sum_{k=0}^N \frac{\lambda^k}{k!}} \\[6pt] &= \frac{\lambda \sum_{k=0}^{N-1} \frac{\lambda^k}{k!}}{\sum_{k=0}^N \frac{\lambda^k}{k!}} \\[6pt] &= \frac{\lambda E_{N-1}(\lambda)}{E_N(\lambda)}. \\[6pt] \end{align}$$

Under the first method we estimate $\hat{\lambda} = \bar{x}_n$ for a non-truncated Poisson distribution and then we truncate, which gives the estimated mean:

$$\widehat{\mathbb{E}(K)} = \frac{\bar{x}_n E_{N-1}(\bar{x}_n)}{E_N(\bar{x}_n)}.$$

Under the second method we estimate directly from the truncated Poisson distribution, which gives:

$$\widehat{\mathbb{E}(K)} = \bar{x}_n.$$

The second estimator is an unbiased estimator for IID data, and in this example the first estimator is lower than the second, so it will tend to under-estimate the true mean of the truncated Poisson distribution (i.e., it has negative bias).

$\endgroup$
2
$\begingroup$

If the threshold value is $1000,$ then it seems safe to use a Poisson distribution with mean $\lambda = 20,$ because $P(X > 1000) \approx 0,$ for $X\sim\mathsf{Pois}(\lambda = 20).$ [Computation in R.]

1 - ppois(1000, 20)
[1] 0

A simulation of a million samples of size $n=200$ from $\mathsf{Pois}(20),$ with truncation of any observations above $1000.$.

set.seed(1128)
m = 10^6;  lam = 20;  mx = 1000;  n= 200;  a = numeric(m)
for(i in 1:m) {
 x = rpois(n,lam)
 x[x > mx] = mx    $ truncation to 1000
 a[i] = mean(x)    # sample mean
 }
mean(a);  var(a)
[1] 20.00028
[1] 0.1000547

The distribution of sample means $\bar X_{200}$ [as in code] is approximately normal and a 95% CI for $\lambda$ is $(19.38, 20.62).$ Truncation, if any, has had no noticeable effect.

mean(a) + qnorm(c(.025,.975))*sd(a)
[1] 19.38031 20.62024

hist(a, prob=T, col="skyblue2", main="Simulated Poisson Means")
 curve(dnorm(x, mean(a), sd(a)), add=T, lwd=2, col="brown")

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.