# What statistic best estimates the sample mean in case of missing data in a distribution?

I have samples of particles and am interested in the particle lengths. The problem is that the samples are assessed using image analysis. As the particles overlap, the measurements are incomplete and longer lengths are over-represented relative to shorter ones.

How can I estimate the average particle length in each sample from the biased set of measured lengths?

You can assume:

• The measurement of the longest particle is accurate in each sample. However, the mean measurement or median measurement may not reflect the mean length or median length accurately.
• The number of particles in a sample varies between 10 and 50.
• Each particle typically overlaps with 1 to 5 others.
• The distribution of lengths is unknown but presumably not Gaussian, and likely slightly skewed to large values with few short values.
• The distributions for different samples can vary, but have similar shapes.
• The accuracy of a statistic can be computed as the expectation of its squared error.

What statistic based on this information would most accurately represent the average length for each sample?

• If there is an answer, it would depend on what you mean by "best represents the average". Commented Jun 5 at 10:11
• @PeterFlom Intuitively, if I have N samples, the sum of squared deviations between the true means and my selected statistic for those distributions is minimized, so that the SSE is minimal. Commented Jun 5 at 10:12
• OK. Please edit your question to include that -- not everyone reads comments. (And welcome to CV!) Commented Jun 5 at 10:18
• I have had good success using physical and geometric models for such situations, such as estimating the distribution of spherical boulders based on thin vertical probes through geological formations. I believe a careful analysis of the particle geometries and how they relate to the image analysis might similarly be the most effective approach here. I can't offer any specific pointers due to the vague description in this post.
– whuber
Commented Jul 12 at 14:20
• For the obscured particles, do you observe partial data (censored lengths) or you can't even detect them in the data at all (truncation)? Commented Jul 12 at 17:43

The way I interpret the setting suggests solutions based on robust statistics. When measured by the breakdown point, the empirical mean, sample maximum, and sample minimum are not adversarially robust, as changing only one sample can make them arbitrarily large. In contrast, the median is a robust measure of central tendency, as it requires half the samples to be outliers before the median can be moved outside the range of the non-outliers.

If the underlying distribution is symmetric around the median (e.g., Gaussian), the Hodges–Lehmann estimator is a consistent and median-unbiased estimate of the population median. If the underlying distribution is not symmetric, the Hodges–Lehmann estimator converges to the pseudo-median, which is a valid measure of central tendency and remains "close" to the true median even when the distribution is fairly asymmetric.

• I find the language here unusual: "make them as large as desired". Do you mean "make arbitrarily large"? // Thanks, this is a useful way to think about using the median. Note however that part of the problem is missing data, not erroneous data such as outliers. A good example would be something like the spectral distribution of the sun although in my case I don't know the shape of the distribution other than to assume it is unimodal. Commented Jun 5 at 19:04
• This may not work well with the overlaps. Suppose, roughly as suggested, that each particle overlaps 3 others, and is only visible if it is longer than all the particles it overlaps. Then the median of the visible particles would be at the $2^{-1/4}$ quantile, or the 84th percentile of the overall particle population. Similarly the median of the overall particle population would be at the $2^{-4}$ quantile or the 6.25th percentile of the visible population.
– user225256
Commented Jul 13 at 2:02

Here is an answer assuming that the only sample statistics which can be calculated reliably are the size $$k_i$$ and the maximum $$M_i$$ of the sample.

The question is how to estimate the average $$\mu_i$$ for each sample. This is tractable if we assume that all $$n$$ samples are from the same distribution, of a specified form, and can use the collection of $$n$$ sample maxima to estimate the population parameters.

Example

Suppose all the $$k$$’s are equal, and we only need a population average.

Estimating the parameter:

Solve for $$a$$ in $$\frac{\Gamma[1-2a]}{\Gamma[1-a]^2}=\frac{n\sum m_i^2}{(\sum m_i)^2}$$

This can be approximated by $$a\sim \frac1{1+\sqrt{1+\pi^2z^2/6}}$$ where $$z$$ is the ratio of the mean to the standard deviation of the $$M$$’s.

Estimating the average:

Calculate $$\mu=\frac{k^{-a}\sum m_i}{n}$$

This average correctly increases with higher means for the observed maxima, decreases with higher standard deviations for the observed maxima, and decreases with higher sample sizes.

More Detail

Suppose that the particle lengths $$X$$ have a Frechet$$(1/a,s,0)$$ distribution with $$0 and minimum $$0$$. Then \begin{align} P[X\le x] &= \exp(-(x/s)^{-1/a})\\ E[X] &= s\, \Gamma[1-a]\\ E[X^2] &= s^2\Gamma[1-2a]\\ \end{align}

Then $$M_i$$ has a Frechet$$(1/a,k_i^as,0)$$ distribution with minimum 0, and \begin{align} P[M_i\le x] &= \exp(-k_i(x/s)^{-1/a})\\ E[M_i] &= k_i^as\, \Gamma[1-a]\\ E[M_i^2] &= k_i^{2a}\,s^2\Gamma[1-2a] \end{align}

Estimating the parameters:

The parameters $$a$$ and $$s$$ can be estimated by the method of moments, by solving equations with the observed $$k_i$$ and $$m_i$$: \begin{align} s \Gamma[1-a] \sum k_i^a&=\sum m_i\\ s^2\Gamma[1-2a]\sum k_i^{2a}&= \sum m_i^2 \end{align}

Estimating the averages:

The sample averages can be estimated in terms of the known maxima and the averages of the population below them: \begin{align} \mu_i &= \frac{m_i+(k_i-1)E[X|X where $$u_i=(m_i/s)^{-1/a}$$ and this uses the lower incomplete gamma function.

• For any heavy-tailed distribution, the Fisher-Tippett-Gnedenko theorem says that the maxima are approximately Frechet-distributed. So any analytical approach to this situation would probably use the Frechet distribution somewhere, and it’s convenient to assume that the original population is also Frechet.

• The problem would require more assumptions or more data if the samples can have different Frechet distributions.

Here is an answer assuming that each measured value is reasonably accurate, but that the measurements are incomplete due to overlaps: you might get the median particle length as the $$6^{th}$$ or $$13^{th}$$ percentile of the measured lengths.

Suppose a particle is only visible and measurable if it is bigger than the particles it overlaps. Let $$U$$ be a variable for the (mostly unseen) original population and let $$V$$ be a variable for the visible population. Let a particle of size $$x$$ be bigger than $$F_U(x)$$ of the original population and bigger than $$F_V(x)$$ of the visible population.

Case with equal numbers of overlaps: Suppose each particle overlaps exactly three other particles. Then a particle of size $$x$$ has a probability $$F_U(x)^3$$ of being seen, and $$F_V(x) = \frac{\int_0^{F_U(x)} p^3 dp}{\int_0^1 p^3 dp} = F_U(x)^4$$ So when $$F_U(x)=1/2$$, $$F_V(x)=1/16$$, and the median in the original population is at roughly the $$6^{th}$$ percentile in the visible population.

Case with uniform distribution of overlaps: Suppose a particle can overlap 1, 2, 3, 4, or 5 other particles, with an even distribution among those five possibilities. Then \begin{align}F_V(x) &= \frac{\int_0^{F_U(x)} \frac15(p+p^2+p^3+p^4+p^5)dp}{\int_0^1 \frac15(p+p^2+p^3+p^4+p^5)dp }\\ &= \frac{\frac{F_U(x)^2}2+ \frac{F_U(x)^3}3+ \frac{F_U(x)^4}4+ \frac{F_U(x)^5}5+ \frac{F_U(x)^6}6}{ \frac12+\frac13+\frac14+\frac15+\frac16} \end{align} So when $$f_U(x)=1/2$$, $$f_V(x)=367/2784$$, and the median in the original population is at roughly the $$13^{th}$$ percentile in the visible population.

Sensitivities and Means: This calculation seems sensitive to the distribution of particle overlaps, even though it is not sensitive to the distribution of particle sizes.

For a mean in the simple case, the equations $$F_U(x)=F_V(x)^{1/4}$$ and $$f_u(x)dx =\frac14 F_V(x)^{-3/4} f_v(x)dx$$ suggest that the mean of the $$U$$’s might be a weighted average of the $$V$$’s, weighting each length by the $$-3/4$$ power of its position in the distribution. But I don’t see how to calculate such a mean reliably, since it so overweights the smallest values and has such sensitivity to them.