# Can I estimate the mean of a dataset if I have its standard deviation and a portion of the full data that is higher than some threshold?

I have a partial set of measurement data that is limited due to my tool's sensitivity. I know that the data is approximately normally distributed and I have a standard deviation from another data set that was within my measurement tool's range.

Is it possible to use this limited data to estimate its mean? In one case I have the upper 59% of the data and in another case I have the top 11%.

• Yes you can, by expressing the joint distribution of what is observed, namely the upper 59% of a normally distributed sample. Dec 7, 2023 at 9:45
• I'm not sure that I follow. Do you mean to take a scatter plot with the upper 59% on one axis and the full normal distribution on the other? Dec 7, 2023 at 15:50

Given an iid sample $$(X_1,\ldots,X_n)$$, with density $$f_\theta$$, the sub-vector of order statistics $$V=(X_{(\alpha n)},\ldots,X_{(n)})$$ [where $$0<\alpha<1$$ and $$\alpha n\in\mathbb N^*$$] has density $$\frac{n!}{(\alpha n-1)!}F_\theta(x_{(\alpha n)})^{\alpha n-1}\prod_{i=\alpha n}^n f_\theta(x_{(i)})\ \mathbb I_{x_{(\alpha n)}\le\cdots\le x_{(n)}}$$ This turns into the likelihood when seen as a function of $$\theta$$, hence it can be on principle maximised in $$\theta$$.

Furthermore, since this is a missing data model, the EM algorithm can be implemented here. More precisely, this means considering the missing part of the sample, $$W=(X_{(1)},\ldots,X_{(\alpha n-1)})$$, in the complete log-likelihood $$-n \log\sigma-\frac{1}{2\sigma^2}\sum_{i=1}^n (X_{(i)}-\mu)^2\\=-n \log\sigma-\frac{1}{2\sigma^2}\sum_{i=1}^{\alpha n-1} (X_{(i)}-\mu)^2\\\qquad-\frac{1}{2\sigma^2}\sum^{n}_{i=\alpha n} (X_{(i)}-\mu)^2$$ and taking its expectation $$\mathbb E_{\mu_0,\sigma_0}\left[-\frac{1}{2\sigma^2}\sum_{i=1}^{\alpha n-1} (X_{(i)}-\mu)^2\Big|V\right]$$ even though the first two moments of a Normal order statistic are not available in closed form. However, the update in the M-step of the EM algorithm only involves $$\mu_{t+1}=\mathbb E_{\mu_t,\sigma}\left[\overline{X_{1:n}}\big|X_{(\alpha n)},\ldots, X_{(n)}\right]$$ which can be expressed as $$\frac{\alpha n-1}{n} \left\{\mu_t-\sigma f_{\mu_t,\sigma}(x_{(\alpha n)})\big/F_{\mu_t,\sigma}\right\}+\frac{1}{n}\sum_{i=\alpha n}^n X_{(i)}$$ with some intermediate steps$$^1$$, including the derivation of a truncated Normal mean. In conclusion, the EM algorithm can be implemented without any Monte Carlo step.

$$^1$$Another explanation for the result is that $$W$$ conditional on $$V$$ is (i) conditional on $$X_{(\alpha n)}$$ only and (ii) distributed from the original distribution truncated at $$X_{(\alpha n)}$$.

This is simple using the method of moments.

Suppose $$X$$ is normally distributed, $$X\sim N(\mu,\sigma)$$. And suppose $$Y$$ is the highest $$59\%$$ of $$X$$. Then linear transformations of the numerical calculations for $$N(0,1)$$ give: \begin{align} E[Y]&=\mu + 0.948\sigma\\ E[Y^2]&=\mu^2 + 2(0.948)\mu\sigma + 1.216\sigma^2 \end{align}

In more generality, let $$Y$$ be the highest $$r$$ of $$X$$, where $$0. Let $$Q_r$$ be the quantile of the normal distribution for $$1-r$$, so $$Q_{0.59}=-0.228$$ and $$Q_{0.11}=1.227$$. Then \begin{align} E[Y]&=\mu + K_r\sigma\\ E[Y^2]&=\mu^2 + 2K_r \mu \sigma + \sigma^2(1 - K_r Q_r)\\ \text{where }K_r&=\frac{e^{-Q_r^2/2}}{\sqrt{2\pi}(1-r)} \end{align}

So $$E[Y]$$ and $$E[Y^2]$$ can be estimated from the sample, and then $$\mu$$ and $$\sigma$$ can be determined by solving the two equations. Or, if a value for $$\sigma$$ is taken from another source, $$\mu$$ can be estimated from the first equation alone.