2
$\begingroup$

I have a partial set of measurement data that is limited due to my tool's sensitivity. I know that the data is approximately normally distributed and I have a standard deviation from another data set that was within my measurement tool's range.

Is it possible to use this limited data to estimate its mean? In one case I have the upper 59% of the data and in another case I have the top 11%.

$\endgroup$
2
  • 3
    $\begingroup$ Yes you can, by expressing the joint distribution of what is observed, namely the upper 59% of a normally distributed sample. $\endgroup$
    – Xi'an
    Dec 7, 2023 at 9:45
  • $\begingroup$ I'm not sure that I follow. Do you mean to take a scatter plot with the upper 59% on one axis and the full normal distribution on the other? $\endgroup$
    – Shuesh
    Dec 7, 2023 at 15:50

2 Answers 2

5
$\begingroup$

Given an iid sample $(X_1,\ldots,X_n)$, with density $f_\theta$, the sub-vector of order statistics $V=(X_{(\alpha n)},\ldots,X_{(n)})$ [where $0<\alpha<1$ and $\alpha n\in\mathbb N^*$] has density $$\frac{n!}{(\alpha n-1)!}F_\theta(x_{(\alpha n)})^{\alpha n-1}\prod_{i=\alpha n}^n f_\theta(x_{(i)})\ \mathbb I_{x_{(\alpha n)}\le\cdots\le x_{(n)}}$$ This turns into the likelihood when seen as a function of $\theta$, hence it can be on principle maximised in $\theta$.

Furthermore, since this is a missing data model, the EM algorithm can be implemented here. More precisely, this means considering the missing part of the sample, $W=(X_{(1)},\ldots,X_{(\alpha n-1)})$, in the complete log-likelihood $$-n \log\sigma-\frac{1}{2\sigma^2}\sum_{i=1}^n (X_{(i)}-\mu)^2\\=-n \log\sigma-\frac{1}{2\sigma^2}\sum_{i=1}^{\alpha n-1} (X_{(i)}-\mu)^2\\\qquad-\frac{1}{2\sigma^2}\sum^{n}_{i=\alpha n} (X_{(i)}-\mu)^2$$ and taking its expectation $$\mathbb E_{\mu_0,\sigma_0}\left[-\frac{1}{2\sigma^2}\sum_{i=1}^{\alpha n-1} (X_{(i)}-\mu)^2\Big|V\right]$$ even though the first two moments of a Normal order statistic are not available in closed form. However, the update in the M-step of the EM algorithm only involves $$\mu_{t+1}=\mathbb E_{\mu_t,\sigma}\left[\overline{X_{1:n}}\big|X_{(\alpha n)},\ldots, X_{(n)}\right]$$ which can be expressed as $$\frac{\alpha n-1}{n} \left\{\mu_t-\sigma f_{\mu_t,\sigma}(x_{(\alpha n)})\big/F_{\mu_t,\sigma}\right\}+\frac{1}{n}\sum_{i=\alpha n}^n X_{(i)}$$ with some intermediate steps$^1$, including the derivation of a truncated Normal mean. In conclusion, the EM algorithm can be implemented without any Monte Carlo step.


$^1$Another explanation for the result is that $W$ conditional on $V$ is (i) conditional on $X_{(\alpha n)}$ only and (ii) distributed from the original distribution truncated at $X_{(\alpha n)}$.

$\endgroup$
0
$\begingroup$

This is simple using the method of moments.

Suppose $X$ is normally distributed, $X\sim N(\mu,\sigma)$. And suppose $Y$ is the highest $59\%$ of $X$. Then linear transformations of the numerical calculations for $N(0,1)$ give: \begin{align} E[Y]&=\mu + 0.948\sigma\\ E[Y^2]&=\mu^2 + 2(0.948)\mu\sigma + 1.216\sigma^2 \end{align}

In more generality, let $Y$ be the highest $r$ of $X$, where $0<r<1$. Let $Q_r$ be the quantile of the normal distribution for $1-r$, so $Q_{0.59}=-0.228$ and $Q_{0.11}=1.227$. Then \begin{align} E[Y]&=\mu + K_r\sigma\\ E[Y^2]&=\mu^2 + 2K_r \mu \sigma + \sigma^2(1 - K_r Q_r)\\ \text{where }K_r&=\frac{e^{-Q_r^2/2}}{\sqrt{2\pi}(1-r)} \end{align}

So $E[Y]$ and $E[Y^2]$ can be estimated from the sample, and then $\mu$ and $\sigma$ can be determined by solving the two equations. Or, if a value for $\sigma$ is taken from another source, $\mu$ can be estimated from the first equation alone.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.