I have a bunch of measurements $D_1, D_2, ..., D_N$ and their associated uncertainties $\sigma_1, \sigma_2, ..., \sigma_N$. Suppose that the (unknown) true values of the data points are $T_1, T_2, ..., T_N$. I can also calculate for any $C$ and any measured value $D_i$ the probability $P_i(C)$ that $T_i<C$. If the measurement uncertainties are normal then $P_i(C)$ will just be the integral from $-\infty$ to $C$ of a Gaussian, but in principle I can do the same for other PDFs.

The only thing I know about the underlying population from which the $D_i$ were drawn is that there exists some upper limit $C_\text{lim}$ for which all the $T_i<C_\text{lim}$ even though, due to measurement uncertainty, some of the $D_i>C_\text{lim}$. I'd like a sensible way to judge whether a candidate value $C$ for $C_\text{lim}$ is plausible, given the data. At first I thought I should just multiply together all the $P_i(C)$, but I found that this gives me unrealistically high upper limits.

The reason is that if I have many measurements some of their true values will be near $C_\text{lim}$ and, due to noise, will be measured at a higher value. Thus I expect that a candidate $C$ will still be reasonable even if it is slightly lower than a few of the data points. What I'm after is an objective probability, or perhaps a plausibility, $Pl(C)$ that all the $D_i$ are consistent with having lower true values. If possible I would like to avoid assuming anything of the underlying population.

I suspect the answer I want is $Pl(C)=\exp\left[\dfrac{1}{N}\displaystyle\sum_{i=1}^N \log_e P_i(C)\right]$. It gives me sensible-looking results when I apply it to various mock data sets for which I already know the true $C_\text{lim}$. But I have no explanation for why it should be right. Is it correct? How should I approach a problem like this?


Here is one of my mock data sets. I generated 160 random data points with true values $T_i$ between 5 and 10. Then I simulated measurement error by moving them up or down by a $\sigma=1$ normally distributed random value. Even though $C_\text{lim}=10$ there are quite a few measured data points above $D_i=10$. I would expect then that $Pl(10)\sim 0.5$ because it is definitely plausible that the true values are all below 10. And I'd expect also $Pl(8)\approx 0$ and $Pl(12)\approx 1$.

Mock data set

  • 1
    $\begingroup$ One problem your calculation will have is that for $C$ large enough, all $C_j > C$ will have essentially the same $Pl(C_j) (=0)$. Not sure what to do about this, unless you can put a prior distribution on $C$ or learn more about the distribution of the $D_i$ than just that they have an upper bound. $\endgroup$ – jbowman Aug 9 '17 at 14:45
  • $\begingroup$ For very large $C_j,$ I'll have $P_i(C_j) $essentially equal to 1, since all my measured data points will be well beneath it, so my $Pl(C_j)$ will also be 1. This is telling me nothing except that it is highly likely that my true upper bound is somewhere beneath that. I expect $Pl(C)$ as a function of C to look sort of like an error function with the inflection point somewhere near the true upper bound $C_{lim}$ $\endgroup$ – TheBigH Aug 9 '17 at 16:12
  • $\begingroup$ Would you share your dataset? Is $D_{1}$ a single data point/measurement? Is $C$ independent of $i$ in $D_{i}$, i.e. $C \ne C_{i}$? $\endgroup$ – user137329 Aug 10 '17 at 17:34
  • $\begingroup$ Is your $P_{i}\left(C\right) = N_{\mu=D_{i},\sigma=\sigma_{i}}\left(C\right)$? $\endgroup$ – user137329 Aug 10 '17 at 17:50
  • 1
    $\begingroup$ I've improved the description of the problem to hopefully make it more precise. $\endgroup$ – TheBigH Aug 16 '17 at 13:51

Quick comment that your expression simplifies to $Pl \left( C \right) = \left( \prod_{i=1}^{N}P_{i} \left( C \right) \right)^{1/N}$

-- added after the simulation data was posted --

So if all of the $\sigma_{i}$ are the same, you might try to 'deconvolute' that uncertainty to get at the upper limit. Say your underlying population of $\{T_{i}\}$ were normal with standard deviation $\Sigma$, then you could estimate $\hat{\Sigma}=\sqrt{s^{2}-\sigma^{2}}$ where $s$ is the variance of the $D_{i}$?

  • 2
    $\begingroup$ Yep. In fact that was the form I first thought of it, but I prefer the summing of logs because it seems to give me fewer rounding issues on a computer. $\endgroup$ – TheBigH Aug 9 '17 at 14:30
  • 1
    $\begingroup$ or you could try a deconvolution by dividing Fourier Transforms ... terpconnect.umd.edu/~toh/spectrum/Deconvolution.html $\endgroup$ – user137329 Aug 17 '17 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.