# Probability distribution for a limiting value

I have a bunch of measurements $D_1, D_2, ..., D_N$ and their associated uncertainties $\sigma_1, \sigma_2, ..., \sigma_N$. Suppose that the (unknown) true values of the data points are $T_1, T_2, ..., T_N$. I can also calculate for any $C$ and any measured value $D_i$ the probability $P_i(C)$ that $T_i<C$. If the measurement uncertainties are normal then $P_i(C)$ will just be the integral from $-\infty$ to $C$ of a Gaussian, but in principle I can do the same for other PDFs.

The only thing I know about the underlying population from which the $D_i$ were drawn is that there exists some upper limit $C_\text{lim}$ for which all the $T_i<C_\text{lim}$ even though, due to measurement uncertainty, some of the $D_i>C_\text{lim}$. I'd like a sensible way to judge whether a candidate value $C$ for $C_\text{lim}$ is plausible, given the data. At first I thought I should just multiply together all the $P_i(C)$, but I found that this gives me unrealistically high upper limits.

The reason is that if I have many measurements some of their true values will be near $C_\text{lim}$ and, due to noise, will be measured at a higher value. Thus I expect that a candidate $C$ will still be reasonable even if it is slightly lower than a few of the data points. What I'm after is an objective probability, or perhaps a plausibility, $Pl(C)$ that all the $D_i$ are consistent with having lower true values. If possible I would like to avoid assuming anything of the underlying population.

I suspect the answer I want is $Pl(C)=\exp\left[\dfrac{1}{N}\displaystyle\sum_{i=1}^N \log_e P_i(C)\right]$. It gives me sensible-looking results when I apply it to various mock data sets for which I already know the true $C_\text{lim}$. But I have no explanation for why it should be right. Is it correct? How should I approach a problem like this?

EDITS:

Here is one of my mock data sets. I generated 160 random data points with true values $T_i$ between 5 and 10. Then I simulated measurement error by moving them up or down by a $\sigma=1$ normally distributed random value. Even though $C_\text{lim}=10$ there are quite a few measured data points above $D_i=10$. I would expect then that $Pl(10)\sim 0.5$ because it is definitely plausible that the true values are all below 10. And I'd expect also $Pl(8)\approx 0$ and $Pl(12)\approx 1$.

• One problem your calculation will have is that for $C$ large enough, all $C_j > C$ will have essentially the same $Pl(C_j) (=0)$. Not sure what to do about this, unless you can put a prior distribution on $C$ or learn more about the distribution of the $D_i$ than just that they have an upper bound. – jbowman Aug 9 '17 at 14:45
• For very large $C_j,$ I'll have $P_i(C_j)$essentially equal to 1, since all my measured data points will be well beneath it, so my $Pl(C_j)$ will also be 1. This is telling me nothing except that it is highly likely that my true upper bound is somewhere beneath that. I expect $Pl(C)$ as a function of C to look sort of like an error function with the inflection point somewhere near the true upper bound $C_{lim}$ – TheBigH Aug 9 '17 at 16:12
• Would you share your dataset? Is $D_{1}$ a single data point/measurement? Is $C$ independent of $i$ in $D_{i}$, i.e. $C \ne C_{i}$? – user137329 Aug 10 '17 at 17:34
• Is your $P_{i}\left(C\right) = N_{\mu=D_{i},\sigma=\sigma_{i}}\left(C\right)$? – user137329 Aug 10 '17 at 17:50
• I've improved the description of the problem to hopefully make it more precise. – TheBigH Aug 16 '17 at 13:51

Quick comment that your expression simplifies to $Pl \left( C \right) = \left( \prod_{i=1}^{N}P_{i} \left( C \right) \right)^{1/N}$
So if all of the $\sigma_{i}$ are the same, you might try to 'deconvolute' that uncertainty to get at the upper limit. Say your underlying population of $\{T_{i}\}$ were normal with standard deviation $\Sigma$, then you could estimate $\hat{\Sigma}=\sqrt{s^{2}-\sigma^{2}}$ where $s$ is the variance of the $D_{i}$?