how to set an upper threshold for a property that is calculated from experimental results I can't really place this problem theoretically, and I wonder if anyone could please point me to relevant posts/literature or provide some advice.
A property $X$ of a sample can be measured using some instrument.
From historical data, it is known that if the measurement of the same sample is repeated many times, the distribution of the values of $X$ is approximately Gaussian, with mean $X_{mean}$ and standard deviation $\sigma_X$, where $\sigma_X$ is ~independent from $X_{mean}$ and ~constant over time.
In day-to-day activities, an in vitro assay is run by taking two separate samples and measuring $X$ for each, thus obtaining $X_1$ and $X_2$.
I suppose they can't really be called 'means', but given the above, we believe that they come from distributions with the same standard deviation $\sigma_X$.
From the assay setup it is also expected that $X_2 > X_1$, and this is generally the case.
Next, a quantity $C = X_2 - X_1$ is calculated.
From the theory underlying the assay, it only makes sense to report $C$ when $C > 0$.
Here is the core of the problem: the people who run the assay have decided that whenever $C$ is smaller than a given value, it is 'not reliable', i.e. they have set a threshold below which they don't consider $C$ 'sufficiently different from 0' to be reported, so in that case they report $C<C_{threshold}$.
At the moment the chosen threshold is $0.16$, and despite our inquiries, no coherent explanation could be obtained about how this value was decided.
What I and other people in the company would like to understand is: 1) whether it makes sense to set a threshold at all; 2) if so, whether the current $C_{threshold}$ is appropriate, given the data, or if it is too pessimistic and should be calculated differently.
Especially because important decisions are made based on $C$, and $C=0.02$ is 10 times better than $C=0.2$, so it is very damaging to 'mix together' in a big category of 'less than' distinct values of $C$ that perhaps the assay is in fact able to discriminate.
Suppose for instance that $\sigma_X = 0.01$, which is quite realistic, BTW.
The standard deviation of $C$, i.e. of the difference between two uncorrelated $X$ is $\sqrt 2 \cdot \sigma_X$, I believe.
Could one simply test use a z test to find the minimal $C>0$ that it is still significantly different from 0?
$$z = \frac {C-0} {\sqrt 2 \cdot \sigma_X} $$
at $\alpha=0.05$, two-tailed:
$$1.96 = \frac {C} {1.414 \cdot 0.01} $$
hence:
$$C_{threshold} = 0.028$$
So any experimentally determined value of $C$ smaller than $0.028$ would be reported as $C<0.028$.
What do you think?
Thanks
 A: Here is my take on this, what I think is going on. First of all I would like to point out that any test done on one observation (what you call "sample" $X_1$ and $X_2$) is very ambiguous and not reliable. Anyway, we need some assumptions:


*

*$X_1$ and $X_2$ come from two different normal distribution with the same common standard deviation $\sigma$

*The theoretical means $\mu_1$ and $\mu_2$ of each distribution are known


We know that the sum of two normal distributions is also normal, given by the following formula (we subtract $X_1$ from $X_2$ since that is also how we perform our test)
$$N\Big(\mu_2-\mu_1,\sqrt{\sigma_1^2+\sigma_2^2}\Big)=N\Big(\mu_2-\mu_1,\sqrt{2}\sigma\Big)$$
You do not provide the theoretical values of $\mu_1$ and $\mu_2$, so I am going to just make them up. So let's choose $\mu_1=0$ and $\mu_2=0.05$.
In this case our distribution looks like this

From this distribution we can obtain a two-sided or one-sided interval for the difference. We do this by estimating the percentiles, for ex. $5$-$th$ percentile for a one-sided lower interval.
So for example the one-sided lower interval at $\alpha=0.05$ would be
> qnorm(0.05,0.05,sqrt(2)*0.01)
[1] 0.02673826

Whereas a two-sided interval would be
> qnorm(c(0.025,0.975),0.05,sqrt(2)*0.01)
[1] 0.02228192 0.07771808

