# how to set an upper threshold for a property that is calculated from experimental results

I can't really place this problem theoretically, and I wonder if anyone could please point me to relevant posts/literature or provide some advice.

A property $$X$$ of a sample can be measured using some instrument.
From historical data, it is known that if the measurement of the same sample is repeated many times, the distribution of the values of $$X$$ is approximately Gaussian, with mean $$X_{mean}$$ and standard deviation $$\sigma_X$$, where $$\sigma_X$$ is ~independent from $$X_{mean}$$ and ~constant over time.

In day-to-day activities, an in vitro assay is run by taking two separate samples and measuring $$X$$ for each, thus obtaining $$X_1$$ and $$X_2$$.
I suppose they can't really be called 'means', but given the above, we believe that they come from distributions with the same standard deviation $$\sigma_X$$.
From the assay setup it is also expected that $$X_2 > X_1$$, and this is generally the case.

Next, a quantity $$C = X_2 - X_1$$ is calculated.
From the theory underlying the assay, it only makes sense to report $$C$$ when $$C > 0$$.

Here is the core of the problem: the people who run the assay have decided that whenever $$C$$ is smaller than a given value, it is 'not reliable', i.e. they have set a threshold below which they don't consider $$C$$ 'sufficiently different from 0' to be reported, so in that case they report $$C.
At the moment the chosen threshold is $$0.16$$, and despite our inquiries, no coherent explanation could be obtained about how this value was decided.

What I and other people in the company would like to understand is: 1) whether it makes sense to set a threshold at all; 2) if so, whether the current $$C_{threshold}$$ is appropriate, given the data, or if it is too pessimistic and should be calculated differently.
Especially because important decisions are made based on $$C$$, and $$C=0.02$$ is 10 times better than $$C=0.2$$, so it is very damaging to 'mix together' in a big category of 'less than' distinct values of $$C$$ that perhaps the assay is in fact able to discriminate.

Suppose for instance that $$\sigma_X = 0.01$$, which is quite realistic, BTW.
The standard deviation of $$C$$, i.e. of the difference between two uncorrelated $$X$$ is $$\sqrt 2 \cdot \sigma_X$$, I believe.

Could one simply test use a z test to find the minimal $$C>0$$ that it is still significantly different from 0?

$$z = \frac {C-0} {\sqrt 2 \cdot \sigma_X}$$

at $$\alpha=0.05$$, two-tailed:

$$1.96 = \frac {C} {1.414 \cdot 0.01}$$

hence:

$$C_{threshold} = 0.028$$

So any experimentally determined value of $$C$$ smaller than $$0.028$$ would be reported as $$C<0.028$$.

What do you think?

Thanks

• I don't understand, you say you take two samples $X_1$ and $X_2$ from a process. You state that $X$ is normally distributed but then say that is it expected that $X_2$ should have a higher mean than $X_1$, why is that? If $X$ is truly normally distributed this is not true. Nevertheless, if you are comparing two means from two samples which you hypothesize come from different distributions, then you can calculate at which point you cannot determine a statistically significant difference anymore Feb 19, 2019 at 8:20
• Yes, $X_1$ and $X_2$ do come from different distributions in the sense that the true means of those distribution are in theory different (although their SD is the same). So yes, my point is indeed to understand if and how one should determine a threshold for the smallest statistically significant difference that can be reported. Feb 20, 2019 at 9:15

Here is my take on this, what I think is going on. First of all I would like to point out that any test done on one observation (what you call "sample" $$X_1$$ and $$X_2$$) is very ambiguous and not reliable. Anyway, we need some assumptions:

1. $$X_1$$ and $$X_2$$ come from two different normal distribution with the same common standard deviation $$\sigma$$
2. The theoretical means $$\mu_1$$ and $$\mu_2$$ of each distribution are known

We know that the sum of two normal distributions is also normal, given by the following formula (we subtract $$X_1$$ from $$X_2$$ since that is also how we perform our test)

$$N\Big(\mu_2-\mu_1,\sqrt{\sigma_1^2+\sigma_2^2}\Big)=N\Big(\mu_2-\mu_1,\sqrt{2}\sigma\Big)$$

You do not provide the theoretical values of $$\mu_1$$ and $$\mu_2$$, so I am going to just make them up. So let's choose $$\mu_1=0$$ and $$\mu_2=0.05$$.

In this case our distribution looks like this

From this distribution we can obtain a two-sided or one-sided interval for the difference. We do this by estimating the percentiles, for ex. $$5$$-$$th$$ percentile for a one-sided lower interval.

So for example the one-sided lower interval at $$\alpha=0.05$$ would be

> qnorm(0.05,0.05,sqrt(2)*0.01)
[1] 0.02673826


Whereas a two-sided interval would be

> qnorm(c(0.025,0.975),0.05,sqrt(2)*0.01)
[1] 0.02228192 0.07771808

• That's interesting, thanks. I agree, it's not ideal to take only 1 measurement per sample, but unfortunately that is what is done in practice. I did not provide $\mu_1$ or $\mu_2$ because my aim was to find the smallest $\mu_1 - \mu_2$ that 'makes sense' to report. Very pragmatically, if I have $X_1 = 2$ and $X_2 = 3$, then reporting $C=1$ seems OK, because the uncertainty on $C$ is perhaps small compared to $C$ itself; but what if I have $X_1 = 2.9$ and $X_2 = 3$? Is the uncertainty on this estimate of the true $\mu_1 - \mu_2$ still small enough? Where does one draw the line? Feb 20, 2019 at 12:12
• @user6376297 I don't understand. Feb 21, 2019 at 8:28
• OK, I will make some examples. You showed that qnorm(c(0.025,0.975),0.05,sqrt(2)*0.01) -> 0.02228192 0.07771808. From the description of the qnorm function, I take this to mean that a normally distributed random variable with true mean 0.05 and sd sqrt(2)*0.01 will produce, when sampled a large number of times, values that fall about 95% of the time between 0.022 and 0.078. Feb 21, 2019 at 10:53
• Now: qnorm(c(0.025,0.975),0.01,sqrt(2)*0.01) -> -0.01771808 0.03771808. We know that if a single sample is taken, one has no choice but to report that value as the best estimate of the variable's true mean. Knowing that it doesn't make sense to have a true mean less than 0, would any experimentally found value be acceptable? Would you report 0.05 as the best estimate of the true mean? Would you report 0.01? What if you measured -0.02? Do you think there should be a threshold T below which one can only say 'true mean < T' rather than 'true mean = measured value'? Feb 21, 2019 at 10:58
• Sorry I can't see how to explain this differently, it is an actual real world problem, not a theoretical one, and this is the situation we are facing. I suspect we may have to use a Bayesian approach, because we keep talking about the distribution of the experimental value given a fixed true mean, but in reality we only know that we found a single, fixed experimental value, and we want to know the distribution of the true mean. Feb 21, 2019 at 11:01