Best method of quantifying probability of new datum belonging to either of two distanced normal distributions?

Question

I have two samples A and B from two separate normally distributed populations. The population mean of B is higher than that of A, but both are unknown. My aim is to find a threshold value between the two distributions such that a new datum can be assigned to A if its value falls below this central point, and to B if it lies above, with a certain level of accuracy. I would like to obtain the percentage chance of a datum from population B being incorrectly assigned to A and vice versa.

I have solved for the point at which the Z-scores of the two samples are equal, and found the percentage that lie above and below this point for both distributions. As I am not terribly well-versed in statistics, I am left wondering if this is the correct approach. For example, whether Student's T-distribution should instead be used, or another method altogether.

I have provided the below information, for the sake of interest.

$$ \bar{A} = 103.72, s_A = 8.62 \\ \bar{B} = 161.17, s_B = 13.62 $$

Answer 1

Your question is a bit vague and it seems the your figure does not quite match the rest of the problem. I think you may have put parts of two similar problems together in your Question. I'll do my best to give most of the information you requested.

You say the means of the two normal populations are unknown with $\mu_A \le \mu_B,$ and I will assume the two population standard deviations are also unknown. If it is somehow known that the two population standard deviations are equal, $\sigma_A = \sigma_B,$ then a pooled 2-sample t test of $H_0: \mu_A = \mu_B$ against $H_1: \mu_A < \mu_B$ is appropriate.

I would use your example with values for the two sample means and standard deviations, but I would need to know the two sample sizes in order to show how to do the test. So I will use data with somewhat similar sample means and standard deviations, and with sample sizes $n_A = n_B = 40,$ as sampled in R below:

set.seed(2020)
x.a = rnorm(40, 104, 10)
x.b = rnorm(40, 160, 10)
summary(x.a);  length(x.a);  sd(x.a)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  73.61  100.93  106.45  105.76  113.37  128.35 
[1] 40
[1] 12.00162
summary(x.b);  length(x.b);  sd(x.b)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  142.2   154.1   160.7   160.2   165.1   192.0 
[1] 40
[1] 9.79959

stripchart(list(x.a, x.b), pch="|", ylim=c(.5, 2.5))

From the summaries and the stripchart, we can see that all values of sample A are below all values of sample B. There is a complete separation of the two samples.

With such complete separation, there is little doubt that the pooled t test will reject the null hypothesis. [The parameter var.eq=T calls for the pooled test; without it, R does a Welch two-sample t test when two samples are provided.]

t.test(x.a, x.b, alt="less", var.eq=T)

        Two Sample t-test

data:  x.a and x.b
t = -22.228, df = 78, p-value < 2.2e-16
alternative hypothesis: true difference in means is less than 0
95 percent confidence interval:
      -Inf -50.37798
sample estimates:
mean of x mean of y 
 105.7579  160.2139 

You can find the formulas for doing a pooled two-sample t test in a basic statistics text. Maybe you should find the formulas and use the sample sizes, means and standard deviations to compute the pooled variance estimate, often called $s_p^2$ and then the test statistic $T = 22.228.$

If you choose to do the test at the significance level $\alpha = 1\%$ then the critical value $c = 2.429$ of the test can be found from a printed table of Student's t distributions on the row for degrees of freedom $DF = n_A + n_B - 2 = 38$ or by using software as below.

qt(.99, 38)
[1] 2.428568

You asked for a value that separates the two distributions. Such a value is $c$ and there are probability $0.01$ of rejecting $H_0$ when it is true. Because the two distributions are so widely separated the probability of failing to reject $H_0$ when it is false is very small. This means that we reject the null hypothesis at the 1% level because $T =22.23 > 2.429.$

[If you know about P-values, the very small P-value (below 1%) is another indication to reject $H_0.$ Ordinarily, you can't get exact P-values from printed distribution tables.]

Note: If the distributions were as in the figure you show, then you might choose the critical value to be $c = 1.5$ Then if you were to rely on a single observation to decide between A and B, The probability that an observation from A would fall above $c$ is $0.0668,$ which could be found by standardizing and using printed tables of the standard normal cumulative distribution function. This probability can be found using R (where pnorm is a normal CDF).

1 - pnorm(1.5, 0, 1)
[1] 0.0668072

Similarly, or by symmetry, the probability that a single observation from B would fall below $c$ is the same.

pnorm(1.5, 3, 1)
[1] 0.0668072

---

Addendum, per Comment. Your intuition that it is important to take variability into account is correct. Here is output from a recent release of Minitab, which explicitly shows the pooled standard deviation. First, I use the summarized data in your Question, and assume both samples are of size 20.

Two-Sample T-Test and CI 

Sample   N    Mean  StDev  SE Mean
1       20  103.72   8.62      1.9
2       20   161.2   13.6      3.0

Difference = μ (1) - μ (2)
Estimate for difference:  -57.45
95% upper bound for difference:  -51.37
T-Test of difference = 0 (vs <): 
  T-Value = -15.94  P-Value = 0.000  DF = 38
Both use Pooled StDev = 11.3976

Now, to illustrate the role variability plays, I multiply the sample standard deviations by 10, which amounts to multiplying the variances by 100, and keep the sample sizes the same. [Of course these are no longer real data, but we can pretend.]

The effect is to make the denominator of the $T$-statistic larger, so that the statistic itself is smaller. Now the P-value is $0.06 > 0.05,$ so the null hypothesis is not rejected at the 5% level.

Two-Sample T-Test and CI 

                            SE
Sample   N   Mean  StDev  Mean
1       20  103.7   86.2    19
2       20    161    136    30

Difference = μ (1) - μ (2)
Estimate for difference:  -57.4
95% upper bound for difference:  3.3
T-Test of difference = 0 (vs <): 
  T-Value = -1.59  P-Value = 0.060  DF = 38
Both use Pooled StDev = 113.9756

Answer 2

Here the aim "is to find a threshold value between the two distributions such that a new datum can be assigned to $A$ if its value falls below this central point, and to $B$ if it lies above, with a certain level of accuracy".

Suppose we measure the accuracy as (probability of wrong assignment for data in $A$) + (probability of wrong assignment for data in $B$).

Then we are looking for a threshold value $t$ to minimize

$$P[A>t\ |\ A\sim N(m_A,s_A)] + P[B<t\ |\ B\sim N(m_B,s_B)]$$

The derivative of this with respect to $t$ should be 0:

$$\frac{-e^{-(t-m_A)^2/(2s_A^2)}}{\sqrt{2\pi} s_A} +\frac{e^{-(t-m_B)^2/(2s_B^2)}}{\sqrt{2\pi} s_B} = 0$$

This can be solved analytically with some algebra and the quadratic formula: $$(t-m_A)^2/(2s_A^2) + \ln s_A= (t-m_B)^2/(2s_B^2) + \ln s_B$$ $$t = \frac{b\pm\sqrt{b^2-ac}}{a},\text{ where}$$ $$a=\frac{1}{s_A^2}-\frac{1}{s_B^2},\ \ b=\frac{m_A}{s_A^2}-\frac{m_B}{s_B^2},\ \ c=\frac{m_A^2}{s_A^2}-\frac{m_B^2}{s_B^2}+\ln\left(\frac{s_A^2}{s_B^2}\right)$$

For the particular numerical values in the question, this gives $a=0.00807$, $b=0.527$, $c=3.84$, and $t=126.9$ as the option in between $m_A$ and $m_B$. The measure of accuracy is $0.95\%$.

For other ways of measuring accuracy we would get other values of $t$; this is one way of getting a reasonable value.