0
$\begingroup$

In Googling this question, I see that there are a variety of similar tests but I couldn't find anything given the exact way I'm approaching this problem. This might be something obvious but I'm not sure.

Here's the problem:

Let's say that I have two normally distributed populations, A and B. For population A, the mean is known with absolute precision and the standard deviation can be determined from some number of samples (but not the mean - bear with me on that). For population B, the standard deviation will be derived from the same samples as population A, but the mean is completely unknown.

What I'm trying to figure out is the likelihood that population B is greater than population A given only one sample from population B and the other information I know of both populations.

What I'm trying to come up with is an alternative to the softmax function for classifying the likelihood that some Y value exceeds a threshold using a neural network. The threshold will never change, so I'm looking at it as the mean for population A, which has absolute precision. Obviously, standard deviation needs to be considered for both populations, but I'm making a subjective calculation, which is outside the scope of this question. It's sufficient to say that I can estimate the standard deviation through some number of samples.

If I only label the Y values as 1 or 0 based on whether the threshold is exceeded or not, I'm missing additional information that can be gained from the true Y values. Rather than training the network using softmax or sparse categorial crossentropy, I could independently calculate probabilities of each category (above or below the threshold) given the observation, the threshold, and the presumed standard deviation and then train the network based on something like mean absolute error.

I'm hoping someone can derive a formula and would be interested to know if there's already anything similar to this that can be used for training neural networks.

$\endgroup$
1
  • $\begingroup$ What do you mean that the standard deviation can be determined from some number of samples? Do you mean that the standard deviation is estimated as usual? Do you mean that the standard deviation is calculated exactly? (I don’t see how that could happen, but your setup is unusual, so I want to keep an open mind.) $\endgroup$
    – Dave
    Commented Jan 10, 2023 at 5:16

1 Answer 1

0
$\begingroup$

I believe this is the answer. Hopefully, someone more knowledgeable can confirm.

What I was looking for was a one-tailed p-value derived from a two-sample t-test with the assumption of equal variances.

I found this calculator: http://www.usablestats.com/calcs/2samplet&summary=1

And this more detailed explanation: https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm

I was able to recreate the results from the calculator in the first link by modifying the code here: https://stackoverflow.com/questions/17559897/python-p-value-from-t-statistic, with the following modifications:

  • Changed the t_statistic denominator calculation to the correct denominator for the equal variances test
  • Using the signed t_statistic rather than the absolute and not multiplying by 2 since it's a one-tailed test
  • Updating the second sf parameter to be the combined degrees of freedom
  • Returning 1 - pval rather than the pval

This is the python code I'm using to calculate the likelihood of exceeding the threshold where y is the observation:

def get_t_prob(self, y, threshold, std, std_count):
    variance = std**2
    t_statistic = (y-threshold)/np.sqrt(variance/std_count + variance)
    pval = stats.t.sf(t_statistic, std_count - 1)
    return 1 - pval

Again, this makes sense to me, but if anyone can verify that it's correct, that would be great.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.