1
$\begingroup$

I have 11 subjects and also have the percentage value for the accuracy of a method I developed to predict future actions for each individual. I want to use a right tail one-sample t-test to find out the minimum accuracy of the method for the population from the limited information I have from 11 subjects. I carry out right tail one-sample t-tests for null hypothesis mean value of 60%,65%,70%,75%,80% (alternate hypothesis is the mean is greater than the mean in null hypothesis) and evaluate for which mean value p become less than 0.01. One of my colleagues told me it is not the right way of doing things as false positives can accumulate and suggested to use Bonferroni correction. In Bonferroni correction the new significance level is 0.01/no. of tests. So my new alpha=0.01/5=0.002. However if I do t-tests for 60%,70%,80% then my new alpha=0.01/3=0.0033. This does not feel right. It looks like I can just select the way I divide my range to satisfy the p value. Is this stupid way of doing things. How will you do it? Will appreciate any help.

$\endgroup$
1
$\begingroup$

Your question is quite confusing so let me state it with some mathematical notation.

You have 11 samples from some random variable $X$. You want to test the hypothesis that $\bar{x}>\mu$.

You should formulate your null hypothesis before you start to analyse the data but you haven't, you haven't chosen a specific value of $\mu$. Instead you want to test if $\bar{x}<60\%$ and if the result is not significant you will test if $\bar{x}<65\%$, and so on until you find what value of $\mu$ gives a statistically significant result.

You're not actually testing any hypothesis, you are finding the cut-off value of the percentage so that the test of any hypothesized mean above that would have resulted in a rejecting that hypothesis.

Bonferoni or any other type of multiple-comparison correction is not used for this type of thing. I think it's beyond the scope of this question to explain when multiple comparison corrections are used.

$\endgroup$
  • $\begingroup$ Yeah I don't have specific value of $\mu$ since this method is new and there aren't any method with which it could be compared. Also if I am carrying out right tail one-sample t-test should not I be testing $\bar{x} >60\%$ and then $\bar{x}>65\%$ and so on? Since I am not testing hypothesis but rather testing for the cut-off value of the percentage can it be said that the statistical significance of the results cannot be stated? $\endgroup$ – hellfragger Sep 20 '16 at 21:11
  • $\begingroup$ Yes I mixed up the inequality sign. That's right, the statistical significance of the results cannot be stated. If you wanted to get a value for the significance you would have to use half of your samples for finding the cut-off value and use the other half to test the significance of that cut off value in the way a normal t-test is done. $\endgroup$ – Hugh Sep 20 '16 at 21:17
1
$\begingroup$

There are two possibilities as I see it. None of them have recourse to multiple testing. Your willingness to do corrections for multiple testing shows your honesty, your intent not to do p-hacking, but there are better ways.

Firstly, as you are speaking of accuracy, is there some baseline accuracy that a random model would reach? For example, if you know that the action to predict will in 60% of the cases be option A and in 30% of cases option B and in 10% of cases option C, then 0.6 should be your baseline accuracy. A "model" that doesn't use any information other than knowing option A is the most prevalent could always predict option A and be 60% accurate. In this case have

  • $H_0 \quad\mu=0.6$
  • $H_a \quad\mu\neq 0.6$

and do a bilateral test to see if your model does anything else than what a no information model could predict. The test should be bilateral because the risk of having models that are worse than no information models, models that do active harm instead of being just useless, is real.

Secondly, if there is absolutely no way to determine a baseline to compare performance to, don't do hypothesis testing. Having $H_0 \quad\mu = 0$ would not be an acceptable baseline in that case since models that are absolutely always wrong are as difficult to construct as models that are absolutely always right.

In this case, limit yourself to exploratory analysis: Construct a confidence interval around your parameter $\hat{\mu}$ and leave it at that.

You should also double check if you meet the assumptions of t-tests.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.