Which would be better, a statistical method that yields false positive errors on 5% of occasions and false negative on 20%, or a statistical method that yields 10% false positives but only 5% false negatives?

The answer, of course, has to begin with “It depends…”. But what factors is it dependent upon, and how are those factors taken into account in real world application of hypothesis testing?

I would say that it depends upon some kind of utility function that takes into account circumstances and consequences of decisions, but I have never noticed any specification or discussion of such a function in research papers in my area of basic pharmacology, and I assume that they are similarly absent from research papers from many other areas of science. Does that matter?

It would be safe to assume that researchers are responsible for the experimental design and analysis in most of the research papers that I read, but at least sometimes a statistician will be consulted (usually after the data are in hand). Do statisticians discuss loss functions with researchers before advising on or performing a data analysis, or do they just use one that is unconsidered and implicit?


My answer is from the perspective of a researcher and statistician, or some weird mixture of both.

There may be two assumptions, that are at least misleading. The first one is that "better" can be defined within statistics. And the second one is that there is a numerical expression you can evaluate.

What is assumed to be better depends always on the circumstances. For example some medical tests for disease are more or less completely disregarding false positives, but ensure that there are next to none false negatives. E.g. a quick test in a medical emergency. The second test might be the other way round -- quickly reduce the number of subjects which were false positives in the first round.

It is important to distinguish between the math -- the method to extract numbers -- and their interpretation. You have to be knowledgeable in statistics, in order to not draw the wrong conclusion. But what to do with a certain conclusion is no longer part of the math.

Regarding the accepted procedures, there are agreements in the different scientific communities. These are permanently discussed and revised. There might be certain $\chi^2$ limits for medical and pharmacological studies, that need to be met. And they are different for other fields.

It can be emphasized that the methodological discussion is independent of the topical discussion. In order to call a certain result e.g. an "observation" or a "discovery" must be decided independent of the matter.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.