# How to rigorously justify chosen false-positive/false-negative error rates and underlying cost ratio?

Context

A group of social scientists and statisticians (Benjamin et al., 2017) have recently suggested that the typical false-positive rate ($\alpha$ = .05) used as a threshold for determining "statistical significance" needs to adjusted to a more conservative threshold ($\alpha$ = .005). A competing group of social scientists and statisticians (Lakens et al., 2018) have responded, arguing against the use of this--or any other--arbitrarily selected threshold. The following is a quote from Lakens et al. (p. 16) that helps to exemplify the subject matter of my question:

Ideally, the alpha level is determined by comparing costs and benefits against a utility function using decision theory. This cost-benefit analysis (and thus the alpha level)differs when analyzing large existing datasets compared to collecting data from hard-to-obtain samples. Science is diverse, and it is up to scientists to justify the alpha level they decide to use. ... Research should be guided by principles of rigorous science, not by heuristics and arbitrary blanket thresholds.

Question

I'm wondering how one could go about justifying a chosen alpha in a way that is "guided by principles of rigorous science", as Lakens et al. suggest, in most social science contexts (i.e., outside of select cases where one has a more concrete quality, like profit, to optimize)?

Following the dissemination of Lakens et al., I have started seeing online calculators circulating to help researchers make this decision. When using them researchers need to specify a "cost ratio" of false-positive and false-negative errors. However, as this calculator here suggests, determining such a cost ratio can involve a lot of quantitative guess-work:

While some error costs are easy to quantiy in monetary terms (direct costs), others are difficult to put a dolar amount to (indirect costs). ...Despite being challenging to quantify, you should make an effort to put a number to them.

For example, though Lakens et al. suggest studying hard-to-reach samples as a factor one might consider in justifying alpha, it seems that one is still left guessing at how hard-to-reach that sample is, and thereby, how to adjust the selection of alpha accordingly. As another example, it would seem difficult to me to quantify the cost of publishing a false-positive, in terms of how much time/money others would subsequently commit to pursuing research premised on the mistaken inference.

If determining this cost ratio is largely a matter of subjective best-guess-making, I'm left wondering if these decisions can ever (again, outside of optimizing something like profit) be "justified". That is, in a way that exists outside of the assumptions made about sampling, trade-offs, impact, etc.,? In this way, determining a cost ratio of false-positive/false-negative errors seems, to me, to be something akin to selecting a prior in Bayesian inference--a decision that can be somewhat subjective, influence outcomes, and therefore debated--though I'm not sure that's a reasonable comparison.

Summary

To make my inquiry concrete:

1. Can false-positive/false-negative rates and their cost ratios ever be "rigorously" justified in most social science contexts?
2. If so, what are generalizable principles one could follow to justify these analytic choices (and maybe an example or two of them in action)
3. If not, is my analogy of the potential subjectivity in choosing cost ratios--as being akin to Bayesian prior selection--a reasonable one?

References

Benjamin, D. J., Berger, J., Johannesson, M., Nosek, B. A., Wagenmakers, E.,... Johnson, V. (2017, July 22). Redefine statistical significance. Retrieved from psyarxiv.com/mky9j

Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A.,... Zwaan, R. A. (2018, January 15). Justify Your Alpha. Retrieved from psyarxiv.com/9s3y6

• Can you define how you are using "objectively justified?" Type I error probabilities are a researcher preference... as are a priori type II error probabilities. In what way would a researcher "objectively justify," say, a preferred research agenda, preferred research collaborator or funder, or preferred research assistant training and mentoring approach? – Alexis Feb 20 '18 at 2:57
• Objectively justified as in more than a mere preference. The Lakens et al., paper is currently circulating under the short-hand "JYA" [Justify Your Alpha] title, and my read of their argument, based on the above quotations, is that not any old preference will do. To be clear: I'm not necessarily lodging the argument that one can objectively justify a chosen Type I/II error rate. Rather, my inquiry is premised on interpreting Lakens et al. to suggest that you can, and if that is the case, then I don't understand how one would do it. – jsakaluk Feb 20 '18 at 4:05
• I don't see the word "objectively" in this quote from Lakens et al. Do they really use it in their paper? If so, maybe you could add another quote to provide some more specific context? If not, then I am not sure you can say things like "objectively justifying a chosen alpha, as Lakens et al. suggest". – amoeba Feb 24 '18 at 15:15
• I've updated the post, which is now stripped of "objectivity". It wasn't my intent to mischaracterize the argument, but I can understand if readers thought I wrote carelessly. Lakens et al. do use the descriptor of "guided by principles of scientific rigor", so my question is now on surer footing. I'm still left wondering, however, what that can mean; how is the guesswork seemingly necessary any more rigorous than a heuristic? If it makes a difference, I'm especially curious how a scientific realist would reach a standard of "scientifically rigorous" justification for alpha. – jsakaluk Feb 25 '18 at 5:12
• Again, it's not my term, and part of my question is aimed at getting answers that can help me understand what that term could possibly mean--I ask for exemplar generalized principles--within the constraints of what Lakens et al say it does not mean (i.e., not a heuristic; not an arbitrary threshold). If you have multiple definitions of "scientifically rigourous justification" in mind, that could generate different $\alpha$s, and that fulfil the non-heuristic/non-aribitrary constraints, I would welcome reading them. – jsakaluk Feb 25 '18 at 19:16

(also posted on twitter, but re-posted here) My attempt at an answer: I don't think a justification can be "purely" objective, but it can be based on criteria which are defensible on rational/empirical grounds. I think RSS is an example of a way you could justify p <.005 for certain types of research, but I also think there are other circumstances where a different alpha would be more optimal than <.005 (either higher or lower) depending on what alpha is feasible and what the purpose of the study is. So for instance, if you have 5,000 participants and the smallest effect size of interest is .10, you may want to use p <.001 and have 90% power (numbers are all made up)In contrast, say you run a small experiment as initial “proof of concept” for line of research. You might have N = 100, p < .10, 90% power, then make conclusions based on internal meta-analysis of 4 experiments using p < .005.

I have been thinking about the same question a lot lately, and I’d guess many others in psychology are as well.

First off, each of your questions relates to whether a choice is made objectively versus subjectively, but (as others here have noted) you haven’t fully explained what would constitute (in your view) an objective versus subjective choice.

You may be interested in the Gelman & Hennig 2015 paper that unpacks a variety of values wrapped up in common usage of the “objective” and “subjective” labels in science. In their formulation, “objective” relates to values of transparency, consensus, impartiality, and correspondence to observable reality, whereas “subjective” relates to values of multiple perspectives and context-dependence.

Related to your Question 3, in the Bayesian view, probability is defined as quantifying uncertainty about the world. From what I understand, there is a tension apparent across “subjectivist Bayesian” (probabilities reflect individual states of belief) and “objectivist Bayesian” schools of thought (probabilities reflect consensus plausibility). Within the objectivist school, there is a stronger emphasis on the justification of the prior distribution (and the model more generally) in a transparent way that comports with consensus and that can be checked, but the choice of model is certainly context-dependent (i.e., depends on the state of consensus knowledge for a particular problem).

In the frequentist conception, probabilities reflect the number of times an event will occur given infinite independent replications. Within the Neyman-Pearson framework, one stipulates a precise alternative hypothesis and a precise alpha, accepts the precise null or the precise alternative (that the population effect is exactly equal to the one stipulated) on the basis of the data, and then reports the long-run frequency of doing so in error.

Within this framework, we rarely have a precise point estimate of the population effect size but rather a range of plausible values. Therefore, conditional on a given alpha, we don’t have a precise estimate of the Type 2 error rate, but rather a range of plausible Type 2 error rates. Similarly, I’d agree with your general point that we typically do not have a precise sense of what the costs and benefits of either a Type 1 error or a Type 2 error will actually be. Meaning we are often faced with a situation where we have very incomplete information about what our hypothesis should be in the first place, and even less information about what would be the relative costs and benefits of accepting vs rejecting this hypothesis.

1. Can false-positive/false-negative rates and their cost ratios ever be objectively justified in most social science contexts?

I think so, in that a justification can be transparent, can comport with consensus, can be impartial, and can correspond to reality (to the extent that we are using the best available information we have about costs and benefits).

However, I think that such justifications are also subjective, in that there can be multiple valid perspectives regarding how to set alpha for a given problem, and in that what constitutes an appropriate alpha can be meaningfully context-dependent.

For example, in recent years, it has become clear that many effects in the literature reflect Type M or Type S errors. They may also reflect Type 1 errors, to the extent that a replication study is able to provide evidence for the null of exactly zero effect.

Related to this observation, there is an emerging consensus that the p-value threshold for a claim with certainty should be kept the same or made more stringent (i.e., no one is arguing for a blanket increase of alpha to .10 or .20). Similarly, there is an emerging consensus that p values should not be used as a criterion for publication (e.g., the Registered Report format).

To me, this reflects a kind of “objective” source of information — i.e., to my reading there is a growing consensus that false claims are costly to the field (even if we can’t put a dollar amount on these costs). To my reading, there is no clear consensus that failing to meet a p-value threshold is a dramatic cost to the field. If there are costs, they may be mitigated if failing to meet a p-value threshold doesn’t impact whether the estimate makes it into a published paper.

1. If so, what are generalizable principles one could follow to justify these analytic choices (and maybe an example or two of them in action)

I am not sure, but I would lean toward some kind of principle that the decisions should be made on the basis of transparent (local or global) consensus judgements about the costs and benefits of different kinds of analytic choices in a particular context, even in the face of woefully incomplete information about what these costs and benefits might be.

1. If not, is my analogy of the potential subjectivity in choosing cost ratios--as being akin to Bayesian prior selection--a reasonable one?

Yes, across frequentist and Bayesian traditions, there is room for subjectivity (i.e., multiple perspectives and context-dependence) as well as objectivity (i.e., transparency, consensus, impartiality, and correspondence to observable reality) in many different aspects of a statistical model and how that model is used (the chosen prior, the chosen likelihood, the chosen decision threshold, etc.).

• This is a nice answer. One piece that I'm not so certain about is the claim of correspondence. If we understand this term in the same way (I'm thinking in terms of the correspondence theory of truth), then actually, it sounds like correspondence might be on shaky ground if we don't have a precise idea of the costs of Type I/II errors. Instead, it sounds like there is a better claim to coherence (given these initial assumptions, the rest of the numbers "make sense"), or pragmatism (our guess of Type I/II error costs is a useful fiction for study planning). – jsakaluk Feb 22 '18 at 17:57
• Perhaps I'm trying too hard to marry "justification" to a correspondence/realist perspective, and in these other ways of understanding, Type I/II error rates can be chosen in a way that is "justified"? – jsakaluk Feb 22 '18 at 17:58
• Thanks for pointing me to these ideas. I'd say that in any given context, we may have good information about what future costs and benefits are likely to be, or we may have very poor information. In a very rough sense, there is growing consensus that false positives (p<threshold, true effect is exactly zero) can be more harmful to the field than instances of failing to meet a significance threshold (but publishing the estimate anyway). In specific local circumstances, there may be more serious costs associated with failing to meet a significance threshold. – sourdough Feb 22 '18 at 19:09
• Tangentially, the concepts of “alpha” and “Type 2 error” exist only in the NP framework where the analyst has specified two precise hypotheses and commits to accepting one or the other at the end of the procedure. However, in common practice, analysts are often warned not to accept the null on the basis of a nonsignificant estimate with uncertain power, essentially reverting to a Fisher-style interpretation where there is no accepting the null and no “Type 2 error”. – sourdough Feb 22 '18 at 19:09
• It's amusing to me that the social networks, social class, and social interactions behind your description of "consensus" are somehow divorced from the subjective beliefs and values that underlie all of them. – Alexis Feb 25 '18 at 0:42