A group of social scientists and statisticians (Benjamin et al., 2017) have recently suggested that the typical false-positive rate ($\alpha$ = .05) used as a threshold for determining "statistical significance" needs to adjusted to a more conservative threshold ($\alpha$ = .005). A competing group of social scientists and statisticians (Lakens et al., 2018) have responded, arguing against the use of this--or any other--arbitrarily selected threshold. The following is a quote from Lakens et al. (p. 16) that helps to exemplify the subject matter of my question:
Ideally, the alpha level is determined by comparing costs and benefits against a utility function using decision theory. This cost-benefit analysis (and thus the alpha level)differs when analyzing large existing datasets compared to collecting data from hard-to-obtain samples. Science is diverse, and it is up to scientists to justify the alpha level they decide to use. ... Research should be guided by principles of rigorous science, not by heuristics and arbitrary blanket thresholds.
I'm wondering how one could go about justifying a chosen alpha in a way that is "guided by principles of rigorous science", as Lakens et al. suggest, in most social science contexts (i.e., outside of select cases where one has a more concrete quality, like profit, to optimize)?
Following the dissemination of Lakens et al., I have started seeing online calculators circulating to help researchers make this decision. When using them researchers need to specify a "cost ratio" of false-positive and false-negative errors. However, as this calculator here suggests, determining such a cost ratio can involve a lot of quantitative guess-work:
While some error costs are easy to quantiy in monetary terms (direct costs), others are difficult to put a dolar amount to (indirect costs). ...Despite being challenging to quantify, you should make an effort to put a number to them.
For example, though Lakens et al. suggest studying hard-to-reach samples as a factor one might consider in justifying alpha, it seems that one is still left guessing at how hard-to-reach that sample is, and thereby, how to adjust the selection of alpha accordingly. As another example, it would seem difficult to me to quantify the cost of publishing a false-positive, in terms of how much time/money others would subsequently commit to pursuing research premised on the mistaken inference.
If determining this cost ratio is largely a matter of subjective best-guess-making, I'm left wondering if these decisions can ever (again, outside of optimizing something like profit) be "justified". That is, in a way that exists outside of the assumptions made about sampling, trade-offs, impact, etc.,? In this way, determining a cost ratio of false-positive/false-negative errors seems, to me, to be something akin to selecting a prior in Bayesian inference--a decision that can be somewhat subjective, influence outcomes, and therefore debated--though I'm not sure that's a reasonable comparison.
To make my inquiry concrete:
- Can false-positive/false-negative rates and their cost ratios ever be "rigorously" justified in most social science contexts?
- If so, what are generalizable principles one could follow to justify these analytic choices (and maybe an example or two of them in action)
- If not, is my analogy of the potential subjectivity in choosing cost ratios--as being akin to Bayesian prior selection--a reasonable one?
Benjamin, D. J., Berger, J., Johannesson, M., Nosek, B. A., Wagenmakers, E.,... Johnson, V. (2017, July 22). Redefine statistical significance. Retrieved from psyarxiv.com/mky9j
Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A.,... Zwaan, R. A. (2018, January 15). Justify Your Alpha. Retrieved from psyarxiv.com/9s3y6