Type 1 and Type 2 Errors with Costs

I know the ex-ante cost of type 1 and type 2 errors in my study. How do I select my alpha, given that I know that alpha governs both type 1 and type 2 errors?

Suppose my null is that $$\beta=0$$ and my alternate is that $$\beta=10$$. I know that if the null is true and I accept the alternate i.e. type 1 error, through some policy choice, I will incur a cost of 100 dollars. If alternate is true but I make an error of selecting null and basing policy on null, I will incur a cost of 50 dollars. SD is same at 2 for both the cases i.e. when null is true or alternate is true.

$$\alpha$$ governs both type 1 and type 2. How should I optimally set it?

• What do you mean "select" alpha? You already stated that you know your type 1 error. And besides, alpha is chosen at the beginning and is fixed. Commented Aug 23, 2019 at 6:05
• @user2974951 I mean that I can select what width of CI interval to look at. I know the cost of the type one error. What alpha or cutoff should I use. Using smaller alpha reduces type I error but increases type 2 error, whose cost I also know. I am unable to write down a maximization problem for the optimal alpha. Commented Aug 23, 2019 at 21:34
• That's not how it works, you select your alpha at the beginning, this is fixed, for ex. 5 %. Then you estimate what your type 2 error / power will be based on this. You don't try to "optimize" type 1 and type 2. This could be considered p-hacking. Commented Aug 24, 2019 at 6:25
• @user2974951 I understand that it is bad practice. I am wondering if it can be done for self-study reasons. Commented Aug 24, 2019 at 8:46
• Well, in that case, yes it is possible. Can you post an example in your question? Commented Aug 25, 2019 at 13:40

I don't think this is really about Type 1 and 2 errors. This looks more like an optimization problem. Nevertheless, considering what we discussed in the comments, solving this is a simple problem.

What we need to do is, for every value of alpha (CDF of the null distribution in the upper tail) multiply it by 100, that is how much you will pay on average for type 1 errors. To this you add how much you will pay for type 2 at this point, which is the CDF / probability / p-value of the second distribution in the lower tail.

You can repeat this for many such points and find the minimum, that is minimize loss.

• Do I need a prior on with what probability is null hypothesis correct and with what probability is alternate hypothesis correct? Because, $\alpha$ is type 1 error only under null and similarly the blue region represents type 2 probability only true under alternate. So do I need the prior to write the optimization problem? Commented Aug 28, 2019 at 8:00
• @user52932 What do you mean by prior? You need the distribution of the two hypotheses, so mean and SD in case of a normal distribution, that is all. The hypotheses are clearly defined beforehand. Commented Aug 28, 2019 at 8:15
• I mean, that what is the correct loss function here? Isnt the correct loss function: (Probability that null is true) * (Probability of error under null true) * (cost of type 1 error) + (Probability that alternate is true) * (Probability of error under alternate true) * (cost of type 2 error). I know the cost. I think $\alpha$ controls (Probability of error under null true) and (Probability of error under alternate true). But don't I still need the outer probability i.e. which of the two distribution above happens? Commented Aug 28, 2019 at 8:20
• @user52932 There is no probability on the null or alternative being true, these two are fixed, you fix them at the start and you suppose they are true, no probability. If you had the probabilities of null alternative being true, why even do all this? You would choose the most likely ones and do your optimization on them? Commented Aug 28, 2019 at 8:32

It seems to me that you are finding this difficult because you are focussed on only two aspects where there are (at least) four that are important: sample size; effect size; false positive errors; and false negative errors. You might find the section 2.5 of this chapter helpful when thinking about experimental power and design: https://link.springer.com/chapter/10.1007/164_2019_286

A larger sample size will always yield a smaller error rate than a smaller sample size, but sample size rarely comes for free. That means that you need to account for the cost of data acquisition (time, money, effort, etc.) when trying to optimise the design of an experiment. Consider the fact that if the data are free then you can simultaneously minimise false positive error rates and maximise power by simply obtaining a massive dataset!

Effect size is equally important, but in some ways more nebulous. It is not really sufficient to say that a false negative error costs \$x, as false negative errors sometimes involve missing an important (i.e. large) effect and sometimes a trivially small effect. If you are interested mostly in finding large true effects then you can probably use a design with lower power by either designing a smaller sample or using a lower pre-set alpha value. It is common to input either the 'expected' effect size or the effect size that one would like to have designed power to avoid missing (or both) when doing a power analysis, but inspection of the full power vs effect size curve is more useful than the usual analysis where a single number for power or sample size is obtained. See the graphs in Figure 4 here: https://link.springer.com/chapter/10.1007/164_2019_286

If you just think about the error rates with all-or-none costings then you will generate an experimental design that is sub-optimal in the real-world considerations that should be primary.