Let us assume I have developed two designs (A, B) of a product and I want to see which one performs better in terms of sales. So I run an experiment where some customers can buy A whilst others can buy B. This gives me two means (e. g. the A-customers bought on average 5x, the B-customers 7x) with which I perform a t-test. Let us further assume that the consequences of a type I error and a type II error are as follows:

  • type I error: I change the design of the product but (contrary to what I believe) this has absolutely no effect on sales.

  • type II error: I do not change the product's design although I should have done so because it would have earned me some additional cash, say $100 in each of the upcoming months.

Under the assumption that changing the manufacturing process to produce the new design creates no additional costs, the type II error is obviously more critical than the type I error. So my inclination would be to set the significance level $\alpha$ higher than the usual $\alpha=0.05$ and certainly higher than $\beta$ (which of course I have to control via $n$ and the effect size). But by how much? What would be the optimal ratio $q=\beta/\alpha$?

EDIT: @whuber You seem to think that my question is unclear in a general sense or needs a data background, but the exact same question is also discussed elsewhere, see for example Which Statistical Error Is Worse: Type 1 or Type 2? (albeit without the concrete business context I'm offering here).

2. EDIT: Here is a study that tackles the above problem: Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests.


closed as unclear what you're asking by whuber Sep 29 '17 at 19:38

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    $\begingroup$ Is there enough information given here to make this a reasonable question? If your alternatives are between having no effect on sales and possibly having a positive effect, with no costs for either option, then by all means choose the latter! You don't need any data or statistical analysis to make that decision. But what about the possibility that after changing the product's design it earns you less profit? $\endgroup$ – whuber Sep 29 '17 at 19:33
  • $\begingroup$ My question clearly is not about choosing between two options as your comment indicates. It is about how to chose the test parameters. $\endgroup$ – Joe Sep 29 '17 at 19:38
  • $\begingroup$ Then you need to provide definitions of what you mean, because the usual meanings of "Type I" and "Type II" errors are associated with decisions that are made based on the data. $\endgroup$ – whuber Sep 29 '17 at 19:38
  • 1
    $\begingroup$ Normally reasearches are advised to chose $\alpha=0.05$ and $\beta = 0.2$ which amounts to $q=4$ (e. g. andrews.edu/~calkins/math/edrm611/edrm11.htm). This might be a good choice in research fields where falsely rejecting the null is worse than falsely sticking to it. But in the above case $q=4$ is obviously not well chosen. My question is how to chose $q$ there. $\endgroup$ – Joe Sep 29 '17 at 19:48
  • $\begingroup$ In a very clearly optimizing the expected profit setting like you describe hypothesis testing may be the wrong approach. A decision theoretic approach would seem more logical and decision boundaries may not correspond to any fixed traditional error rates. $\endgroup$ – Björn Sep 30 '17 at 6:04