14
$\begingroup$

Null hypothesis significance testing seems to be widely used in business. The most obvious example is A/B Testing, where a business will perform a test comparing two variants of some aspect of their business, the old and the new, and switch to the new if the test reveals a positive difference. As an MBA student, I have noticed that NHST seems to be the only approach taught to most business students.

I can't help but think that the question "is the difference between A and B statistically significant?" is sometimes very different from "should I choose B over A?", yet we are taught to use the former question to determine the proper answer to the latter. For example:

  1. A statistically significant difference between two conditions does not mean that this difference is large enough to offset other factors. For example, the cost of the switch to a new variant itself might be greater than what the statistically significant difference between variants brings in new revenue.

  2. In some cases, we may not have enough data to find a statistically significant difference, but it may still make sense to use what data we do have to inform a decision. For example, if it costs nothing to pick one variant or the other, it may make sense to pick the variant with a greater expected value, even if we can't say for sure if the difference is significant. The idea is that we aren't necessarily interested in being right 95% of the time, but in being right more often than we are wrong, or by a larger margin.

However, it seems common in practice to base a decision solely on whether or not the null hypothesis is rejected. Statistical testing seems to automatically give an aura of rigor to an analysis and is usually sufficient to support a conclusion, without much of a discussion around what the proper interpretation of a test should be in such a context. Those who recognize that NHST has some limitations in this regard will usually dismiss these limitations by saying that "it's better than nothing", but I feel that in some cases, unfortunately, it may not be.

My question is: Are these concerns legitimate; and when and how is NHST appropriate for use in business?

|cite|improve this question
$\endgroup$
  • 3
    $\begingroup$ I was about to type a long answer to this, but I think this summarizes it: never, in my experience. Most business decisions that have to be made are too complicated to be summarized through one procedure (NHST included), and I question decision-making processes that are done solely through one metric. $\endgroup$ – Clarinetist Dec 3 '19 at 5:16
  • $\begingroup$ You are assuming things are better in science? en.wikipedia.org/wiki/Replication_crisis $\endgroup$ – James Dec 3 '19 at 17:08
  • $\begingroup$ @James I'm aware of the replication crisis and I'm not saying things are worse in business than in science, but even if we momentarily ignore how good or bad things are in each field, my question is about whether or not the goals and the proper methods to use are the same in business and in science. $\endgroup$ – Vincent B. Lortie Dec 3 '19 at 19:49
  • $\begingroup$ @Clarinetist could you clarify: would you then argue that NHST is not "better than nothing"? $\endgroup$ – Vincent B. Lortie Dec 3 '19 at 19:50
  • $\begingroup$ @VincentB.Lortie If you can externalise the ill-effects of doing the wrong thing, bad methods can be very useful. In both business and science the short-term games played to win power and influence can be won by stratgies that give sub-optimal results from the wider point of view. $\endgroup$ – James Dec 4 '19 at 11:47
1
$\begingroup$

I will respond to this both as an economist/econometrician and as a decades-long business professional in the private sector.

1) As another answer noted, we should separate in our minds "statistical significance" from "economic significance" (the "size" aspect of the situation).

2) Statistical significance depends on the "amount of risk of being wrong one is willing to take". The established scientific tradition is to be "as conservative and guarded as possible" against claims that a significant difference does exist. This is reflected in the standard 1%, 5%, 10% "significance levels" one is taught to use when running significance testing. But businesses may very well want/decide to accept much more of such risk, so you can go ahead an run significance testing at a any level of significance you choose, say, 40%.

3) Classical statistics and expected values are more suited to decisions about mid-term / repetitive situations. In businesses, we have to make many short-term/one-off decisions. Then, the Baeysian approach to inference may be more well suited, and also, it may make more sense to consider the "most likely" outcomes rather than "expected values".

4) Cost considerations should of course enter a bushiness decision. This boils down to model correctly your loss/payoff function. In traditional statistical testing, the loss/payoff function is implicitly assumed to be symmetric around "zero-difference", because the object of science is accuracy itself, and so the direction of inaccuracy does not matter. But in economic activity such symmetries in costs/payoffs are rarely the case. See my two answers in this post as well as this post.

|cite|improve this answer
$\endgroup$
14
$\begingroup$

Your concerns are completely legitimate.

1) This gets to the effect size. Given enough data, we can detect tiny differences. However, all we are concluding from getting FROM $\mu_X - \mu_Y = 0.0000001\ne0$ is that the difference between the two means is not zero. While we may be very confident in that conclusion, we reserve the right to say, "Yeah, but we aren't interested in a difference that small."

2) I disagree here. The population with the lower mean may exhibit a higher sample mean. In fact, I will give you an R simulation showing how common this is.

set.seed(2019)
V <- rep(NA,2500)
for (i in 1:length(V)){
    x <- rnorm(25,0,1)
    y <- rnorm(25,0.25,1)
    V[i] <- mean(x) - mean(y)
}
length(V[V>0])/length(V)*100

We expect the entries of V to be less than zero, since the population mean of $Y$ is greater than the population mean of $X$ (0.25 versus 0). However, this simulation gives me $\bar{x} > \bar{y}$ in 19.08% of the cases. In other words, your plan would result in the worse approach being implemented about a fifth of the time. Perhaps you are willing to take that risk, and there are quantitative ways to defend this stance, but you should be aware that it is common.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ For 2, you're right that the sample means might not be properly ordered, but in your example you are still right 81% of the time, which in business may be way more than good enough. I'm oversimplifying here, but if you win 1\$ 80% of the time and lose 1\$ 20% of the time, you're still winning overall, no? $\endgroup$ – Vincent B. Lortie Dec 2 '19 at 19:12
  • 1
    $\begingroup$ @VincentB.Lortie There are ways to defend such a stance (decision theory). Just be aware that you'll be wrong some amount of the time. Also, my code is just one example. You can fiddle with the sample sizes, other tests (maybe variance), and effect size to see that the ratio can get close to a coin flip as to which decision is right. Do better than "Well, boss, let's just flip a coin." $\endgroup$ – Dave Dec 2 '19 at 19:19
  • $\begingroup$ You've helped me articulate part of what I think is wrong: being wrong some amount of the time is natural in business, but when it comes to AB tests, for example, you're flipping that coin anyway, except you're systematically betting tails (control) unless a 95% procedure tells you the coin is biased towards heads (variation). If the first 2 flips turn up heads and you have reason to think the coin may not be fair, it seems silly to keep betting tails until you have more data. $\endgroup$ – Vincent B. Lortie Dec 2 '19 at 19:36
  • 3
    $\begingroup$ Being wrong some amount of the time is the design of statistics. When we say we're testing at the $\alpha=0.05$-level, we are accepting that we will go with a wrong alternative hypothesis 5% of the time. (In fact, if it's less than 5%, our testing methodology is under-powered.) The subject that studies trade-offs between ways of being wrong (false negative versus false positive) is called decision theory. $\endgroup$ – Dave Dec 2 '19 at 19:42
  • 4
    $\begingroup$ Probably everything should be informed by decision theory. Even t-testing could be informed by decision theory. Choosing $\alpha=0.05$ is a typical approach, but there's nothing special about $5\%$. That value is picked as a trade-off between false negatives and false positives. However, if you want to have exceptional power and are willing to accept being wrong 19% of the time, feel free to pick $\alpha=0.19$ and test at that level. If you want to get into decision theory in detail, I suggest scanning CV and the internet in general and posting a new question when an issue arises. $\endgroup$ – Dave Dec 2 '19 at 19:51
1
$\begingroup$

The concerns are legitimate but I do not think that statistical testing is the end all-be all of decisions in actual business practice. Statistics, correctly used, merely helps frame the information that is available for decision making. As already mentioned, effect-size is a good way of visualizing what it is that you are actually buying when you make a decision to change because you rejected the null hypothesis. Deciding whether or not it is worth it at this point is something that may no longer involve statistics.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.