When is NHST appropriate in business? Null hypothesis significance testing seems to be widely used in business. The most obvious example is A/B Testing, where a business will perform a test comparing two variants of some aspect of their business, the old and the new, and switch to the new if the test reveals a positive difference. As an MBA student, I have noticed that NHST seems to be the only approach taught to most business students.
I can't help but think that the question "is the difference between A and B statistically significant?" is sometimes very different from "should I choose B over A?", yet we are taught to use the former question to determine the proper answer to the latter. For example:


*

*A statistically significant difference between two conditions does not mean that this difference is large enough to offset other factors. For example, the cost of the switch to a new variant itself might be greater than what the statistically significant difference between variants brings in new revenue.

*In some cases, we may not have enough data to find a statistically significant difference, but it may still make sense to use what data we do have to inform a decision. For example, if it costs nothing to pick one variant or the other, it may make sense to pick the variant with a greater expected value, even if we can't say for sure if the difference is significant. The idea is that we aren't necessarily interested in being right 95% of the time, but in being right more often than we are wrong, or by a larger margin.
However, it seems common in practice to base a decision solely on whether or not the null hypothesis is rejected. Statistical testing seems to automatically give an aura of rigor to an analysis and is usually sufficient to support a conclusion, without much of a discussion around what the proper interpretation of a test should be in such a context. Those who recognize that NHST has some limitations in this regard will usually dismiss these limitations by saying that "it's better than nothing", but I feel that in some cases, unfortunately, it may not be.
My question is: Are these concerns legitimate; and when and how is NHST appropriate for use in business?
 A: Your concerns are completely legitimate.
1) This gets to the effect size. Given enough data, we can detect tiny differences. However, all we are concluding from getting FROM $\mu_X - \mu_Y = 0.0000001\ne0$ is that the difference between the two means is not zero. While we may be very confident in that conclusion, we reserve the right to say, "Yeah, but we aren't interested in a difference that small."
2) I disagree here. The population with the lower mean may exhibit a higher sample mean. In fact, I will give you an R simulation showing how common this is.
set.seed(2019)
V <- rep(NA,2500)
for (i in 1:length(V)){
    x <- rnorm(25,0,1)
    y <- rnorm(25,0.25,1)
    V[i] <- mean(x) - mean(y)
}
length(V[V>0])/length(V)*100

We expect the entries of V to be less than zero, since the population mean of $Y$ is greater than the population mean of $X$ (0.25 versus 0). However, this simulation gives me $\bar{x} > \bar{y}$ in 19.08% of the cases. In other words, your plan would result in the worse approach being implemented about a fifth of the time. Perhaps you are willing to take that risk, and there are quantitative ways to defend this stance, but you should be aware that it is common.
A: The concerns are legitimate but I do not think that statistical testing is the end all-be all of decisions in actual business practice. Statistics, correctly used, merely helps frame the information that is available for decision making. As already mentioned, effect-size is a good way of visualizing what it is that you are actually buying when you make a decision to change because you rejected the null hypothesis. Deciding whether or not it is worth it at this point is something that may no longer involve statistics.
A: I will respond to this both as an economist/econometrician and as a decades-long business professional in the private sector.
1) As another answer noted, we should separate in our minds "statistical significance" from "economic significance" (the "size" aspect of the situation).
2) Statistical significance depends on the "amount of risk of being wrong one is willing to take". The established scientific tradition is to be "as conservative and guarded as possible" against claims that a significant difference does exist. This is reflected in the standard 1%, 5%, 10% "significance levels" one is taught to use when running significance testing. But businesses may very well want/decide to accept much more of such risk, so you can go ahead an run significance testing at a any level of significance you choose, say, 40%.
3) Classical statistics and expected values are more suited to decisions about mid-term  / repetitive situations. In businesses, we have to make many short-term/one-off decisions. Then, the Baeysian approach to inference may be more well suited, and also, it may make more sense to consider the "most likely" outcomes rather than "expected values".
4) Cost considerations should of course enter a bushiness decision. This boils down to model correctly your loss/payoff function. In traditional statistical testing, the loss/payoff function is implicitly assumed to be symmetric around "zero-difference", because the object of science is accuracy itself, and so the direction of inaccuracy does not matter. But in economic activity such symmetries in costs/payoffs are rarely the case. See my two answers in this post as well as this post.
