Null hypothesis significance testing seems to be widely used in business. The most obvious example is A/B Testing, where a business will perform a test comparing two variants of some aspect of their business, the old and the new, and switch to the new if the test reveals a positive difference. As an MBA student, I have noticed that NHST seems to be the only approach taught to most business students.
I can't help but think that the question "is the difference between A and B statistically significant?" is sometimes very different from "should I choose B over A?", yet we are taught to use the former question to determine the proper answer to the latter. For example:
A statistically significant difference between two conditions does not mean that this difference is large enough to offset other factors. For example, the cost of the switch to a new variant itself might be greater than what the statistically significant difference between variants brings in new revenue.
In some cases, we may not have enough data to find a statistically significant difference, but it may still make sense to use what data we do have to inform a decision. For example, if it costs nothing to pick one variant or the other, it may make sense to pick the variant with a greater expected value, even if we can't say for sure if the difference is significant. The idea is that we aren't necessarily interested in being right 95% of the time, but in being right more often than we are wrong, or by a larger margin.
However, it seems common in practice to base a decision solely on whether or not the null hypothesis is rejected. Statistical testing seems to automatically give an aura of rigor to an analysis and is usually sufficient to support a conclusion, without much of a discussion around what the proper interpretation of a test should be in such a context. Those who recognize that NHST has some limitations in this regard will usually dismiss these limitations by saying that "it's better than nothing", but I feel that in some cases, unfortunately, it may not be.
My question is: Are these concerns legitimate; and when and how is NHST appropriate for use in business?