The following excerpt is from the entry, What are the differences between one-tailed and two-tailed tests?, on UCLA's statistics help site.
... consider the consequences of missing an effect in the other direction. Imagine you have developed a new drug that you believe is an improvement over an existing drug. You wish to maximize your ability to detect the improvement, so you opt for a one-tailed test. In doing so, you fail to test for the possibility that the new drug is less effective than the existing drug.
After learning the absolute basics of hypothesis testing and getting to the part about one vs two tailed tests... I understand the basic math and increased detection ability of one tailed tests, etc... But I just can't wrap around my head around one thing... What's the point? I'm really failing to understand why you should split your alpha between the two extremes when your is sample result can only be in one or the other, or neither.
Take the example scenario from the quoted text above. How could you possibly "fail to test" for a result in the opposite direction? You have your sample mean. You have your population mean. Simple arithmetic tells you which is higher. What is there to test, or fail to test, in the opposite direction? What's stopping you just starting from scratch with the opposite hypothesis if you clearly see that the sample mean is way off in the other direction?
Another quote from the same page:
Choosing a one-tailed test after running a two-tailed test that failed to reject the null hypothesis is not appropriate, no matter how "close" to significant the two-tailed test was.
I assume this also applies to switching the polarity of your one-tailed test. But how is this "doctored" result any less valid than if you had simply chosen the correct one-tailed test in the first place?
Clearly I am missing a big part of the picture here. It all just seems too arbitrary. Which it is, I guess, in the sense that what denotes "statistically significant" - 95%, 99%, 99.9%... Is arbitrary to begin with.