Let’s suppose the following:
We are statistician, machine learner, data analyst, and so on, working for a company which produces tires for trucks. The company produces tires per hour with the following parameters:
$\mu = 80 $ tires per hour
$\sigma = 8$
Now the company introduced a new process, and they want to know whether this one really works or is a fraud. If the company adopts the new process, they have to invest thousands of dollars.
I make a hypothesis test to know if it’s worth the investment.
I draw a sample of 25 tires (with the new process), and I do the hypothesis test with significance level of 0.05. The mean with the new process is $\mu= 83$
I do the hypothesis test:
Acording to normal table, our p-value is 0.031. So we have approx. 3.1% probability to make type I error. Therefore, we say to the company that they should invest thousands of dollars in the new technique.
My question is: what happens if we had bad luck and by chance we take the most extreme sample, and in reality the new technique to produce tires doesn't work?
In the real life what would you do? Would you take more random samples and the media of all those samples?
In case you have just one opportunity and you can't take more samples: do you lower the significance level to 0.01?
What other statistics techniques or machine learning techniques would you use?