How do you make sure the hypothesis test really works?

Question

Let’s suppose the following:

We are statistician, machine learner, data analyst, and so on, working for a company which produces tires for trucks. The company produces tires per hour with the following parameters:

$\mu = 80 $ tires per hour

$\sigma = 8$

Now the company introduced a new process, and they want to know whether this one really works or is a fraud. If the company adopts the new process, they have to invest thousands of dollars.

I make a hypothesis test to know if it’s worth the investment.

I draw a sample of 25 tires (with the new process), and I do the hypothesis test with significance level of 0.05. The mean with the new process is $\mu= 83$

$\alpha =0.05$

I do the hypothesis test:

$\frac{83-80}{\frac{8}{\sqrt{25}}}= 1.875$

Acording to normal table, our p-value is 0.031. So we have approx. 3.1% probability to make type I error. Therefore, we say to the company that they should invest thousands of dollars in the new technique.

My question is: what happens if we had bad luck and by chance we take the most extreme sample, and in reality the new technique to produce tires doesn't work?

In the real life what would you do? Would you take more random samples and the media of all those samples?

In case you have just one opportunity and you can't take more samples: do you lower the significance level to 0.01?

What other statistics techniques or machine learning techniques would you use?

Answer 1

It may help to further define "the new process works" and "fraud". Maybe 83 tires/hr is not very different from 80 tires/hr. Maybe its only worth it if the difference is 15 or something. Similarly, I would say the process is a "fraud" if the rate was much lower than 80. You can setup an equivalence bound and do two one-sided test for equivalence to establish equivalence of the two processes.

You can also look into utility theory rather than null hypothesis testing. You can answer questions like how much do we need to make the rate increase by to ensure we recoup the loss in investment after X days.

Answer 2

My question is: what happens if we had bad luck and by chance we take the most extreme sample, and in reality the new technique to produce tires doesn't work?

It can happen, you can be unlucky. The bad news is that not much can be done about verifying it. To be more precise, you can collect more data to minimize the chance of bad luck. However, if you could collect more data, why not do this in the first place?

The good news is that hypothesis tests are designed for handling such cases by themselves. This is why before the experiment, you set the $\alpha$ and $\beta$ levels (see, e.g. Banerjee et al, 2009), so to decide what are the acceptable risks for the type I (false positive) and type II (false negative) errors (see type-i-and-ii-errors). Before the experiment, you need to conduct power analysis (see power-analysis) and decide on the appropriate sample size. As you can see, we are back at collecting enough data. In each case, there still is a risk that you get incorrect results, but this is how you bound the risk to acceptable levels.

Answer 3

There's nothing you can do. That's why p-values come in the $[0,1]$ range rather than being just either $0$ or $1$. If you want to do hypothesis testing, you need to set a critical level beforehand as your threshold for what you accept or not. If you want to accuse someone of committing fraud, you'll probably need to set a p-value much lower from the start.