# Hypothesis testing - Newbie blockers - Update and more

Brief : I'm from manufacturing industry, a processing machine in our production line used to do pressing, polishing and QA one after the other. Now we have a new machine that will perform these at the same time. Ideally the new machine will produce units in less time than the old machine. I want to prove that time taken by new machine is significantly less than old machine.

Null hypothesis - There is no significance difference in time taken by both machines to produce one unit.

Alternate hypothesis - Time taken by new machine is less than old machine.

Initial Plan : I initially planned on performing bootstrapping to identify the population distribution. Assuming the data was normally distributed from bootstrap, I planned on Two sample t-test, else Mann Whitney U-test. There is also some extreme outliers ~2% in a data of 50K or more records. I thought of removing these outliers completely as they are less than 5%.

Questions : My problems are, (1) during research I came across normality test (Shapiro-Wilk) which I though might help to statistically confirm the normality. (2) Then I ran into proportion tests for sample size, which is also being recommended. (3) Then came across (winsorized mean), for replacing outliers with non-outliers.

With just a regular (trust me, am not over researching) research, I'm flooded with over information, which is quite confusing. What should be the ideal framework for my use case. What would you all recommend that I do, correct or refer???

I planned on performing all the steps I as per my plan. But, now having second thoughts if that will be correct. Any advice, feedback, corrections will be really helpful.

Update:

Thank you all so much for inputs. I was able to research and decide the trade-offs.

Let me summarize what I've done and require your suggestions and input again for a particular blocker in sample size.

1. Data 1 size = 10 million records
2. Data 2 size = 1 million records
3. population distribution (for both) = log normal distribution: right skewed. It almost looks like needle and a reallllyyyyyy long tail.
4. S.D and variance for both populations are different.
5. Outlier - used IQR method (Q1 - 1.35 * IQR and Q3 + 1.35 * IQR) to trim outliers.
6. Converted data to log 10 and achieved normal distribution
7. performed two sample t-test with n=30 and 5% alpha, rejected the null hypothesis.

New blocker: I read two books (Practical statistics for Data scientists by Peter Bruce and Andrew Bruce and Statistics by Robert S Witte and John S Witte) and Data camp course to learn hypothesis test. In all of them, they had always assumed sample size, likewise I just assumed a sample size of 30 and performed the test. Now I ran into (Power analysis) and other tools to calculate the sample size. My doubt is

1. Is it necessary to calculate the sample size?
2. What other essentials like these am I missing out on (I've included every step of my test process above)?
3. Any recommended steps/procedures/tests that you would suggest before I start the hypothesis test?
• (1) How many data points do you have? (2) Note that Mann-Whitney does not test equality of means, but something quite different. If you do want to test equality of means and are concerned about any lack of normality (and with large enough sample sizes, this is much less of an issue), a bootstrap or permutation test would be more appropriate. (Sorry for proposing yet another method...) Commented Oct 29, 2023 at 7:57
• (3) Can you edit your post to include a sample or all of your data? Commented Oct 29, 2023 at 8:02
• Words like "significance" or "significant" do not belong in hypotheses. Those are words that would go in a conclusion, if you use them at all (the ASA suggested not). Commented Oct 29, 2023 at 9:08
• You can't prove anything with statistics (which about quantifying uncertainty). What about an alternative framework where you estimate the time to produce units, with the first machine and with the second, with sufficient precision? Commented Oct 29, 2023 at 13:02
• Failure to reject H0 in a goodness of fit test like the Shapiro-Wilk does not demonstrate that the null is true, only that your sample size was too small to reject. The relevant consideration is not really whether you can detect non-normality, but its impact on your inference. The probability of the former increases with sample size, but important parts of the latter reduce with sample size, so you're more likely to reject normality in cases where it matters less, and to fail to reject where undetected large deviations from normality matter more. Many answers on site address this issue. Commented Oct 29, 2023 at 22:51

The answer is, as always, it depends. Statistics is a lot less about 'when X do Y' and a lot more about making trade-offs between assumptions that may or may not even be verifiable or come back to bite you.

The Mann-Whitney U will give you a test for your hypothesis, i.e. a sufficiently extreme test statistic is evidence that one of the two samples "stochastically dominates" the other. Specifically, you reject the hypothesis that if you pick a given rank from each of your samples, it's equally likely that one value is bigger than the other or vice versa. The great thing about this is that it's relatively free from assumptions, a trade-off is that you don't get any information about how big the difference really is because everything happens in the rank scale.

A t-test directly compares the mean of two samples and thereby comes with a few additional assumptions, an important one being that the mean is an appropriate summary statistic (through homogeneity of variance and normally distributed [symmetric] residuals). The big element in your favor is that you have a sample size of 50,000 if I understand well, which is very likely enough that you can rely on the central limit theorem. There's no fixed number of samples where this is or isn't appropriate, but the idea is that your sample mean will always be normally distributed even if your sample isn't, and it will commonly be relied upon in much smaller samples than yours.

Actively testing for normality is not useful because such tests have very low power for small samples (where the CLT may not hold), and have very high power for large samples in that they will reject even the smallest deviation from the theoretical normal distribution (whereas you can rely on the CLT at that point). I'm also not sure why you would bootstrap your sample to check its distribution, can you not just plot a histogram or some other empirical (cumulative) distribution curve?

To summarize the choice of test: the parametric t-test comes with advantages (more power, actual parameters that describe your samples) at the cost of additional assumptions, but I wouldn't worry too much about the latter for the reasons above. At these sample sizes I would expect the MWU test to have very high power as well, so more likely than not they will be consistent. A trade-off of this high power is that you may end up calling very small differences statistically significant, so always keep an eye on the real-world impact of the numbers you get (e.g. is it worth replacing all machines to save 5 seconds on a 3 hour production time?).

Finally, let's address outliers. It's not clear how you established that 2% of your data consists of 'extreme outliers', very often this means that they are outside some sort of distributional assumption which may or may not even be appropriate. As a counterexample, what if both machines are equally fast when they work, but one of them is less reliable and when they break down the production time is much longer? What if it is exactly these data points that you're calling outliers? You throw them out, see no difference in the remaining data, and declare both machines equal - a conclusion that is only true when the machines are not breaking down. It's always a good idea to investigate extreme values and check if there is perhaps a technical explanation (for example, production time should be in minutes but this one was entered in years by accident). In absence of one I would not throw them out mindlessly. The same applies to winsorization, this will force additional distributional assumptions onto your data (are these correct?) and will hurt the generalizability of your findings.

• Update, Thank you all very much for the detailed answer. I did a lot of research referred few books and was able decide trade-offs in terms of test selection, outlier handling, normality test and others. On surface it didn't seem too vast. But. man it is too vast.
– AKK
Commented Nov 3, 2023 at 14:26