An industrial process is in place that increases the strength of a metal component. We are tuning a couple of settings on the system to optimize the strength. There are already some settings in place, but I have found some new settings that I would like to make a recommendation to change the system to. The new settings that I found to be an improvement were found using a sample that I tested different settings on. I would like to perform a hypothesis test to see if my recommended changes makes an improvement to the system. My question is whether I will be able to do a one sample hypothesis test, or if I must do a two sample hypothesis test.
Method 1: If I do a one-sample hypothesis test, I would use the average strength of the component from the previous sample under the current settings as the null hypothesis and the average strength of the metal component from the new sample under my new recommended settings.
Method 2: If I do a two-sample hypothesis test, I would run one new sample under the old parameter settings and the other sample under the new settings and test the difference in average strength to determine if there was a significant improvement.
I fear that if I use a one-sample test in this way (method 1), that I am introducing some sort of bias into the test. Which method is best and why?