I have an experiment that produces one measurement every time it is run. I can change something in the experimental setup, and I want to test if this change in the setup results in a change of the measured value. I am repeating the measurement multiple times for each setup, but only one measurement with one setup can be done at a time. I assume that the change in the setup is not changing the standard deviation of the measurements (but I do not know what the deviation is), and that the measurements in both setups are normally distributed around a true value.
My hypothesis is that the true values for the two processes will be different, and I would like to test this hypothesis. However, having gathered the data and found the means to be different, I am unsure how much confidence I should assign to the result. After all, the mean values could be different by chance and the true values actually identical.
What is the best way to evaluate this?
Calculating the standard deviation of the mean seems like a good start. The further the two means +/- their standard deviations are apart, the more confident I can be. But I would like to make this a bit more quantitative.
My idea is that I need to calculate the probability of getting two different mean values if I perform two times N measurements. The lower the probality, the more confidence I can have in my result. But how do I do this calculation? To me this looks like a textbook problem where I just have to know the right formula, but apparently I haven't taken the corresponding course.
(Note: no quantum physics here, everything is classical)