Return to Answer

Edit 2: This addendum was triggered by the comment starting with "Thanx @Bernhard for this extra explanation. The test is not cheap, ..." You have used the following function call: t.test(data$result1, data$result2, conf.level = 0.90, paired = T) A side not advice: Never use paired = T as it will stop working, once somebody enters T = 0 into your R session. Take the time and effort to write paired = TRUE. Now that is out of the way, this call performs a t test for a null hypothesis, that die true difference between the reagent is 0.000000000000000000000000000000000000000000000000000.... That is not how chemical analyses work so that obviously that is an irrelavant hypothesis . Nobody expect a difference to be perfectly zero. That is why I proposed to disregard the $p$-value of the null hypothesis altogether and concentrate on the confidence interval. Once you accept the idea, that a zero difference is not your goal, it is no longer of interest, whether zero is within the confidence interval.

However given a fixed sample size the t test can no longer detect arbitrarily small deviations from a given value. Power estimations and thereby sample size calculations depend on the concept of a null hypothesis test. For sample size computation an easy way to think about this is a one sided t test testing, whether the true difference is smaller than -5.4 and an additional one sided t test testing, whether the the difference is larger then 5.4. I do not recommend doing both these tests but one could use the idea for a sample size calculation employing the R function i refered to.

A German saying goes roughly like this: "the metall worker measures in tenth of a millimeter, a joiner measures in whole millimeters, the carpenter measures in centimeters and the brick layer - you're lucky if he stays within your real estate." Different trades/crafts require different levels of precision. A statistician will not be the one to tell you, which deviations in measurements are acceptable within your trade. As for blood samples I guess a 10% difference would often be just acceptable for blood sugar but certainly not for arterial blood pH.

You will have to define your acceptable deviation and whether you need a 95% chance of that being met or a sex-sigma chance, dependent on the impact a wrong measuremet might have.

Only after that CrossValidated and our advice come into play. You may for example use R's t.test function with paired = TRUE for paired measurements to obtain confidence intervals or use some Bayesian statistics. Using normal-normal conjugacy estimating the true mean of the differences and the expected normal distribution should be doable even within a spreadsheet. https://statswithr.github.io/book/bayesian-inference.html#three-conjugate-families https://statswithr.github.io/book/inference-and-decision-making-with-multiple-parameters.html#sec:normal-gamma

Edit: In your first comment on this answer you specified, that you want to go with R's t.test function and that an acceptable mean deviation is 5.4 on a 90% confidence level. Your call to t.test gave you a p value for an irrelevant null hypothesis, so do not care to much about that. It also gave you a confidence interval from -5.1859 to -0.6274. The confidence interval of a t-test is a good estimator for a credible interval (gained with a reasonable flat prior). We are not too far off to state, that the true difference between measuremens with the old an the new reagents lies in the [-5.2 ; -0.6] interval which does not include the acceptable mean deviation of +/- 5.4.

A German saying goes roughly like this: "the metall worker measures in tenth of a millimeter, a joiner measures in whole millimeters, the carpenter measures in centimeters and the brick layer - you're lucky if he stays within your real estate." Different trades/crafts require different levels of precision. A statistician will not be the one to tell you, which deviations in measurements are acceptable within your trade. As for blood samples I guess a 10% difference would often be just acceptable for blood sugar but certainly not for arterial blood pH.

You will have to define your acceptable deviation and whether you need a 95% chance of that being met or a sex-sigma chance, dependent on the impact a wrong measuremet might have.

Only after that CrossValidated and our advice come into play. You may for example use R's t.test function with paired = TRUE for paired measurements to obtain confidence intervals or use some Bayesian statistics. Using normal-normal conjugacy estimating the true mean of the differences and the expected normal distribution should be doable even within a spreadsheet. https://statswithr.github.io/book/bayesian-inference.html#three-conjugate-families https://statswithr.github.io/book/inference-and-decision-making-with-multiple-parameters.html#sec:normal-gamma

Edit: In your first comment on this answer you specified, that you want to go with R's t.test function and that an acceptable mean deviation is 5.4 on a 90% confidence level. Your call to t.test gave you a p value for an irrelevant null hypothesis, so do not care to much about that. It also gave you a confidence interval from -5.1859 to -0.6274. The confidence interval of a t-test is a good estimator for a credible interval (gained with a reasonable flat prior). We are not too far off to state, that the true difference between measuremens with the old an the new reagents lies in the [-5.2 ; -0.6] interval which does not include the acceptable mean deviation of +/- 5.4. Thus the true absolute deviation is smaller then the acceptable deviation.

A German saying goes roughly like this: "the metall worker measures in tenth of a millimeter, a joiner measures in whole millimeters, the carpenter measures in centimeters and the brick layer - you're lucky if he stays within your real estate." Different trades/crafts require different levels of precision. A statistician will not be the one to tell you, which deviations in measurements are acceptable within your trade. As for blood samples I guess a 10% difference would often be just acceptable for blood sugar but certainly not for arterial blood pH.

You will have to define your acceptable deviation and whether you need a 95% chance of that being met or a sex-sigma chance, dependent on the impact a wrong measuremet might have.

Only after that CrossValidated and our advice come into play. You may for example use R's t.test function with paired = TRUE for paired measurements to obtain confidence intervals or use some Bayesian statistics. Using normal-normal conjugacy estimating the true mean of the differences and the expected normal distribution should be doable even within a spreadsheet. https://statswithr.github.io/book/bayesian-inference.html#three-conjugate-families https://statswithr.github.io/book/inference-and-decision-making-with-multiple-parameters.html#sec:normal-gamma

A German saying goes roughly like this: "the metall worker measures in tenth of a millimeter, a joiner measures in whole millimeters, the carpenter measures in centimeters and the brick layer - you're lucky if he stays within your real estate." Different trades/crafts require different levels of precision. A statistician will not be the one to tell you, which deviations in measurements are acceptable within your trade. As for blood samples I guess a 10% difference would often be just acceptable for blood sugar but certainly not for arterial blood pH.

You will have to define your acceptable deviation and whether you need a 95% chance of that being met or a sex-sigma chance, dependent on the impact a wrong measuremet might have.

Only after that CrossValidated and our advice come into play. You may for example use R's t.test function with paired = TRUE for paired measurements to obtain confidence intervals or use some Bayesian statistics. Using normal-normal conjugacy estimating the true mean of the differences and the expected normal distribution should be doable even within a spreadsheet. https://statswithr.github.io/book/bayesian-inference.html#three-conjugate-families https://statswithr.github.io/book/inference-and-decision-making-with-multiple-parameters.html#sec:normal-gamma

Edit: In your first comment on this answer you specified, that you want to go with R's t.test function and that an acceptable mean deviation is 5.4 on a 90% confidence level. Your call to t.test gave you a p value for an irrelevant null hypothesis, so do not care to much about that. It also gave you a confidence interval from -5.1859 to -0.6274. The confidence interval of a t-test is a good estimator for a credible interval (gained with a reasonable flat prior). We are not too far off to state, that the true difference between measuremens with the old an the new reagents lies in the [-5.2 ; -0.6] interval which does not include the acceptable mean deviation of +/- 5.4.