# Reagent batch change, sample size, results significancy

We are working on a biological protocol measuring a patient's feature using a blood sample. This protocol has proven to have some variability (CV <= 10%).

During this process we sometime need to change a reagent batch. To be sure the new batch does not alter the results, we are running the protocol using the 2 reagent batches (current and new) on the same samples. We do collect the results in a spreadsheet.

So far we are using a sample size of 5 to 10 samples, with no good rational.

So far we have consider the reagent lot to be OK if the mean of the Coefficients of Variation of the N samples (current vs. new, sample by sample) to be < 10% (again, no good rational).

My first question : how can we calculate an optimal sample size that will ensure we can run with the new reagent. As the reagent are commercial and come with a CE mark quality certificate, the objective is to make sure the reagent is not bad (expired, exposed to a to low/high temperature during transportation/storage, etc.), not to make sure it has the exact same results as the current one.

Once we have collected the results for the 2 reagent for N samples:

data <- data.frame(
sample_id=c(1,2,3,4,5,6,7,8),
result1=c(10.83167, 17.96167, 34.97500, 37.21833, 23.19833, 29.56167, 36.32167, 40.11833),
result2=c(14.80000, 17.71333, 37.17833, 43.74500, 24.86500, 26.80500, 40.80667, 47.52667)
)


My second question: how can I know my second reagent batch is equivalent to the first one?

So far here is what we have done:

data <- data %>% rowwise() %>% mutate(mean=mean(c(result1, result2)), sd=sd(c(result1, result2)))
data$$cv <- (data$$sd/data$mean)*100  So we have data: str(data) Classes ‘rowwise_df’, ‘tbl_df’, ‘tbl’ and 'data.frame': 8 obs. of 6 variables: $$sample_id: num 1 2 3 4 5 6 7 8$$ result1 : num 10.8 18 35 37.2 23.2 ... $$result2 : num 14.8 17.7 37.2 43.7 24.9 ...$$ mean : num 12.8 17.8 36.1 40.5 24 ... $$sd : num 2.806 0.176 1.558 4.615 1.179 ...$$ cv : num 21.895 0.984 4.319 11.4 4.904 ...  We have tried: mean(data$cv)
8.82453442580761


and

t.test(data$$result1, data$$result2, conf.level = 0.90, paired = T)

Paired t-test

data:  data$$result1 and data$$result2
t = -2.4161, df = 7, p-value = 0.04636
alternative hypothesis: true difference in means is not equal to 0
90 percent confidence interval:
-5.1858973 -0.6274352
sample estimates:
mean of the differences
-2.906666


The standard deviation of the results using this protocol is expected to be 1.8.

But we are not sure how we can interpret these results.

My third question: how can I know I have done enough samples?

Once we have an answer to the equivalence between the 2 reagent batches (whatever the method), how can we make sure this result is strong/significant enough ?

We are using R for the statistical analyses.

Thanx in advance for any help.

• What's the difference between your first and third question? – StatsStudent Sep 2 at 9:44
• @StatsStudent : Question 1: how can we anticipate how many samples we should run ? Question 3 : now that we ran N samples, is the result significant ? – Olivier D. Sep 2 at 18:51

A German saying goes roughly like this: "the metall worker measures in tenth of a millimeter, a joiner measures in whole millimeters, the carpenter measures in centimeters and the brick layer - you're lucky if he stays within your real estate." Different trades/crafts require different levels of precision. A statistician will not be the one to tell you, which deviations in measurements are acceptable within your trade. As for blood samples I guess a 10% difference would often be just acceptable for blood sugar but certainly not for arterial blood pH.

You will have to define your acceptable deviation and whether you need a 95% chance of that being met or a sex-sigma chance, dependent on the impact a wrong measuremet might have.

Only after that CrossValidated and our advice come into play. You may for example use R's t.test function with paired = TRUE for paired measurements to obtain confidence intervals or use some Bayesian statistics. Using normal-normal conjugacy estimating the true mean of the differences and the expected normal distribution should be doable even within a spreadsheet. https://statswithr.github.io/book/bayesian-inference.html#three-conjugate-families https://statswithr.github.io/book/inference-and-decision-making-with-multiple-parameters.html#sec:normal-gamma

Edit: In your first comment on this answer you specified, that you want to go with R's t.test function and that an acceptable mean deviation is 5.4 on a 90% confidence level. Your call to t.test gave you a p value for an irrelevant null hypothesis, so do not care to much about that. It also gave you a confidence interval from -5.1859 to -0.6274. The confidence interval of a t-test is a good estimator for a credible interval (gained with a reasonable flat prior). We are not too far off to state, that the true difference between measuremens with the old an the new reagents lies in the [-5.2 ; -0.6] interval which does not include the acceptable mean deviation of +/- 5.4. Thus the true absolute deviation is smaller then the acceptable deviation.

Edit 2: This addendum was triggered by the comment starting with "Thanx @Bernhard for this extra explanation. The test is not cheap, ..." You have used the following function call: t.test(data$result1, data$result2, conf.level = 0.90, paired = T) A side not advice: Never use paired = T as it will stop working, once somebody enters T = 0 into your R session. Take the time and effort to write paired = TRUE. Now that is out of the way, this call performs a t test for a null hypothesis, that die true difference between the reagent is 0.000000000000000000000000000000000000000000000000000.... That is not how chemical analyses work so that obviously that is an irrelavant hypothesis . Nobody expect a difference to be perfectly zero. That is why I proposed to disregard the $$p$$-value of the null hypothesis altogether and concentrate on the confidence interval. Once you accept the idea, that a zero difference is not your goal, it is no longer of interest, whether zero is within the confidence interval.

However given a fixed sample size the t test can no longer detect arbitrarily small deviations from a given value. Power estimations and thereby sample size calculations depend on the concept of a null hypothesis test. For sample size computation an easy way to think about this is a one sided t test testing, whether the true difference is smaller than -5.4 and an additional one sided t test testing, whether the the difference is larger then 5.4. I do not recommend doing both these tests but one could use the idea for a sample size calculation employing the R function i refered to.

• Our acceptable deviation is 1*sigma to 3*sigma (sigma=1.8) ; so say 3*sigma=5.4 We are ok with 90 or 95% confidence. As you can read in the question, we have already done the t.test with paired=T. We need help in interpreting the results of this t.test, and help us understand if it is significant or not. Thanks, Olivier. – Olivier D. Sep 2 at 13:26
• Excellent answer, IMHO. But I'll add another when I wrap up some work here. +1 @OlivierD. – StatsStudent Sep 2 at 20:18
• OK @StatsStudent ; to be honest this answer didn't help much – Olivier D. Sep 3 at 11:59
• Well it helped as much as it made you specify an acceptable deviation. Once StatsStudent posts his answer he is likely to refer to that. Meanwhile, I made an addendum to my answer, hopefully expanding usefully with the directions you gave in the first comment. – Bernhard Sep 3 at 13:24
• @StatsStudent Thank you. Looking forward to read yours. – Bernhard Sep 3 at 13:25