Reagent batch change, sample size, results significancy

Question

We are working on a biological protocol measuring a patient's feature using a blood sample. This protocol has proven to have some variability (CV <= 10%).

During this process we sometime need to change a reagent batch. To be sure the new batch does not alter the results, we are running the protocol using the 2 reagent batches (current and new) on the same samples. We do collect the results in a spreadsheet.

So far we are using a sample size of 5 to 10 samples, with no good rational.

So far we have consider the reagent lot to be OK if the mean of the Coefficients of Variation of the N samples (current vs. new, sample by sample) to be < 10% (again, no good rational).

My first question : how can we calculate an optimal sample size that will ensure we can run with the new reagent. As the reagent are commercial and come with a CE mark quality certificate, the objective is to make sure the reagent is not bad (expired, exposed to a to low/high temperature during transportation/storage, etc.), not to make sure it has the exact same results as the current one.

Once we have collected the results for the 2 reagent for N samples:

data <- data.frame(
    sample_id=c(1,2,3,4,5,6,7,8),
    result1=c(10.83167, 17.96167, 34.97500, 37.21833, 23.19833, 29.56167, 36.32167, 40.11833),
    result2=c(14.80000, 17.71333, 37.17833, 43.74500, 24.86500, 26.80500, 40.80667, 47.52667)
    )

My second question: how can I know my second reagent batch is equivalent to the first one?

So far here is what we have done:

data <- data %>% rowwise() %>% mutate(mean=mean(c(result1, result2)), sd=sd(c(result1, result2)))
data$cv <- (data$sd/data$mean)*100

So we have data:

str(data)
Classes ‘rowwise_df’, ‘tbl_df’, ‘tbl’ and 'data.frame': 8 obs. of  6 variables:
 $ sample_id: num  1 2 3 4 5 6 7 8
 $ result1  : num  10.8 18 35 37.2 23.2 ...
 $ result2  : num  14.8 17.7 37.2 43.7 24.9 ...
 $ mean     : num  12.8 17.8 36.1 40.5 24 ...
 $ sd       : num  2.806 0.176 1.558 4.615 1.179 ...
 $ cv       : num  21.895 0.984 4.319 11.4 4.904 ...

We have tried:

mean(data$cv)
8.82453442580761

and

t.test(data$result1, data$result2, conf.level = 0.90, paired = T)

    Paired t-test

data:  data$result1 and data$result2
t = -2.4161, df = 7, p-value = 0.04636
alternative hypothesis: true difference in means is not equal to 0
90 percent confidence interval:
 -5.1858973 -0.6274352
sample estimates:
mean of the differences 
              -2.906666 

The standard deviation of the results using this protocol is expected to be 1.8.

But we are not sure how we can interpret these results.

My third question: how can I know I have done enough samples?

Once we have an answer to the equivalence between the 2 reagent batches (whatever the method), how can we make sure this result is strong/significant enough ?

We are using R for the statistical analyses.

Thanx in advance for any help.

Answer 1

A German saying goes roughly like this: "the metall worker measures in tenth of a millimeter, a joiner measures in whole millimeters, the carpenter measures in centimeters and the brick layer - you're lucky if he stays within your real estate." Different trades/crafts require different levels of precision. A statistician will not be the one to tell you, which deviations in measurements are acceptable within your trade. As for blood samples I guess a 10% difference would often be just acceptable for blood sugar but certainly not for arterial blood pH.

You will have to define your acceptable deviation and whether you need a 95% chance of that being met or a sex-sigma chance, dependent on the impact a wrong measuremet might have.

Only after that CrossValidated and our advice come into play. You may for example use R's t.test function with paired = TRUE for paired measurements to obtain confidence intervals or use some Bayesian statistics. Using normal-normal conjugacy estimating the true mean of the differences and the expected normal distribution should be doable even within a spreadsheet. https://statswithr.github.io/book/bayesian-inference.html#three-conjugate-families https://statswithr.github.io/book/inference-and-decision-making-with-multiple-parameters.html#sec:normal-gamma

Edit: In your first comment on this answer you specified, that you want to go with R's t.test function and that an acceptable mean deviation is 5.4 on a 90% confidence level. Your call to t.test gave you a p value for an irrelevant null hypothesis, so do not care to much about that. It also gave you a confidence interval from -5.1859 to -0.6274. The confidence interval of a t-test is a good estimator for a credible interval (gained with a reasonable flat prior). We are not too far off to state, that the true difference between measuremens with the old an the new reagents lies in the [-5.2 ; -0.6] interval which does not include the acceptable mean deviation of +/- 5.4. Thus the true absolute deviation is smaller then the acceptable deviation.

Edit 2: This addendum was triggered by the comment starting with "Thanx @Bernhard for this extra explanation. The test is not cheap, ..." You have used the following function call: t.test(data$result1, data$result2, conf.level = 0.90, paired = T) A side not advice: Never use paired = T as it will stop working, once somebody enters T = 0 into your R session. Take the time and effort to write paired = TRUE. Now that is out of the way, this call performs a t test for a null hypothesis, that die true difference between the reagent is 0.000000000000000000000000000000000000000000000000000.... That is not how chemical analyses work so that obviously that is an irrelavant hypothesis . Nobody expect a difference to be perfectly zero. That is why I proposed to disregard the $p$-value of the null hypothesis altogether and concentrate on the confidence interval. Once you accept the idea, that a zero difference is not your goal, it is no longer of interest, whether zero is within the confidence interval.

However given a fixed sample size the t test can no longer detect arbitrarily small deviations from a given value. Power estimations and thereby sample size calculations depend on the concept of a null hypothesis test. For sample size computation an easy way to think about this is a one sided t test testing, whether the true difference is smaller than -5.4 and an additional one sided t test testing, whether the the difference is larger then 5.4. I do not recommend doing both these tests but one could use the idea for a sample size calculation employing the R function i refered to.