1
$\begingroup$

We are working on a biological protocol measuring a patient's feature using a blood sample. This protocol has proven to have some variability (CV <= 10%).

During this process we sometime need to change a reagent batch. To be sure the new batch does not alter the results, we are running the protocol using the 2 reagent batches (current and new) on the same samples. We do collect the results in a spreadsheet.

So far we are using a sample size of 5 to 10 samples, with no good rational.

So far we have consider the reagent lot to be OK if the mean of the Coefficients of Variation of the N samples (current vs. new, sample by sample) to be < 10% (again, no good rational).

My first question : how can we calculate an optimal sample size that will ensure we can run with the new reagent. As the reagent are commercial and come with a CE mark quality certificate, the objective is to make sure the reagent is not bad (expired, exposed to a to low/high temperature during transportation/storage, etc.), not to make sure it has the exact same results as the current one.

Once we have collected the results for the 2 reagent for N samples:

data <- data.frame(
    sample_id=c(1,2,3,4,5,6,7,8),
    result1=c(10.83167, 17.96167, 34.97500, 37.21833, 23.19833, 29.56167, 36.32167, 40.11833),
    result2=c(14.80000, 17.71333, 37.17833, 43.74500, 24.86500, 26.80500, 40.80667, 47.52667)
    )

My second question: how can I know my second reagent batch is equivalent to the first one?

So far here is what we have done:

data <- data %>% rowwise() %>% mutate(mean=mean(c(result1, result2)), sd=sd(c(result1, result2)))
data$cv <- (data$sd/data$mean)*100

So we have data:

str(data)
Classes ‘rowwise_df’, ‘tbl_df’, ‘tbl’ and 'data.frame': 8 obs. of  6 variables:
 $ sample_id: num  1 2 3 4 5 6 7 8
 $ result1  : num  10.8 18 35 37.2 23.2 ...
 $ result2  : num  14.8 17.7 37.2 43.7 24.9 ...
 $ mean     : num  12.8 17.8 36.1 40.5 24 ...
 $ sd       : num  2.806 0.176 1.558 4.615 1.179 ...
 $ cv       : num  21.895 0.984 4.319 11.4 4.904 ...

We have tried:

mean(data$cv)
8.82453442580761

and

t.test(data$result1, data$result2, conf.level = 0.90, paired = T)

    Paired t-test

data:  data$result1 and data$result2
t = -2.4161, df = 7, p-value = 0.04636
alternative hypothesis: true difference in means is not equal to 0
90 percent confidence interval:
 -5.1858973 -0.6274352
sample estimates:
mean of the differences 
              -2.906666 

The standard deviation of the results using this protocol is expected to be 1.8.

But we are not sure how we can interpret these results.

My third question: how can I know I have done enough samples?

Once we have an answer to the equivalence between the 2 reagent batches (whatever the method), how can we make sure this result is strong/significant enough ?

We are using R for the statistical analyses.

Thanx in advance for any help.

$\endgroup$
  • $\begingroup$ What's the difference between your first and third question? $\endgroup$ – StatsStudent Sep 2 at 9:44
  • $\begingroup$ @StatsStudent : Question 1: how can we anticipate how many samples we should run ? Question 3 : now that we ran N samples, is the result significant ? $\endgroup$ – Olivier D. Sep 2 at 18:51
2
$\begingroup$

A German saying goes roughly like this: "the metall worker measures in tenth of a millimeter, a joiner measures in whole millimeters, the carpenter measures in centimeters and the brick layer - you're lucky if he stays within your real estate." Different trades/crafts require different levels of precision. A statistician will not be the one to tell you, which deviations in measurements are acceptable within your trade. As for blood samples I guess a 10% difference would often be just acceptable for blood sugar but certainly not for arterial blood pH.

You will have to define your acceptable deviation and whether you need a 95% chance of that being met or a sex-sigma chance, dependent on the impact a wrong measuremet might have.

Only after that CrossValidated and our advice come into play. You may for example use R's t.test function with paired = TRUE for paired measurements to obtain confidence intervals or use some Bayesian statistics. Using normal-normal conjugacy estimating the true mean of the differences and the expected normal distribution should be doable even within a spreadsheet. https://statswithr.github.io/book/bayesian-inference.html#three-conjugate-families https://statswithr.github.io/book/inference-and-decision-making-with-multiple-parameters.html#sec:normal-gamma

Edit: In your first comment on this answer you specified, that you want to go with R's t.test function and that an acceptable mean deviation is 5.4 on a 90% confidence level. Your call to t.test gave you a p value for an irrelevant null hypothesis, so do not care to much about that. It also gave you a confidence interval from -5.1859 to -0.6274. The confidence interval of a t-test is a good estimator for a credible interval (gained with a reasonable flat prior). We are not too far off to state, that the true difference between measuremens with the old an the new reagents lies in the [-5.2 ; -0.6] interval which does not include the acceptable mean deviation of +/- 5.4. Thus the true absolute deviation is smaller then the acceptable deviation.

Edit 2: This addendum was triggered by the comment starting with "Thanx @Bernhard for this extra explanation. The test is not cheap, ..." You have used the following function call: t.test(data$result1, data$result2, conf.level = 0.90, paired = T) A side not advice: Never use paired = T as it will stop working, once somebody enters T = 0 into your R session. Take the time and effort to write paired = TRUE. Now that is out of the way, this call performs a t test for a null hypothesis, that die true difference between the reagent is 0.000000000000000000000000000000000000000000000000000.... That is not how chemical analyses work so that obviously that is an irrelavant hypothesis . Nobody expect a difference to be perfectly zero. That is why I proposed to disregard the $p$-value of the null hypothesis altogether and concentrate on the confidence interval. Once you accept the idea, that a zero difference is not your goal, it is no longer of interest, whether zero is within the confidence interval.

However given a fixed sample size the t test can no longer detect arbitrarily small deviations from a given value. Power estimations and thereby sample size calculations depend on the concept of a null hypothesis test. For sample size computation an easy way to think about this is a one sided t test testing, whether the true difference is smaller than -5.4 and an additional one sided t test testing, whether the the difference is larger then 5.4. I do not recommend doing both these tests but one could use the idea for a sample size calculation employing the R function i refered to.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Our acceptable deviation is 1*sigma to 3*sigma (sigma=1.8) ; so say 3*sigma=5.4 We are ok with 90 or 95% confidence. As you can read in the question, we have already done the t.test with paired=T. We need help in interpreting the results of this t.test, and help us understand if it is significant or not. Thanks, Olivier. $\endgroup$ – Olivier D. Sep 2 at 13:26
  • $\begingroup$ Excellent answer, IMHO. But I'll add another when I wrap up some work here. +1 @OlivierD. $\endgroup$ – StatsStudent Sep 2 at 20:18
  • $\begingroup$ OK @StatsStudent ; to be honest this answer didn't help much $\endgroup$ – Olivier D. Sep 3 at 11:59
  • $\begingroup$ Well it helped as much as it made you specify an acceptable deviation. Once StatsStudent posts his answer he is likely to refer to that. Meanwhile, I made an addendum to my answer, hopefully expanding usefully with the directions you gave in the first comment. $\endgroup$ – Bernhard Sep 3 at 13:24
  • $\begingroup$ @StatsStudent Thank you. Looking forward to read yours. $\endgroup$ – Bernhard Sep 3 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.