4
$\begingroup$

Scenario: Two different persons have measured lesions in millimeters before and after medicine, so the data are continuous. I have calculated both of these readers' difference in measurements and want to use the mean difference for further calculations. To do this I think I have to test whether or not these measurements are not too different? How do I do that? I thought about a paired t-test since the measurements have been done on the same patients ... but I have the feeling it is not right.

Ideas?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

I'd use a Bland-Altman plot. In this plot the x-axis is the mean of your two measurements, and the y-axis is the difference between the two measurements. From this plot you can estimate the bias and the 95% limits of agreement of the two methods. In R you can use the BA.plot function in package MethComp

Bland Altman plot

Improve this answer
$\endgroup$
2
$\begingroup$

There are two things you want to be mindful of:

  • Potential bias/systematic error: Observers systematically under or over-estimating the difference.
  • Reliability/random measurement error: Observers disagreeing randomly on any particular measure while, on average, none of them reports systematically higher or lower measures.

A paired sample t-test would in fact be appropriate to show that there was a difference of the first kind between the two observers but turning it on its head and taking the lack of significant difference as implying that they provided similar measures is not. The main reason for that is that the power of the test (and it's complement, the type II error rate) is hugely dependent on the sample size.

With a large sample, a tiny discrepancy between observers will be significant and you would be reluctant to aggregate the two measures for no good reason. With a small sample, you could be unaware of large discrepancies. This does not seem very useful. One way to actually test if the discrepancy is not too big is called “equivalence testing”. You would need to define beforehand what “too big” means for you, i.e. how much of a discrepancy you are willing to tolerate. Even so, the power of the test will depend on the sample size but at least the hypothesis being tested is formulated in a way that matches your question.

Note than none of this directly addresses the issue of reliability. It's perfectly possible for observers to provide sets of measures with the same mean and yet have individual measures with a lot of random error. This would also call the whole measurement process into question as it would suggest that observers are unable to agree on the evolution of any particular lesion or that additional observers would be necessary to obtain reliable measures. One way to look at that would be to do a scatterplot/look at the correlation between the two sets of measures.

At the end of the day, I am not sure you need a test. There is no absolute threshold for bias or reliability that could tell you that the measurement is good enough for any purpose and no reason why any small detectable discrepancy should prevent you from doing anything else. In fact, taking the average from several measures is especially useful when these measures are themselves unreliable.

What I would rather do is use plots: side-by-side stripcharts, boxplots or density plots for potential bias or differences in the distribution of measures coming from each observer, scatterplot for the reliability/noise in the individual measures.

Improve this answer
$\endgroup$
0
$\begingroup$

Paired t-test can be used for repeated measurements under the assumption that the differences are normally distributed. If that is not the case you might want to use Wilcoxon signed rank test, which covers non-normal distributions also.

Improve this answer
$\endgroup$
3
  • $\begingroup$ -1 Thanks for providing some relevant information but this completely misses the fact that the original poster seems interested in equivalence, not differences. $\endgroup$
    – Gala
    Jul 24, 2013 at 16:25
  • $\begingroup$ I got that from you answer. Thank you for sharing that to me. It is amazing how much you can learn from interactions on stackexchange, i.e. from all of you. $\endgroup$
    – rapaio
    Jul 24, 2013 at 16:59
  • $\begingroup$ @Gaël Laurans It all turns on how you interpret what is otherwise an uncomfortably vague question: it only refers to whether measurements are "not too different" and never uses "equivalent." Your answer and that of Aghila are thoughtful, informative expositions of interesting and valid interpretations--but that does not make Aurelian's answer wrong or misguided. I think it correctly answers another valid interpretation of the question. Perhaps all it lacks is a clear statement that it interprets the question as asking for a test of a difference of central values. $\endgroup$
    – whuber ♦
    Jul 25, 2013 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.