Which test to see if two set of continuous data are comparable Scenario: 
Two different persons have measured lesions in millimeters before and after medicine, so the data are continuous. I have calculated both of these readers' difference in measurements and want to use the mean difference for further calculations.
To do this I think I have to test whether or not these measurements are not too different? How do I do that? I thought about a paired t-test since the measurements have been done on the same patients ... but I have the feeling it is not right.
Ideas?
 A: I'd use a Bland-Altman plot. In this plot the x-axis is the mean of your two measurements, and the y-axis is the difference between the two measurements. From this plot you can estimate the bias and the 95% limits of agreement of the two methods. In R you can use the BA.plot function in package MethComp

A: There are two things you want to be mindful of:


*

*Potential bias/systematic error: Observers systematically under or over-estimating the difference.

*Reliability/random measurement error: Observers disagreeing randomly on any particular measure while, on average, none of them reports systematically higher or lower measures.


A paired sample t-test would in fact be appropriate to show that there was a difference of the first kind between the two observers but turning it on its head and taking the lack of significant difference as implying that they provided similar measures is not. The main reason for that is that the power of the test (and it's complement, the type II error rate) is hugely dependent on the sample size.
With a large sample, a tiny discrepancy between observers will be significant and you would be reluctant to aggregate the two measures for no good reason. With a small sample, you could be unaware of large discrepancies. This does not seem very useful. One way to actually test if the discrepancy is not too big is called “equivalence testing”. You would need to define beforehand what “too big” means for you, i.e. how much of a discrepancy you are willing to tolerate. Even so, the power of the test will depend on the sample size but at least the hypothesis being tested is formulated in a way that matches your question.
Note than none of this directly addresses the issue of reliability. It's perfectly possible for observers to provide sets of measures with the same mean and yet have individual measures with a lot of random error. This would also call the whole measurement process into question as it would suggest that observers are unable to agree on the evolution of any particular lesion or that additional observers would be necessary to obtain reliable measures. One way to look at that would be to do a scatterplot/look at the correlation between the two sets of measures.
At the end of the day, I am not sure you need a test. There is no absolute threshold for bias or reliability that could tell you that the measurement is good enough for any purpose and no reason why any small detectable discrepancy should prevent you from doing anything else. In fact, taking the average from several measures is especially useful when these measures are themselves unreliable.
What I would rather do is use plots: side-by-side stripcharts, boxplots or density plots for potential bias or differences in the distribution of measures coming from each observer, scatterplot for the reliability/noise in the individual measures.
A: Paired t-test can be used for repeated measurements under the assumption that the differences are normally distributed. If that is not the case you might want to use Wilcoxon signed rank test, which covers non-normal distributions also. 
