3
$\begingroup$

I am doing a study in which I am measuring a set of lengths using two different methods. I want to test how close the measurements given by the two methods are for corresponding lengths. Is there a way to do this? I know that a matched pairs t-test can give a p-value for the means being different, but how can I test how close the differences are to 0?

In other words, since the two methods are measuring the same lengths, they should most ideally give the same measurement for any given length. I want to test how close this is to being true and quantify the amount of error that is present. So say I plotted the lengths given by one method against those given by the other. If they did give the same measurement for every length, it should form the line $Y=X$. Basically, I want to test the hypothesis that the data does form the line $Y=X$, calculate the mean error (maybe the average distance from $Y=X$), and perhaps produce a p-value for it.

$\endgroup$
  • $\begingroup$ Give more detail! You probably do not want testing (you do not have an hypothesis!), you want confidence intervals. $\endgroup$ – kjetil b halvorsen Sep 4 '16 at 19:50
  • 1
    $\begingroup$ 1. Is it definitely the mean of the distribution of differences you wish to test (rather than a more general kind of location-difference)? 2. Note that a p-value (what you ask for in the body text) is not the same as "probability that matched pairs have mean difference of 0"; indeed ordinary hypothesis tests don't give you such a probability. Unless you specifically want a Bayesian approach I'd suggest you remove the word "probability" from your title. $\endgroup$ – Glen_b Sep 4 '16 at 20:02
  • $\begingroup$ In case it isn't clear, I'm a beginner to stats. I'm not sure how to explain precisely what I want. I'm looking for any advice as to the best way to test how closely the two methods are measuring the same values. $\endgroup$ – user61118 Sep 4 '16 at 20:06
  • $\begingroup$ I thought that a p value would be nice, but if that isn't the best approach, anything else that may seem to apply would be fine. $\endgroup$ – user61118 Sep 4 '16 at 20:06
  • $\begingroup$ Try searching for Bland and Altman plot which people in health research often use in these circumstances. $\endgroup$ – mdewey Sep 4 '16 at 20:13
1
$\begingroup$

I think you are mixing different tools and different goals. I summarize several techniques available for what I understand you want to do:

  • t-test for paired samples tests the hypothesis that both mesures have the same mean and gives a p-value about it.
  • Confidence intervals estimate an interval where the true difference in means is expected to be with a given probability (usually 95%). That seems what comes most close to "test how close the measurements given by the two methods are for corresponding lengths". Most statistical packages produce confidence intervals when doing a t-test.
  • If you are interested in measuring the errors between the two measures (an not just the difference of means), standard deviation of differences or coeficient of variation might be useful. Studying the distribution of errors and checking for normality can give you interesting information, too.
  • How are the two measures related, or how are they related to error might be interesting too. You have plotted one measure against the other, but next steps could be plotting error against one measure (specially if errors are much smaller than measures, as usually happens) and performing a regression analysis.

Probably my answer is more a menu of things you can answer now than a real answer to your needs, but if you use it to make your (next) question more specific I hope we will be able to help you more.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

I believe the simplest way to answer your question would be to make a scatter plot with the following bivariate data for each measurement method you wish to test: x=actual length, y=measured length by (insert method here). If your method was perfect, there would indeed be a y=x least squares line with a sample correlation of r=1. If you want to test whether or not one method is significantly better than another, you could test the significance of differences between the two competing methods via their scatter plot correlation coefficients using Fisher's r-to-z transformation. You can find the probability that the differences in measurement methods would occur randomly using the formula in this article: http://www.statisticssolutions.com/comparing-correlation-coefficients/.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

You are seeking to quantify the agreement of these two measurements (see Wikipedia and/or John Uebersax's site). A simple measure of the agreement between two continuous variables would be Lin's concordance coefficient, for which confidence intervals can be computed.

Another approach would be to use the methods of Bland and Altman. You would do well to read their famous paper (which is short and easy), "Statistical methods for assessing agreement between two methods of clinical measurement" (pdf). Briefly, using the corresponding values from the two measurement methods, you would form two new sets of values, their means and their differences. A plot of the differences against the means is called a Bland-Altman plot. This will make it easy to assess if there is a curvilinear relationship or changing variance. If nothing is amiss, and you want to perform a hypothesis test, you can use the differences to conduct a one-sample $t$-test against $0$. You can also compute the mean difference and form a confidence interval around that.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.