4
$\begingroup$

I have two datasets, A and B. Both ultimately come from the same source, both have the same data attributes, but the datasets use different measurement methods to derive the attributes. We want to know that both A and B are not only correlated but in agreement.

How do I compare the results of A to B in such a way that shows that the results of data B agree with, but may be more accurate than data A?

Example:

A data result

+----------+----------+----------+
| Location | MeasureA | MeasureB |
+----------+----------+----------+
| 1        | 34.56    | 234.6    |
| 2        | 123.0    | 56.78    |
| .        | .        | .        |
| .        | .        | .        |
| 256      | 68       | 453.12   |
+----------+----------+----------+

B data result

+----------+----------+----------+
| Location | MeasureA | MeasureB |
+----------+----------+----------+
| 1        | 35.1     | 234.4    |
| 2        | 122.7    | 56.7     |
| .        | .        | .        |
| .        | .        | .        |
| 256      | 68.3     | 453.14   |
+----------+----------+----------+

The main goal is to be able to define data pipeline regressions across differing methods without the results being exactly equivalent.

$\endgroup$
4
  • 1
    $\begingroup$ I am not sure if I understood exactly the question. To see the the agreement between the two versions, could you not use something such as Spearman rank Correlation? If you are interested in the distribution of the measures, maybe you could use a qq-plot? $\endgroup$ Commented Feb 14, 2020 at 13:11
  • $\begingroup$ It is not clear what you are asking. You want to show "that the results of data version2 agree with, but are more accurate than data version1". Agree with what? $\endgroup$
    – yrx1702
    Commented Feb 14, 2020 at 18:45
  • $\begingroup$ @DanielTheRocketMan the Spearman correlation is exactly the direction I was looking for. What is a way I can rephrase the question so that it is more understandable and useful for others? $\endgroup$
    – Wes
    Commented Feb 17, 2020 at 17:17
  • $\begingroup$ Maybe you can just change the question to "Comparing the ranking of multidimensional datasets from different code versions" $\endgroup$ Commented Feb 18, 2020 at 0:51

2 Answers 2

2
+50
$\begingroup$

The Bland Altman plot is a much more nuanced tool for examining agreement between different methods. It allows you to examine bias as well as agreement. You can understand if the agreement is uniform across the range of values or if the methods disagree more across different parts of the range.

Determining which method is more accurate is separate step, you need to compare the two methods against a reference. I note both measures differ, do you have independent information on what the answer should be? In the absence of a reference that defines the expected truth you can't assess accuracy in a way that can say 2 is more accurate than 1.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4470095/

$\endgroup$
1
$\begingroup$

One simple way to see the the agreement between the two versions is to use the Spearman rank Correlation.

The Spearman's rank correlation coefficient is a nonparametric measure of rank correlation. It assesses how well the relationship between two variables can be described using a monotonic function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.