# Paired comparison of instruments using different measurement samples

I have instruments A, B, C and D - I'm in search of the best one.

The problem:
For illustrative purposes, let's use an example of evaluating the best among the instruments measuring difference in color between cards in a card-pair. The best instrument is the one which correctly measures the greatest number of darker-colored cards.

Each instrument makes $N = 1152$ measurements of the same 1152 card-pairs. A card can be black, white or a gradient of gray.
If the instrument measures both cards as black, the measurement equals 0. Both white = 0. Black vs. white = 100. 30% gray vs. 65% gray = 35, etc. Let's call this difference and hence each measurement, $\Delta_i$. (Notice that it is defined as the absolute (unsigned) difference.)

How do we score the measurement as correct or incorrect?
We compare it against the "ground truth". In my case, due to certain design limitations and details I shall not go into - "ground truth" is 16 human subjects, each estimating 72 card-pairs. Notice that $16 \cdot 72 = 1152$.
Humans merely say which of the cards is darker. This card is then considered a correct response between two cards in a pair.

Back to the instruments - we score each instrument measurement as follows:
If the measurement is correct, its score equals:

• $ms_i = +\Delta_i/100$.

...if incorrect, its score equals:

• $ms_i = -\Delta_i/100$.

In other words, $\Delta_i$ tells us how "certain" an instrument is in declaring a certain card as darker. A measurement with a $\Delta_i = 100$ "certainty", if correct, receives a maximum score $ms_i = +1.0$.
If incorrect, it is "severely punished" for being that certain, receiving a minimum $ms_i = -1.0$. The principle applies to all other measurements.
This yields a total of $M = 1152$ scores $(ms_i)$ for each instrument.

My question:

• How do I find the best instrument, in the most statistically correct manner?

If possible, I would refrain from solutions that are very mathematically complex; it needs to be both understandable to me, as well as explainable to others in my field - which is IT.

Note: All $\Delta_i$ attain a value between 0..100 with no attention to the way card-pairs are picked (i.e. we can make no underlying assumption of their distribution). Ideally, card-pairs would be chosen as truly random, but we cannot guarantee it.
Moreover, important note: each instrument is given an identical set of $\Delta$s in an identical order, for the needs of fair comparison.

...EXTRA:

For those of you interested in my approach so far, consider the boxplot of 1152 measurement scores $ms_i$ for each of the instruments:

In order to correctly declare the best instrument, I need to use an estimator of location - the mean would reflect "highest scored instrument on average" pretty good. It's not robust and its breakdown point is 0, which is no big deal because the underlying data are actually "grades of accuracy". In order to find the best-graded instrument, I use the mean. (Median is a 50%-trimmed mean, and is not an appropriate measure of accuracy in this case.)

To check instrument's precision and see if it's consistently correct (or incorrect), I need an estimator of spread (dispersion). I've gone through a lot of estimators, and devised a list of the desired properties:

1. Applicable to variables using interval scale and not just ratio scale
2. Applicable to variables containing both negative and positive values
3. Insensitive to mean (average) value close to or approaching zero
4. Insensitive to variables whose mean (average) value can be zero
5. Invariant (robust) to a small number of outliers (see boxplot)
6. Invariant (robust) to underlying distribution of the variable (i.e. non-parametric if possible)
7. Invariant (robust) to asymmetry of the distribution and location estimate (or choice of central tendency, e.g. mean or median, because they differ between instruments; see boxplot)
8. Has the best possible breakdown point.

That crosses-out:

• variance
• std. deviation
• coeff. of variation (CV)
• dispersion index (or variance-to-mean ratio)
• interdecile range
• median absolute deviation from median (MADM)
• mean absolute deviation from mean (MAD)

...and leaves (out of those that I can think of):

• $IQR$
• Quartile coeff. of dispersion (derived from $IQR$)
• $Q_n$ and $S_n$ by Rousseeuw-Croux
• (other stuff I'm not yet very familiar with.)

This is where I'm stuck - I would use the mean (for estimating location), and $IQR$, $Q_n$ and $S_n$ (for estimating spread). The "best" instrument has the greatest mean, with the lowest $IQR$, $Q_n$ and $S_n$.

... is this approach correct or not? =)