This question has been heavily paraphrased and re-asked in a broader, but better way here:
Paired comparison of instruments using different measurement samples

This is going to be a long one, since it's wrecking my nerves for days now. =)

I have 4 algorithms (A, B, C and D).
Each outputs an integer as a result of its calculation. Consider them instruments with slight variations in how they calculate outputs.

That said, I've evaluated all 4 of them, using humans as the "ground truth", since they measure something humans could express too. A single measurement of this "amount" is called a determination, and yields a single determination score.
Each determination score ranges from -1.0 to 1.0, showing how aligned each algorithm is with what the humans said to be true. For each algorithm, there are 1152 measurements, using the same 16 humans for "ground truth".

I would like to rank the algorithms' performances according to the following criteria:

  • C1) Average (mean) of all determination scores;
  • C2) Spread (dispersion, variation, scatter) of values.

The algorithm with the greatest C1) and smallest C2) wins. That algorithm would have the greatest amount of "true" determinations on average (C1), with the smallest inconsistencies in scores, making it the most consistent (C2).

My problem is with C2).
Since I'm comparing spreads across different determination-score-distributions, I'm not sure which of the measures to employ. I'm aware that these belong to estimators of scale or estimators of dispersion.
Here's the boxplot of algorithms A, B, C and D.


I've gone through a lot of estimators, and devised a list of the desired properties:

  1. Applicable to variables using interval scale and not just ratio scale
  2. Applicable to variables containing both negative and positive values
  3. Insensitive to mean (average) value close to or approaching zero
  4. Insensitive to variables whose mean (average) value can be zero
  5. Invariant (robust) to underlying distribution of the variable (i.e. non-parametric if possible)
  6. Invariant (robust) to a small number of outliers
  7. Invariant (robust) to location parameter (or choice of central tendency), skewness or kurtosis of the distribution.

That crosses-out:

  • variance
  • std. deviation
  • coeff. of variation (CV)
  • dispersion index (or variance-to-mean ratio)
  • interdecile range
  • median absolute deviation from median (MADM)
  • mean absolute deviation from mean (MAD)

...and leaves (out of those that I can think of):

  • IQR
  • Quartile coeff. of dispersion (derived from IQR)
  • Qn and Sn by Rousseeuw-Croux
  • other M-estimators and stuff I'm not yet very familiar with.

Here are my questions:
I'm not sure whether I need 7) as a desired property.
As can be seen from the boxplot, means and medians of algorithms are all different. I'd like to avoid a measure which is affected by this:

  1. Could I "uniformly center" the data by subtractions using the mean, would that help?
  2. By doing this "leveling" of all 4 means to zero - would it make the MADM and MAD usable?
  3. Are there any other robust estimators of scale (dispersion) I could use?

To give a one-liner of what I think I'm looking for:

Robust, preferably non-parametric estimator of scale (or dispersion), invariant to change in central-tendency (or location), so that it could be used for comparing (dispersion of) samples.

Thanks for ur time and sry for the long post!

EDIT: Tnx to German Demidov, I've plotted the Bland-Altman plot of A vs. D.
Could I use those to find the more dispersed one?

Bland-Altman plot

  • 1
    $\begingroup$ en.wikipedia.org/wiki/Bland–Altman_plot ? $\endgroup$ Mar 8 '16 at 18:09
  • $\begingroup$ Tnx for the quick reply. I was not aware of the Bland-Altman plots - how may I compare spreads of score distributions with it? Let me add that I'm not interested in the level of "agreement" between the methods, see (C1) above as to why. Maybe the QQ plot would be more appropriate, instead? $\endgroup$ Mar 9 '16 at 1:42
  • 1
    $\begingroup$ $Q_n$ is good for your purpose. And Lehman-Hodges estimator for the mean. But this solution will not be really accurate =) there is no absolutely accurate robust method (except situations where you know proportion of outliers or true mixed distributions). Also I see that 5 outliers happen recurrently. You can remove them (all methods predict wrongly on this data points). $\endgroup$ Mar 9 '16 at 16:21
  • 1
    $\begingroup$ IQR has some problems. You need to increase your sample size $\sim$ 2 times to obtain the estimation with the same standard error (comparing to the standard deviation). $Q_n$ and $S_n$ have higher efficiency, but efficiency of $S_n$ is a bit lower than $Q_n$'s. $\endgroup$ Mar 10 '16 at 16:58
  • 1
    $\begingroup$ wis.kuleuven.be/stat/robust/papers/publications-1993/… - see Standardized Variance for MAD. I'm not sure that it's the same for IQR. You can try both and then choose the result that you like more. $\endgroup$ Mar 10 '16 at 21:31

As for me, it is not the best way of ranking. Your means have real values, so (as I understood) C1 will always be satisfied so you will not have to apply C2 selection criteria. Also you should not forget that you have confidence intervals for means, not just point estimations of means: it can be better by chance. So I would look at more traditional ways to measure performance, such as $r^2$, and of course use cross-validation (or train/test approach). Also I would recommend to exclude several outliers that are wrongly estimated by all 4 algorithms.

But, answering the second question, I would recommend to use $Q_n$ or $S_n$. Both of them have nice properties. Somebody can recommend to use Gini's means difference, but it has 0 breakdown point (but it is somewhat "robust" and also has a lot of good properties). As for C1, I would recommend to use Lehman-Hodges estimator.

  • $\begingroup$ I just noticed these several points as additions to an answer by German Demidov: If by $r^2$ u mean the coefficient of determination, I should point out that I am not interested in regression, since the plotted data containts performance scores of each algorithm, which you noted correctly. That said, it is not of our interest how well the scores fit a certain model, because the model is unimportant. As to everything else, I tentatively agree in search for a better answer. $\endgroup$ Mar 11 '16 at 14:59
  • 1
    $\begingroup$ Using sd you measure the fit to model $y = \alpha$, where $\alpha$ denotes mean. I do not understand why do you need determination scores and why you just can not apply $r^2$ or Bland-Altman plots without "determination scores". I do not what it is since I do not know statistical term with this name, but I assume that you used some heuristical approach to obtain this scores. Why can not you just compare performance of algorithm with the truth without this scores? $\endgroup$ Mar 11 '16 at 15:56
  • $\begingroup$ To answer your last question - because the "truth" is not a binary variable (for an algorithm). It has a certain level of "truthfullness" to each measurement. To answer you fully, consult my question heavily paraphrased and re-asked in a broader, but better way here: stats.stackexchange.com/questions/201190/… $\endgroup$ Mar 11 '16 at 18:29
  • $\begingroup$ ... I just noticed that you might be confusing my "determination scores" with "coeff. of determination". If so, beware that the two are completely unrelated - I apologize for this inconvenience. $\endgroup$ Mar 11 '16 at 18:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.