3
$\begingroup$

I want to rank ten or so sources based on their respective error measurements.

A few notes on the sources and measurements:

  • The sources do not have the same number of measurements
  • The number of measurements range in the 100s to the 1000s
  • There are no expectations on the mean nor variance of any of the measurements
  • The data is not necessarily normally distributed
  • It is expected the differences between some sources' means are not statistically significant

In other words, the data is not exactly text book clean.

1. What would be the most suitable statistical methods to approach this ranking?

I'd also be interested if these assumptions of mine are 'on the right track'?

  1. If some sources have the same mean, that is, the difference between their means is not statistically significant, I would assume this means those particular sources can't be ranked?
  2. I suppose one approach to this problem is to perform an ANOVA test followed by a post hoc test, such as Tukey's HSD test. But it seems this is a poor fit with my data (not the same length, non normally distributed data)?
  3. Instead of performing Tukey's HSD test, can I simply perform pairwise significance tests between sources, and then map out which sources share the same mean, and which have significant means?
  4. Should I stick to the standard two way t-stat test, or is it perhaps better to use the bootstrap method when testing for significance between sources? After all, the measurements do not necessarily follow the normal distribution. And the bootstrap method looks very intuitive and robust.
  5. After having performed the pairwise comparisons, can I then simply proceed and rank according to the means? For example, the significant means above the mean of the non-significant group are given ranks higher than the rank of the non-significant group, and the significant means below the mean of the non-significant group are given ranks lower than the rank of the non-significant group.

Lastly, is there an angle I am missing here? Have I gone off on a tangent here? It has been a while since I dipped into statistics. And I have never been much of a statistician in the first place...

Overall I favor simplicity over complexity, and I'd rather be broadly right than precisely wrong.

Can someone please point me in the right direction?

Improve this question
$\endgroup$
2
  • $\begingroup$ c00klemonster I think you are confusing this with multiple testing and p-value adjustment methods. $\endgroup$
    – Michael R. Chernick
    Commented Mar 5, 2017 at 5:01
  • $\begingroup$ @MichaelChernick I'll be the first to admit I might very well be confusing things in my assumptions 2-6. But how can I then un-confuse myself and ultimately answer my question, what would be the most suitable statistical methods to approach this ranking? $\endgroup$
    – c00kiemonster
    Commented Mar 5, 2017 at 7:50

1 Answer 1

Reset to default
-2
$\begingroup$

There are two books on this topic published by Wiley and another I think that was published by Springer-Verlag. This was the first one published by Wiley in 1972.

Amazon lists some others but not the one I was thinking about. I am trying to find the other one published by Wiley which is much more current.

Regression Modeling Strategies by Frank Harrell (one of our users) is now in hardcover and paperback.

Another one is Order Restricted Inference by Robertson T., Wright F. T. and Dykstra, R. L. published by Wiley in 1988.

To answer gammer once and for all, the frequentist would order the populations based on the largest observation in each population, particularly when all the sample sizes are equal.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Any good links to share? $\endgroup$
    – c00kiemonster
    Commented Mar 5, 2017 at 3:43
  • $\begingroup$ Yes I am looking for them and hope to add them to my answer. $\endgroup$
    – Michael R. Chernick
    Commented Mar 5, 2017 at 3:45
  • $\begingroup$ So, what is the correct method to rank populations? Rank the sample means? $\endgroup$
    – gammer
    Commented Mar 5, 2017 at 4:11
  • 1
    $\begingroup$ I guess you could calculate posterior probabilities of a given ranking system, and choose the most likely one. Is that how it works? I don't have any of those textbooks. $\endgroup$
    – gammer
    Commented Mar 5, 2017 at 5:33
  • 2
    $\begingroup$ I'm surprised that the answer is to rank based on the rankings of their maximums. In a simple simulation with $n=250$ in each of five groups, normally distributed with means $\mu = 0.4, 0.8, 1.2, 1.6, 2.0$, and $\sigma^2 = 1$ for all groups, I find that rankings by their sample means gets the ranking right basically every time ($100\%$ of 10000 runs) and the ranking by the sample maximums gets it right about $27.9\%$ of the time. If the variances are not equal (and not too small), you find that ranking the maximums detects the ordering of the variances better than the ordering of the means. $\endgroup$
    – gammer
    Commented Mar 5, 2017 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.