I want to rank ten or so sources based on their respective error measurements.
A few notes on the sources and measurements:
- The sources do not have the same number of measurements
- The number of measurements range in the 100s to the 1000s
- There are no expectations on the mean nor variance of any of the measurements
- The data is not necessarily normally distributed
- It is expected the differences between some sources' means are not statistically significant
In other words, the data is not exactly text book clean.
1. What would be the most suitable statistical methods to approach this ranking?
I'd also be interested if these assumptions of mine are 'on the right track'?
- If some sources have the same mean, that is, the difference between their means is not statistically significant, I would assume this means those particular sources can't be ranked?
- I suppose one approach to this problem is to perform an ANOVA test followed by a post hoc test, such as Tukey's HSD test. But it seems this is a poor fit with my data (not the same length, non normally distributed data)?
- Instead of performing Tukey's HSD test, can I simply perform pairwise significance tests between sources, and then map out which sources share the same mean, and which have significant means?
- Should I stick to the standard two way t-stat test, or is it perhaps better to use the bootstrap method when testing for significance between sources? After all, the measurements do not necessarily follow the normal distribution. And the bootstrap method looks very intuitive and robust.
- After having performed the pairwise comparisons, can I then simply proceed and rank according to the means? For example, the significant means above the mean of the non-significant group are given ranks higher than the rank of the non-significant group, and the significant means below the mean of the non-significant group are given ranks lower than the rank of the non-significant group.
Lastly, is there an angle I am missing here? Have I gone off on a tangent here? It has been a while since I dipped into statistics. And I have never been much of a statistician in the first place...
Overall I favor simplicity over complexity, and I'd rather be broadly right than precisely wrong.
Can someone please point me in the right direction?