##Problem
I want to have an estimate for the certainty which of several (3-4) variables is the variable with the largest value, given some sample values which should eventually converge to different values.
###Example
So for example I might have 3 series of values for 3 variables:
0.8 0.2 0.4 0.3 0.5 0.4 0.3
0.2 0.5 0.1 0.1 0.05 0.1 0.05
0.1 0.4 0.6 0.4 0.8 0.9 0.9
And now I would expect that the 3rd variable is the one with the largest value because it seems to converge to something around 0.9, larger than the other two. Now I want some kind of "confidence estimate" that the third one is really the variable with the largest value, so I know whether I still need more samples or can select this variable already.
The confidence estimate should use as much information as possible (e.g. how much variance there still is, how many samples were collected etc)... We will always have the same amount of samples for each variable. The estimate should work for ~5-100 samples
Is there some usable formula/statistic test which can be used for this? :)
###Approach so far
At the moment I use some hand-engineered/tinkered solution which kind of works, but it a very ad hoc solution and there could certainly be much better ones?
1. Take variable with largest sample mean v_large
2. For all other variables v:
2.1 Make a modified 1-sided 2-sample t-test of v_large and v
3. Take the minimum of the p-values as the confidence estimate
The modification of the t-test (from ttest2 in matlab) in step 2.1 that I played around with was changing the formula for the t-test a bit to "decay" older values: I weigh variances earlier in the series lower than variances later in the series. Similarly, I compute the mean by weighing samples earlier in the series less than variances later in the series.
This seems to kind of work (i.e. there are thresholds for the confidence estimate which seem to guarantee I select the right value for the data I have tried so far), but I hope there is some better/cleaner solution? :)
More example data
Here are two files with actual example values, always 3 variables, largest is always the third one: