Problem
I want to have an estimate for the certainty which of several (3-4) variables is the variable with the largest value, given some sample values which should eventually converge to different values.
Example
So for example I might have 3 series of values for 3 variables:
0.8 0.2 0.4 0.3 0.5 0.4 0.3
0.2 0.5 0.1 0.1 0.05 0.1 0.05
0.1 0.4 0.6 0.4 0.8 0.9 0.9
And now I would expect that the 3rd variable is the one with the largest value because it seems to converge to something around 0.9, larger than the other two. Now I want some kind of "confidence estimate" that the third one is really the variable with the largest value, so I know whether I still need more samples or can select this variable already.
The confidence estimate should use as much information as possible (e.g. how much variance there still is, how many samples were collected etc)... We will always have the same amount of samples for each variable. The estimate should work for ~5-100 samples
Is there some usable formula/statistic test which can be used for this? :)
Approach so far
At the moment I use some hand-engineered/tinkered solution which kind of works, but it a very ad hoc solution and there could certainly be much better ones?
1. Take variable with largest sample mean v_large
2. For all other variables v:
2.1 Make a modified 1-sided 2-sample t-test of v_large and v
3. Take the minimum of the p-values as the confidence estimate
The modification of the t-test (from ttest2 in matlab) in step 2.1 that I played around with was changing the formula for the t-test a bit to "decay" older values: I weigh variances earlier in the series lower than variances later in the series. Similarly, I compute the mean by weighing samples earlier in the series less than variances later in the series. This seems to kind of work (i.e. there are thresholds for the confidence estimate which seem to guarantee I select the right value for the data I have tried so far), but I hope there is some better/cleaner solution? :)
More example data
Here are two files with actual example values, always 3 variables, largest is always the third one:
Background
The values are coming from predictions whether a brain signal contains a P300 wave (see wikipedia, can't post more links :( ).
I have an online system where I show pictures (one of which should elicit the P300) and record the brain signal with an EEG system. The EEG signal in a certain time interval after showing one picture would be one trial for that picture. I can predict whether the signal has a P300 by using a LDA-classifier trained on training data. I can also average the trials for the same picture to get more reliable predictions, this is what I am doing now. What I want is to know when I can stop showing pictures and make my final decision.
Some people use predictions on the individual trials without averaging for this (e.g. see "Performance optimization of ERP-based BCIs using dynamic stopping ", "Dynamic Stopping Improves the Speed and Accuracy of a P300 Speller"). Other people do use trial averaging and make predictions on the averaged signals, however I have not found the combination of predictions on averaged trials + dynamic stopping :/ And this is what I want to try to do :)
Note that predictions might not converge to 1 for one picture and 0 for the others. Typically one picture may elicit a smaller P300 even though ti not the desired picture...
Also note that the signal and therefore the prediction can be quite noisy as also seen in the prediction values of the example files posted above. That's why I want some robust estimate if i can make the final decision which also works for very noisy signals/predictions.
