I wrote an algorithm that scores all potential X per group, so the data will look like this:

X1: 0.000001
X2  0.8000 (MAX)
X3  0.0003 (CURRENT)

X1: 0.999 (MAX) (CURRENT)
X2  0.0003

X1: 0.0004 (MAX)
X2: 0.0003 (CURRENT)

 etc.. --> 4000 groups

These scores range from 0 to 1 and I assume that the scores are distributed normally. Now I want to "test" whether the MAX scoring X I found significantly differs form the CURRENT X per group. I'm not sure if this is even possible to test, but if so how can this be done? For example, I'm pretty sure the MAX is better when the score is 0.8 compared to 0.0003 in group 1, however for group 3 it is hard to say since both score low and their difference is small.

Reaction to comment @glen_b
The scores were scaled afterwards to fall within 0 and 1 (easier to compare), but actually originated from this distribution:

enter image description here (This plot is simply based on all scores in all groups). I find the scored to be skewed quite a lot towards the < 0 end, perhaps making the normality assumption invalid. However based on the problem we are trying to solve (using my algorithm) I expect the scores to be normally distributed.

  • $\begingroup$ The two statements "scores range from 0 to 1" (assuming you mean to refer to possible values rather than just what was observed in some sample) and "the scores are distributed normally" are contradictory. Either scores can go outside the range 0-1 or they are not actually normally distributed (but may still be approximately normally distributed) $\endgroup$ – Glen_b Apr 10 at 0:35
  • $\begingroup$ Indeed, thats why I stated I assume that the scores are distributed normally @Glen_b $\endgroup$ – KingBoomie Apr 10 at 0:50
  • $\begingroup$ I'm a little confused; do you mean you're choosing to use normality as an approximation? (Otherwise assuming what you know to be false would seem odd -- hence my comment, but choosing to use it as an approximation makes sense to me) $\endgroup$ – Glen_b Apr 10 at 1:45
  • $\begingroup$ @Glen_b aah I see I'm sorry let me clarify, see my edit $\endgroup$ – KingBoomie Apr 10 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.