I have a program to predict some values for people. For validation, I keep track of whether the prediction is correct or not, which gives me a binary vector with a length of about 600.
To test if it is significant above a random predictor, I created 100 random binary vectors of predictions for the same values. This gives me 100 random vectors and 1 real vector.
I want to test significance between the 1 real vector and the set of 100 random prediction vectors.
- Is there a good test to do this?
- Would a good methodology be to calc significance between two vectors at and then average p values?
- Would it be better to do something with distances, for example by calculating the Hamming distance between the real vector and the 100 random vectors, to get 100 distance values? If I do that, how can I know if that is significant or not? By checking skewness from normal?
- I just want to see if my predictions are greater than chance. Random predictions, averaged from 100 trials, are 0.33 accurate. Is there an easier way to test significance?