0
$\begingroup$

I have a program to predict some values for people. For validation, I keep track of whether the prediction is correct or not, which gives me a binary vector with a length of about 600.

To test if it is significant above a random predictor, I created 100 random binary vectors of predictions for the same values. This gives me 100 random vectors and 1 real vector.

I want to test significance between the 1 real vector and the set of 100 random prediction vectors.

  • Is there a good test to do this?
  • Would a good methodology be to calc significance between two vectors at and then average p values?
  • Would it be better to do something with distances, for example by calculating the Hamming distance between the real vector and the 100 random vectors, to get 100 distance values? If I do that, how can I know if that is significant or not? By checking skewness from normal?
  • I just want to see if my predictions are greater than chance. Random predictions, averaged from 100 trials, are 0.33 accurate. Is there an easier way to test significance?
Improve this question
$\endgroup$
4
  • $\begingroup$ Unless I'm mistaken, it sounds like you are greatly overcomplicating this. Do you just want to see if the prediction rate is significantly higher than expected by random chance? (note that the prediction rate by chance is not necessarily 50%. If 25% of patients die, and your algorithm predicted that all of them would live, your algorithm would have a 75% accuracy rate). $\endgroup$
    – David Robinson
    Commented Mar 7, 2013 at 15:20
  • $\begingroup$ Yes, I just want to see if my predictions are greater than chance. Random predictions, averaged from 100 trials, are 0.33 accurate. Is there an easier way to test significance? $\endgroup$
    – Makoto
    Commented Mar 8, 2013 at 2:08
  • $\begingroup$ Much easier, but let's focus on the .33 number. you said these are binary predictions (patient mortality?). You must at least be able to hit 50% by random chance alone $\endgroup$
    – David Robinson
    Commented Mar 8, 2013 at 2:56
  • $\begingroup$ I am predicting where people will go at intersections, so there are an average of 3 branches, hence .33 chance accuracy. However, each prediction is correct or incorrect, hence the binary. $\endgroup$
    – Makoto
    Commented Mar 8, 2013 at 4:03

1 Answer 1

Reset to default
1
$\begingroup$

It seems that I can use the binomial test in this situation, since I can calculate the chance probability from the random vectors and I can treat each prediction independently in this case.

http://en.wikipedia.org/wiki/Binomial_test

I used scipy for python to calculate this: http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binom_test.html#scipy.stats.binom_test

E.g.,

scipy.stats.binom_test(numTrue,n=600, p=.33)
Improve this answer
$\endgroup$
5
  • $\begingroup$ Indeed, that's correct- this can be done using (for example) pbinom in R or BINOMDIST in Excel. Note that there's no reason to generate random vectors to figure out the chance of getting a prediction right by chance. If there are three branches, and each has an equal probability, then the chance of getting a prediction right is .333. (But note that if some branches are more likely than others, this is not the case! For instance, if 75% of patients pick branch A, you can achieve 75% accuracy just by always picking that branch- the algorithm would look significant even though it's useless) $\endgroup$
    – David Robinson
    Commented Mar 8, 2013 at 21:47
  • $\begingroup$ David, the reason I generated the random vectors initially is because the number of branches was not the same overall, so I did it to determine a chance level. You are correct that in some situations a natural bias could give meaningless results. That is why I am comparing my approach to chance and also to other strategies that are cognitively motivated. $\endgroup$
    – Makoto
    Commented Mar 9, 2013 at 13:09
  • $\begingroup$ @Makato- I'm afraid your approach is misled. Performing random predictions wouldn't reveal such a bias- random predictions (of 1/3 each) will always have an accuracy of 1/3, just as your random test did. The best that a dumb algorithm would do is the maximum frequency of a branch- for example, if the branches are .5-.3-.2, the best you could do is .5. You need to use that value rather than the value from random predictions. $\endgroup$
    – David Robinson
    Commented Mar 9, 2013 at 17:01
  • $\begingroup$ My goal is not to reveal a bias and I don't even have data that could be used to find a bias in branch usage, so I think random chance is a fair baseline for my application. $\endgroup$
    – Makoto
    Commented Mar 10, 2013 at 1:45
  • $\begingroup$ @Makato: I think you misunderstand. Almost any algorithm will start to slant to pick the most common outcomes with higher probability, but that is an extremely low standard to hold the algorithm to. Such a test would make it appear that the algorithm was doing far better than chance (with an extremely low p-value) even if all it was doing is choosing one branch more often than the others, even if it were entirely ignoring the input data. $\endgroup$
    – David Robinson
    Commented Mar 10, 2013 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.