Appropriate statistical test to test if probabilities are accurate I have some data that looks like this:
Prob    Outcome
0.09    0
0.10    0
0.10    0
0.11    1
0.84    1
0.99    1
0.86    1
0.78    1
0.86    1
0.00    0
etc.

i.e. a bunch a probabilities each with a single test. What statisitcal test should I use to test the hypothesis that the probabilities are correct?
Further details: The data points are combat probabilities from the game Civilization IV, and I have over 3000 of them in my set. Thus, each probaility is generated using some unknown formula from different input data, depending on the relative strengths of the units in that battle.
It has been suggested that the outcomes do not accurately reflect the probabilities given: for instance, the computer player wins more often that it should, based on the probabilites displayed, which is what we want to test.
So there is a link insofar as we assume the probabilities displayed are generated using the same formula for each line. It's this unknown formula that we want to test for consistency with the actual results.
 A: I suggest that you start with visualisation. Plot the (binned) rate of success for each probability against the probability.
A: edit:
Actually given your plot there, immediately we see that the distribution is about what it should be 

I'd further suggest looking at a plot of the difference between the arcsin of the square root of the success rate in each interval and the arcsin of the square root of claimed probability (evaluated at the mean probability in each interval) plotted against the probability, which should look like horizontal "bar" of dots scattered around zero.  You can also add error-bars to each point, since the standard deviation should be $\sqrt{\frac{1}{4n_i}}$ (where the number of trials in the $i$-th point would be $n_i$), so if you put vertical error bars on each point at $\pm\sqrt{\frac{1}{n_i}}$, about 95% of the error bars should cover zero. If there's a region where they usually don't include zero, that would indicate a bias.

Original:
The alternative that "the computer player wins more often than it should" is relatively easy to test with this data - you can just compute a mean and variance for the expected number of wins overall and so a z-test. (If you want to pursue that I can give more details)
The more general alternative (that the distribution from which your individual bernoullis-with-different-$p$'s come is not identical to the hypothesized one) is a little harder, but could be done using, for example, a chi-square-type or a G-type statistic on each individual trial (perhaps with a simulated distribution to find the p-value), or you could bin them into small intervals (with 3000 of them, say 1% wide) and simulate the distribution of such a statistic on that setup under the null.
A: I would also have a glance to the calibration plot of caret R package. If you data set is dataset, you can encode Outcome as factor and plot a calibration plot:
dataset$Outcome<-factor(dataset$Outcome, levels=c(0,1), labels=c("N","Y"), ordered=TRUE) #encoding in factor
mycalibration<-calibration(Outcome~Prob, data=dataset, class="Y)
plot(calibration) #standard plot using lattice
ggplot(calibration) #plot using ggplot2 (also showing binomial c.i.)

A calibration plot basically divides your data in buckets of predicted probabilities and plots the actual vs the predicted outcomes (ref. class='Y'). See the cuts option to specify the either the number or the single probs. 
