How do I compare multiple arbitrary predictions for a given data set? Background: I am developing a Python Statistics Framework, not because the ones out there are bad but because it will help me learn Python and Statistics. I have taken AP Stats, and read scattered books and articles beyond. I am more than willing to read up on whatever shiny technique will solve my problem: The issue is that I don't know the name of said technique yet. 
Problem: Given two or more things, each of which takes in an X value and returns the probability of that result(Within a certain fixed range, so .00 to .01, .01 to .02 would each be separate blocks catching all the x values within that range and returning .005 and .015 respectively), and a data set, quantitatively figure out which function best matches the data. 
Doing the reverse(taking in a probability and returning an X value) would be a bonus.
Idea: Be able to compare Logistic Regression, a Data Tree, and "If yes within the past 3 years then .8 else .01" style predictions. 
Is there a sane way to do this? Thank you.
 A: The usual tool for comparing different models that predict a probability is the deviance.  The simple version of the deviance for your case is for each outcome you take the log of the predicted probability of that outcome ($log(\hat{p})$ for successes and $log(1-\hat{p})$ for failures), then add all of those log values up for a given model (note that this will be negative), then multiply that sum by $-2$.  The smaller the deviance the better the model fits (a model that predicted perfectly would have a deviance of $0$).
Read the Wikipedia article and the links on that page for more detail.  
A: You said


Problem: Given two or more functions, each of which takes in an X value and returns the probability of that result.


That means the two functions are pdfs.


and a data set, quantitatively figure out which function best matches the data. Doing the reverse(taking in a probability and returning an X value) would be a bonus.


This is equivalent to checking the data for best fitting probability distribution, so yes, this is sane enough.


Idea: Be able to compare Logistic Regression, a Data Tree, and "If yes within the past 3 years then .8 else .01" style predictions.


umm, that is a different thing altogether. data tree and "If yes within the past 3 years then .8 else .01" style predictions are equivalent. The first can be compared with rest two wrt accuracy.
