How do I compare multiple arbitrary predictions for a given data set?

Question

Background: I am developing a Python Statistics Framework, not because the ones out there are bad but because it will help me learn Python and Statistics. I have taken AP Stats, and read scattered books and articles beyond. I am more than willing to read up on whatever shiny technique will solve my problem: The issue is that I don't know the name of said technique yet.

Problem: Given two or more things, each of which takes in an X value and returns the probability of that result(Within a certain fixed range, so .00 to .01, .01 to .02 would each be separate blocks catching all the x values within that range and returning .005 and .015 respectively), and a data set, quantitatively figure out which function best matches the data. Doing the reverse(taking in a probability and returning an X value) would be a bonus.

Idea: Be able to compare Logistic Regression, a Data Tree, and "If yes within the past 3 years then .8 else .01" style predictions.

Is there a sane way to do this? Thank you.

Answer 1

You said

Problem: Given two or more functions, each of which takes in an X value and returns the probability of that result.

That means the two functions are pdfs.

and a data set, quantitatively figure out which function best matches the data. Doing the reverse(taking in a probability and returning an X value) would be a bonus.

This is equivalent to checking the data for best fitting probability distribution, so yes, this is sane enough.

Idea: Be able to compare Logistic Regression, a Data Tree, and "If yes within the past 3 years then .8 else .01" style predictions.

umm, that is a different thing altogether. data tree and "If yes within the past 3 years then .8 else .01" style predictions are equivalent. The first can be compared with rest two wrt accuracy.

Answer 2

The usual tool for comparing different models that predict a probability is the deviance. The simple version of the deviance for your case is for each outcome you take the log of the predicted probability of that outcome ($log(\hat{p})$ for successes and $log(1-\hat{p})$ for failures), then add all of those log values up for a given model (note that this will be negative), then multiply that sum by $-2$. The smaller the deviance the better the model fits (a model that predicted perfectly would have a deviance of $0$).

Read the Wikipedia article and the links on that page for more detail.