Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Background: I am developing a Python Statistics Framework, not because the ones out there are bad but because it will help me learn Python and Statistics. I have taken AP Stats, and read scattered books and articles beyond. I am more than willing to read up on whatever shiny technique will solve my problem: The issue is that I don't know the name of said technique yet.

Problem: Given two or more things, each of which takes in an X value and returns the probability of that result(Within a certain fixed range, so .00 to .01, .01 to .02 would each be separate blocks catching all the x values within that range and returning .005 and .015 respectively), and a data set, quantitatively figure out which function best matches the data. Doing the reverse(taking in a probability and returning an X value) would be a bonus.

Idea: Be able to compare Logistic Regression, a Data Tree, and "If yes within the past 3 years then .8 else .01" style predictions.

Is there a sane way to do this? Thank you.

share|improve this question
1  
Could you post a short table showing what a sample of your data look like (what's in columns, what's in rows)? –  rolando2 Jan 29 '12 at 1:56
add comment

2 Answers

The usual tool for comparing different models that predict a probability is the deviance. The simple version of the deviance for your case is for each outcome you take the log of the predicted probability of that outcome ($log(\hat{p})$ for successes and $log(1-\hat{p})$ for failures), then add all of those log values up for a given model (note that this will be negative), then multiply that sum by $-2$. The smaller the deviance the better the model fits (a model that predicted perfectly would have a deviance of $0$).

Read the Wikipedia article and the links on that page for more detail.

share|improve this answer
add comment

You said

Problem: Given two or more functions, each of which takes in an X value and returns the probability of that result.

That means the two functions are pdfs.

and a data set, quantitatively figure out which function best matches the data. Doing the reverse(taking in a probability and returning an X value) would be a bonus.

This is equivalent to checking the data for best fitting probability distribution, so yes, this is sane enough.

Idea: Be able to compare Logistic Regression, a Data Tree, and "If yes within the past 3 years then .8 else .01" style predictions.

umm, that is a different thing altogether. data tree and "If yes within the past 3 years then .8 else .01" style predictions are equivalent. The first can be compared with rest two wrt accuracy.

share|improve this answer
    
Edited question: I was thinking of functions as things that take in an input, and return an output. I had forgotten that I was dealing with math now, and I have fixed it. –  Keller Scholl Dec 29 '11 at 17:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.