Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am training a logistic regression to predict which runners are most likely to finish a grueling endurance race.

Very few runners complete this race, so I have severe class imbalance and a small sample of successes (maybe a few dozen). I feel like I could get some good "signal" from the dozens of runners who almost made it. (My training data has not only completion, but also how far the ones that didn't finish actually made it.) So I am wondering whether it is a terrible idea or not to include some "partial credit." I came up with a couple functions for partial credit, the ramp and the logistic curve, which could be given various parameters.

enter image description here

The only difference with the regression would be that I would use training data to predict the modified, continuous outcome instead of a binary outcome. Comparing their predictions on a test set (using the binary response) I had fairly inconclusive results -- the logistic partial credit seemed to marginally improve R-squared, AUC, P/R, but this was just one attempt on one use case using a small sample.

I do not care about the predictions being uniformly biased toward completion -- what I care about is correctly ranking contestants on their likelihood to finish, or maybe even estimating their relative likelihood of finishing.

I understand that logistic regression assumes a linear relationship between predictors and the log of the odds ratio, and obviously this ratio has no real interpretation if I start messing with the outcomes. I'm sure this is not smart from a theoretical viewpoint, but it might help get some additional signal and prevent overfitting. (I have almost as many predictors as successes, so it may be helpful to use relationships with partial completion as a check on relationships with full completion).

Is this approach ever used in responsible practice?

Either way, are there other types of models out there (maybe something that explicitly models the hazard rate, applied over distance instead of time) that may be better suited for this type of analysis?

share|improve this question
up vote 11 down vote accepted

This seems like a job for survival analysis, like Cox proportional hazards analysis or possibly some parametric survival model.

Think about this problem in reverse from the way you're explaining it: what are the predictor variables associated with earlier distances to quitting?

Quitting is the event. The distance covered might be considered equivalent to time-to-event in standard survival analysis. You then have a number of events equal to the number of individuals who quit, so your problem with limited numbers of predictors will diminish. All those who quit provide information.

A Cox model, if it works on your data, will provide a linear predictor based on all the predictor variable values, ranking contestants in order of predicted distances to quitting.

share|improve this answer
    
Thanks for this. It sounds like you're saying that using the Cox model, the runners with the longest predicted distance to quitting are also the least likely to quit before the finishing distance, due to the proportional hazards construct. Is that accurate? Also, since you're recommending this, guessing the partial credit idea didn't strike you as well-founded? – C8H10N4O2 Feb 20 at 1:56
    
That's essentially correct. I see the incorporation of distance-to-quitting in a survival model as a way to give "partial credit" in a way that has a well-established theoretical and practical justification. Haven't worked through the details, but I suspect this accomplishes exactly what you intended, as expressed in your graph. – EdM Feb 20 at 15:28
    
I'll give it a try. Thanks! – C8H10N4O2 Feb 20 at 20:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.