6
$\begingroup$

I have a logistic regression model which predicts win/loss on amount of money paid. I run my model every two hours on new data that I acquire and use it to predict the next two hours. However, I keep finding that my model underpredicts win/loss for each amount of money paid. So I'm in this situation where I have a statistical model but it doesn't seem to predict new data as it comes in.

I'm left wondering, what do I do now? My model doesn't predict for the new data, but I need for it to do so.

What are some general strategies for when a model has no predictive power?

As a side note, I should have mentioned that I actually had two models. One which predicted win/loss on 0 to 5 \$ and another for 5.01 \$ and more. It's possible that this may have been a culprit, and I might just want to utilize a robust regression model instead. Not entirely relevant, but just thought I'd mention.

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Assuming your model complies with the assumptions of LR, I don't see much to do here except adding new predictors. $\endgroup$
    – Dominic Comtois
    Apr 10, 2012 at 15:07
  • $\begingroup$ How are you making predictions? If the predicted probability is over .5, you call it a '1' and '0' otherwise? That may not be optimal in various senses of the word. $\endgroup$
    – Macro
    Apr 10, 2012 at 16:24

3 Answers 3

Reset to default
6
$\begingroup$

What do you mean by doesn't predict? Are you implying the model is doing the same as randomly guessing?

Maybe your cutoff (for predicting a 'positive' result) is not adequate? You may way want to try producing some ROC curves based on data you currently have to choose an appropriate cutoff. You would want take into consideration the 'cost' of making a false positive as compared to a false negative when choosing this cutoff.

If you are still not doing well then your predictors are probably not associated with the response.

Improve this answer
$\endgroup$
1
  • $\begingroup$ By "doesn't predict," I mean that when I plot both the predicted win/loss probability for each amount against the actual win/loss for each amount, the line for the model predictions are not even remotely close to following the trend of the actual values. $\endgroup$
    – ATMathew
    Apr 10, 2012 at 15:35
4
$\begingroup$

A general strategy when a model has no predictive power is to start over.

But does it really have no predictive power? That is does it do no better than flipping a coin?

In general, and with only rare exceptions, models will do better on the data they were trained on then on new data.

Beyond that, some more context might help.

Improve this answer
$\endgroup$
3
$\begingroup$

What were you plotting for predicted values and actual values? The model predicts either log-odds, or some other value depending on what you ask predict to return. It could be probability. Actual values are just 0,1. One way around this is to bin your actual values over subranges of the predictor and get the means (probability of 1) or make log-odds values.

You need to specify in your question what you're asking predict to return and what the "actual" values you're comparing it to are.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I took the predicted probabilities for the predicted values, and then the probability if winning at a certain bid amount, and did a line graph to compare whether the model values (predic probs) were near the actual values. In retrospect, this seems like a horrible idea, but I'm not sure how else to assess the predictive power of a model on 'real' and 'new' data. $\endgroup$
    – ATMathew
    Apr 10, 2012 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.