3
$\begingroup$

I am fitting a logistic regression model with a set of features for predicting outcomes in football games (three outcomes available: home wins, away wins or draw).

The features are such as difference in position in the league table between the home team and the away team, difference in previous 10 games number of wins... So they can be positive as well as negative.

I would like to do a goodness of fit of this features. I understand that the correct way of doing it is using the Pearson chi squared test, however this test is used when the features are observed frequencies, which is not my case.

Is there any other test that I can use, or a way of modifying my test to understand how good my features are?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

This very much depends on what do you mean as "goodness of fit." I would suggest something along the following lines: how much better results will my model give compared to a "naive" model. A good metric is how many correct predictions you predict compared to the sample average. For instance, if the average is 333 win - 333 lose - 333 draw, in average you only get about 33% of these correct if you just assign results randomly to the games. But you can predict the results based on your model, and (we hope so) you get more of them right, say 40%. Now the goodness of fit might be (40 - 33)/(100 - 33) = 10%.

You can also use something else as your naive model, for instance it may be a model where you include the position but not information about the previous games... You may also not just care about correct prediction, but also how much off was the prediction..

Improve this answer
$\endgroup$
1
  • $\begingroup$ I am already doing that. What I am looking for is a specific test. $\endgroup$
    – Escachator
    Nov 15, 2015 at 23:31
1
$\begingroup$

You can use the deviance to test how well your data fits your model. If you are are using R to fit your data, then this number will be calculated for you automatically. You can check out the documentation in r here.

Edit:

I pretty sure I know what you're asking now so I can give you a more detailed answer.

The model selection process can pretty much be broken up into two different parts. There is the feature selection and then there is the part where you see how well your model fits the data. When you talk about Goodness of Fit tests you're usually talking about model fitting, at least in my experience.

Any way, if you want to measure whether or not you should keep a specific variable in your model, you want to use either filter methods or wrapper methods.

There is also a feature-selection section for scikit-learn. They give an example of how to do it in python.

Also, just to give you a paper to read if you like, here is a paper that I've been reading. It uses an entropy based method to not only determine whether or not a feature is relevant to the target class, but it also selects features that are independent of each other.

If that doesn't answer your question let me know.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I am actually using scikit with Python $\endgroup$
    – Escachator
    Nov 9, 2015 at 21:42
  • $\begingroup$ @Escachator Well I'm not as familiar with python, but the measure that you are looking for is definitely called "deviance". You could also try calculating the models AIC. Do you know anything about Maximum likelihood estimation? I not, I can type up a more detailed response. $\endgroup$
    – j.jerrod.taylor
    Nov 9, 2015 at 22:09
  • $\begingroup$ in scikit learn deviance corresponds essentially to log loss. but I think the questioner is asking about how to do feature selection before using logistic regression. $\endgroup$
    – seanv507
    Nov 10, 2015 at 9:04
  • $\begingroup$ @Escachator Are you asking about feature selection? Are you asking about testing the feature to see if it should be included in your model? They way you wrote the question, I thought you were asking about how your entire model fit your data. If you're asking about feature selection then the answer will be different and I can update my answer and go into feature selection techniques and suggest a few papers for you to read. $\endgroup$
    – j.jerrod.taylor
    Nov 10, 2015 at 11:13
  • $\begingroup$ @j.jerrod.taylor I am asking how to know if the features are valid in a logistic regression model, i.e. goodness of fit.I am using scikit with python, although as long as I understand the maths the answer should be fine for me. $\endgroup$
    – Escachator
    Nov 10, 2015 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.