Goodness of fit in logistic regression where features are not frequencies

Question

I am fitting a logistic regression model with a set of features for predicting outcomes in football games (three outcomes available: home wins, away wins or draw).

The features are such as difference in position in the league table between the home team and the away team, difference in previous 10 games number of wins... So they can be positive as well as negative.

I would like to do a goodness of fit of this features. I understand that the correct way of doing it is using the Pearson chi squared test, however this test is used when the features are observed frequencies, which is not my case.

Is there any other test that I can use, or a way of modifying my test to understand how good my features are?

Answer 1

This very much depends on what do you mean as "goodness of fit." I would suggest something along the following lines: how much better results will my model give compared to a "naive" model. A good metric is how many correct predictions you predict compared to the sample average. For instance, if the average is 333 win - 333 lose - 333 draw, in average you only get about 33% of these correct if you just assign results randomly to the games. But you can predict the results based on your model, and (we hope so) you get more of them right, say 40%. Now the goodness of fit might be (40 - 33)/(100 - 33) = 10%.

You can also use something else as your naive model, for instance it may be a model where you include the position but not information about the previous games... You may also not just care about correct prediction, but also how much off was the prediction..

Answer 2

You can use the deviance to test how well your data fits your model. If you are are using R to fit your data, then this number will be calculated for you automatically. You can check out the documentation in r here.

Edit:

I pretty sure I know what you're asking now so I can give you a more detailed answer.

The model selection process can pretty much be broken up into two different parts. There is the feature selection and then there is the part where you see how well your model fits the data. When you talk about Goodness of Fit tests you're usually talking about model fitting, at least in my experience.

Any way, if you want to measure whether or not you should keep a specific variable in your model, you want to use either filter methods or wrapper methods.

There is also a feature-selection section for scikit-learn. They give an example of how to do it in python.

Also, just to give you a paper to read if you like, here is a paper that I've been reading. It uses an entropy based method to not only determine whether or not a feature is relevant to the target class, but it also selects features that are independent of each other.

If that doesn't answer your question let me know.