4
$\begingroup$

I'm trying to fit a two-class logistic model, using many many features. When inspecting one of the features, I binned it so I could inspect its behavior. In each bin I count the number of 'good class' occurrences, and divide by total number occurrences. I see that in the upper bins, there's a higher probability for a 'good class'.

see image.

For this kind of functional form to enter the model, I would have to add higher-order terms [for instance, by using natural splines]; and this, I'm afraid, would cause my model to over fit.

So, I thought I could 'help' the regression by explicitly dividing the variable into a few different intervals; the knots would be based on so-called 'eye-balling'. Thus, in each interval the variable could have a different linear coefficient [or I could even make it a fixed number by making the variable categorical].

I hope I've made myself clear, and sorry for the non-mathematical explanation - I'm just trying to make my point clear as possible. Thanks for the help!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
12
$\begingroup$

If you use $n$ bins, you will use $n-1$ degrees of freedom. For the same cost in degrees of freedom, you can fit $n-1$ restricted cubic splines. Thus, there should definitely not be more overfitting if you use splines.

Conversely, discretizing continuous variables is almost always a bad idea, see, e.g., here or here or here. In your case, defining the bins by "eye-balling" will tempt you to change them "just a little bit" to get a better fit... I am not saying you will give in to this temptation, but you will need to account for this leeway in model creation in some way.

Bottom line: don't discretize, use restricted splines.

Improve this answer
$\endgroup$
3
  • $\begingroup$ thanks for the quick reply; and especially for the links; they all talk about "median splits" - a term I wasn't familiar with before. $\endgroup$
    – zorbar
    Commented Jan 6, 2013 at 11:45
  • 1
    $\begingroup$ Yes, in fact they do. Median splits are the most common way of discretizing a continuous variable into two equal bins. Everything these sites warn about in median splits is also true in splitting data into more bins. And when you start to have leeway in determining cut points, you run into "optimal cut point selection", which messes up your inference; see points 10 and 11 in the third link above. $\endgroup$
    – Stephan Kolassa
    Commented Jan 6, 2013 at 12:15
  • $\begingroup$ +1 to @StephanKolassa Splines are good. Another possibility here, mentioned in the question, is polynomial terms. The curve in the question looks like it would be well fit by a cubic. One advantage of polynomial terms is that they can be easier to interpret. $\endgroup$
    – Peter Flom
    Commented Jan 6, 2013 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.