9
$\begingroup$

Let's say I am building a logistic regression model where the dependent variable is binary and can take the values $0$ or $1$. Let the independent variables be $x_1, x_2, ..., x_m$ - there are $m$ independent variables. Let's say for the $k$th independent variable, the bivariate analysis shows a U-shaped trend - i.e., if I group $x_k$ into $20$ bins each containing roughly equal number of observations and calculate the 'bad rate' for each bin - # observations where y = 0 / total observations in each bin - then I get a U shaped curve.

My questions are:

  1. Can I directly use $x_k$ as input while estimating the beta parameters? Are any statistical assumptions violated which might cause significant error in estimating the parameters?
  2. Is it necessary to 'linearize' this variable through a transformation (log, square, product with itself, etc.)?
$\endgroup$
  • 3
    $\begingroup$ Is $X_k$ one of the predictors you want to include in the model? It sounds like you're saying that $P(Y=1)$ is a U-shaped function of $X_k$. A U-shaped curve can often be well approximated by a quadratic function (especially near the valley) - have you considered including a linear and quadratic term in that variable into the model? $\endgroup$ – Macro Jul 17 '12 at 17:09
  • $\begingroup$ @Macro thanks for your suggestion. I have seen some modellers fitting a piecewise-linear function (assuming a sharp U shape) - where each line is estimated from the data with the breaks determined visually and then the output of the linear equation is provided as an input to the logistic regression. I am not a big fan of the approach though. $\endgroup$ – Mozan Sykol Jul 17 '12 at 18:31
12
$\begingroup$

You would want to use a flexible formulation that would capture non-linearity automatically, e.g., some version of a generalized additive model. A poor man's choice is a polynomial $x_k$, $x_k^2$, ..., $x_k^{p_k}$, but such polynomials produce terrible overswings at the ends of the range of their respective variables. A much better formulation would be to use (cubic) B-splines (see a random intro note from the first page of Google here, and a good book, here). B-splines are a sequence of local humps:

http://ars.sciencedirect.com/content/image/1-s2.0-S0169743911002292-gr2.jpg

The height of the humps is determined from your (linear, logistic, other GLM) regression, as the function you are fitting is simply

$$ \theta = \beta_0 + \sum_{k=1}^K \beta_k B\Bigl( \frac{x-x_k}{h_k} \Bigr) $$

for the specified functional form of your hump $B(\cdot)$. By far the most popular version is a bell-shaped smooth cubic spline:

$$ B(z) = \left\{ \begin{array}{ll} \frac14 (z+2)^3, & -2 \le z \le -1 \\ \frac14 (3|x|^3 - 6x^2 +4 ), & -1 < x < 1 \\ \frac14 (2-x)^3, & 1 \le x \le 2 \\ 0, & \mbox{otherwise} \end{array} \right. $$

On the implementation side, all you need to do is to set up 3-5-10-whatever number of knots $x_k$ would be reasonable for your application and create the corresponding 3-5-10-whatever variables in the data set with the values of $B\Bigl( \frac{x-x_k}{h_k} \Bigr) $. Typically, a simple grid of values is chosen, with $h_k$ being twice the mesh size of the grid, so that at each point, there are two overlapping B-splines, as in the above plot.

$\endgroup$
  • 2
    $\begingroup$ How can you tell that he needs something this complicated? Certainly a quadratic term in a covariate x that appears to have that shape could be incorporated in the logistic regression model just the way the OP seems to want it. $\endgroup$ – Michael Chernick Jul 17 '12 at 19:49
  • 1
    $\begingroup$ @MichaelChernick Yes I accepted the answer as it taught me a new concept, but I will probably not need such a complicated solution. $\endgroup$ – Mozan Sykol Jul 17 '12 at 19:58
  • 3
    $\begingroup$ @Michael That's an important point. I am encouraged, though, by a comment by the OP mentioning an ad hoc piecewise linear fitting procedure. Splining works in the same spirit but with greater flexibility and rigor. Quadratic terms might work but that seems like a lot to hope for. $\endgroup$ – whuber Jul 17 '12 at 19:58
  • $\begingroup$ Looks more like piecewise constant to me: the mean value of the response within a bin is equivalent to the regression on the dummy variable/indicator of that bin, and that's piecewise constant... @MichaelChernick: there's no telling, generally, but I am yet to see an application where B-splines would be inferior to polynomial fit. $\endgroup$ – StasK Jul 19 '12 at 5:05
  • $\begingroup$ @StasK If inferior means that it dpesn't fit the data as well then I think a small sample size would favir a simple polynomial. $\endgroup$ – Michael Chernick Jul 19 '12 at 9:57
5
$\begingroup$

Just like linear regression, logistic regression and more generally generalized linear models are required to be linear in the parameters but not necessarily in the covariates. So polynomial terms like a quadratic that Macro suggests can be used. This is a common misunderstanding of the linear term in generalized linear models. Nonlinear models are models that are nonlinear in the parameters. If the model is linear in the parameters and contains additive noise terms that are IID the model is linear even if there are covariates like X$^2$ log X or exp(X). As I now read the question it seems to be edited. My specific answer would be yes to 1 and not necessary to 2.

$\endgroup$
1
$\begingroup$

Another viable alternative that the modeling shop I work for routinely employs, is binning the continuous independent variables and substituting the 'bad rate'. This forces a linear relationship.

$\endgroup$
  • $\begingroup$ I've also seen this. How do you choose the bins? $\endgroup$ – Adam Ryczkowski Oct 1 '15 at 13:01
  • 1
    $\begingroup$ There are a variety of discretization methods you can use. R has the disco package. I created my own algorithm that recursively splits a continuous variable based on information value. I put it in an R package here: github.com/Zelazny7/binnr (work in progress!). I also would substitute the weight-of-evidence instead of the mean. When paired with LASSO regression the results are fantastic! $\endgroup$ – Zelazny7 Oct 2 '15 at 2:50
  • $\begingroup$ Thank you! Can you compare the binnr algorithm with the CRAN's smbinning? $\endgroup$ – Adam Ryczkowski Oct 2 '15 at 9:47
  • $\begingroup$ 'binnr' is written in C and is very fast. It also supports real world conditions like override values and monotonicity. It is possible to tell 'binnr' to only make discrete cuts when the bad rates are monotonic. 'smbinning' uses conditional inference trees which, in my experience, take a long time on decent sized data sets because they rely on permutations. 'binnr' also allows you to interactively make adjustments to the bins with a command line interface $\endgroup$ – Zelazny7 Oct 2 '15 at 12:22
  • $\begingroup$ Did you use it in production? Are you aware of any hidden bugs or unchecked corner cases? I think about using it for a large customer and I will have only 1 day to test it before applying your code. $\endgroup$ – Adam Ryczkowski Oct 2 '15 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.