In many epidemiology papers I'm reading, they estimate odds ratio and confidence intervals using a multivariate logistical regression. For example:

We examined the association between young age at first sexual intercourse (age 14 years) and subsequent risk-taking behaviours by using multivariate logistic regression by which odds ratios (ORs) and the corresponding 95% confidence intervals (95% CIs) were estimated (SAS statistical package version 9.1).

However, isn't the fact that logistic regression is a linear classifier a problem for doing that? How does the logistic regression take into account variables that interact in a complicated way (i.e. together have a big impact but separately do not)?

  • 1
    $\begingroup$ You are questioning whether a logit model is complicated enough, but it it not an intrinsic characteristic of logistic regression. At least for *variables that interact in a complicated way *, one can specify as many interactive relationships (2 ways, 3 ways, or even more) to account for these complexity even for logit. $\endgroup$
    – Carl PCH
    Mar 11 '17 at 22:15
  • 1
    $\begingroup$ but then you have to know these beforehand! What if it is thousands of variables? $\endgroup$ Mar 11 '17 at 22:16
  • $\begingroup$ @carl_pch and regardless, the boundary will always be linear, right? $\endgroup$ Mar 11 '17 at 22:16
  • 1
    $\begingroup$ Logistic regression isn't a classifier. stats.stackexchange.com/questions/127042/… It's a regression model, which is important to remember because regression models get you access to much more than just predictions. General classifiers are useful for prediction but not much else: hypothesis tests, effect sizes, and odds rations and so on. $\endgroup$
    – Sycorax
    Mar 11 '17 at 22:36
  • 1
    $\begingroup$ Yeah, you've correctly surmised that it's a linear model. But your fixation on nonlinearity misses the point. To answer "if used in something non-linear, what happens," see the logistic regression estimated as part of this post. stats.stackexchange.com/questions/164048/… i.e. the model is misspecified when variables are missing. $\endgroup$
    – Sycorax
    Mar 11 '17 at 22:56

Your criticism of logistic regression

Still, linear boundary, if used in something non-linear, what happens?

is based on a mischaracterization of the model.

While the functional form of the logistic regression model is often written

$$ f(x_1, x_2, \ldots, x_k) = \frac{1}{1 + \exp(- \sum_j \beta_j x_j )} $$

It would be more accurately, yet verbosely, written as this

$$ f(x_1, x_2, \ldots, x_k) = \frac{1}{1 + \exp(- \sum_j \beta_j \ f_j(x_1, x_2, \ldots, x_k))} $$

The prediction function is a linear combinations of arbitrary functions of the variables. This means it does not in general yield a linear decision boundary.

You also protest

but then you have to know these beforehand!

in reference to transformations of the predictors, but this is also false in general.

There are plenty of techniques used to incorporate determining the shape of non-linearities into the fitting of a logistic regression. It suffices to specify a general non-linear shape in terms of a basis expansion. For example, a basis of smoothing splines for one or multiple predictors can represent a large space of potential non-linear shapes, and there is also an appealing variational condition which characterizes them, in some sense, as the best you could possibly do.

Logistic regression, in a skilled practitioners hands, and when combined with meta-tools like the bootstrap, cross validation, and regularization, can be used to ask and answer more precise questions than an ad-hoc machine learning model (nothing against machine learning models, they are just a different sort of tool). It is much easier to take apart and look inside a logistic regression than a gradient boosted model, this allows for statistical inference and qualitative learnings.

Even the most powerful of machine learning type methods pay their respect to logistic regression in their choice of structure and loss function. A gradient boosted classifier can be expressed as:

$$ f(x_1, x_2, \ldots, x_k) = \frac{1}{1 + \exp(- \sum_j \beta_j \ T_j(x_1, x_2, \ldots, x_k)) } $$

where $T_j$ is a regression tree. The boosted model is fit to the same loss function as logistic regression, so the only change in structure is in the choice of basis expansion. It's quite reasonable to think of a gradient boosted classifier as just a logistic regression with trees as basis functions, and a greedy algorithm to choose the successive basis elements.

  • $\begingroup$ can you indicate good material for me to learn more about this? $\endgroup$ Mar 12 '17 at 12:45
  • $\begingroup$ Sure. Frank Harrel's "Regression Modeling Strategies" speaks in practical detail about capturing non-linearities in regression. The "Generalized Additive Model" book and "Elements of Statistical Learning" have lots of mathematical details and exercises. In the realm of freely available, I like Cosma Shalizi's "Advanced Data Analysis from an Elementary Point of View", which has lot's of real talk on regression. $\endgroup$ Mar 12 '17 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.