I am creating a machine learning application which will utilize logistic regression (though I haven't ruled out bayesian regression). I have multiple predictor variables that I believe to be non-orthogonal. But let me emphasize here that I believe my question is intrinsically different from those asked here regarding collinearity, as I am not too concerned about collinearity skewing my results (should I be?)

More what I am concerned about is whether logistic regression is powerful enough to take into account that there is some relationship between different features/dimensions. For instance, in text classification the word "base" would have a dramatically different impact when seen with the word "acid" (chemistry context) than with the word "structure" (engineering).

Likewise, a measurement of an ambient temperature of 90degrees fahrenheit in Houston, TX wouldn't be statistically significant unless that temperature was registered mid January (don't worry our climate isn't that screwed up yet!).

Whether, it is text classification or some other classification, are there any methods for aiding the model in determining when two features/dimensions are related?


I am currently reading up on something called n-gram, which looks promising for text classifications, but is there something similar to this for classification regarding continuous values or constant values?


  • $\begingroup$ Is there a particular reason you have chosen to use logistic regression? As you note, some of your problems are examples of classification problems. These may be better suited to learning algorithms such as random forests, SVMs or deep learning. You could quite feasibly have a discrete binned numeric predictor value, which might serve your needs better than an ill fitting regression function. $\endgroup$ Jan 17, 2015 at 16:08
  • $\begingroup$ I was under the impression that logistic regression (not to be confused with linear regression) is explicitly for binary classification problems. As for the reasoning behind it, I chose the two examples above for simplicity, but in actuality, once I get a deeper understanding of machine learning and stats, I would like to move onto more predictions of somewhat more stochastic systems and therefore thought classifiers like SVM might fail me and I'd be better off with a model that works of probability distributions $\endgroup$ Jan 17, 2015 at 16:12
  • $\begingroup$ @image_doctor Logistic regression is indeed a (confusingly named) classification algorithm. $\endgroup$
    – Danica
    Jan 17, 2015 at 17:49

2 Answers 2


I think what you need is interaction terms. For example, in your example about Houston, TX, temperature and date are interacting. We can only say that there is something wrong with the climate only if both temperature is above a threshold and it is mid-January. A model that has temperature and date as separate inputs will not be able to discover that interaction because the effect of these predictors on the outcome are not independent. However, if you add an interaction term (e.g., temperature*date), the regression model can capture that dependence. The Wikipedia page on interaction might be useful: http://en.wikipedia.org/wiki/Interaction_(statistics)

  • $\begingroup$ Thank you for bringing the interaction term up. The concept that simply multiplying two distinct features together can give the algorithm a sense of interdependence is quite mind boggling. By chance, have you come across any good articles on feature extraction/selection or learning theory in general that might help with visualizing how the learning algorithm uncovers links. I have a fair understanding of the math in terms of the sigmoid function and likelihood of parameters, but being able to do the math and visualizing whats going on are two different feats completely. Thanks again! $\endgroup$ Jan 18, 2015 at 21:20

Not really; logistic regression is a linear model, in which the outputs are a monotonic function of $\sum_i w_i x_i$ (where $x_i$ is the $i$th feature and $w_i$ the corresponding weight).

How you deal with these effects depends on the problem you're trying to solve and how much effort you want to put into it:

  • In text classification, one way to resolve the different meaning of "base" when seen with "acid" than with "structure" is word-sense disambiguation. You could then treat the two senses of the word as completely separate entities. n-grams don't really solve this problem, though they can sort of help with it a bit.

  • For the hot-in-January situation, you might represent the variable as difference from a seasonal mean, or something like that.

  • You could try to account for any such relationship using nonlinear methods. Kernel methods, deep neural networks, Bayesian modeling, or density estimation-based approaches might all be reasonable, depending on what exactly you're trying to do.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.