5
$\begingroup$

I was reading this post on quora https://www.quora.com/What-are-the-advantages-of-different-classification-algorithms It states here that :-"Logistic regression is a pretty well-behaved classification algorithm that can be trained as long as you expect your features to be roughly linear and the problem to be linearly separable" What does it mean to have features linear?. Another place this comes up in the post is "Tree Ensembles have different advantages over LogisticR. One main advantage is that they do not expect linear features" I am not sure what does linear features mean?

$\endgroup$

2 Answers 2

4
$\begingroup$

I think the term they used may not be standard term. I guess it means the decision boundary can be roughly approximated by a heyperplane. (a line in 2D space). Here are two examples for "linear feature" (A) or not (B) in 2 dimensional sapce.

enter image description here

Here is my answer to a very related question, which may be helpful for you.

Do all machine learning algorithms separate data linearly?

$\endgroup$
3
  • $\begingroup$ Thank you. So i am guessing this --->"You can do some feature engineering to turn most non-linear features into linear pretty easily" would be taking something like log transformations to get "linear features". $\endgroup$ Jan 1, 2017 at 17:31
  • $\begingroup$ @AkashMehta yes, and the link I provided is on this topic: the OP was trying to ask if all the machine learning algorithm is doing this transformation to linear $\endgroup$
    – Haitao Du
    Jan 2, 2017 at 2:32
  • $\begingroup$ Note that in case B you can introduce a new, linear feature: The distance from the centre! $\endgroup$ Jan 8, 2017 at 17:52
3
$\begingroup$

For another view of linear separability (assuming that's what linear features means) imagine you have a simple dataset with sex as a binary, categorical covariate and results as a binary outcome of some experiment.

sex   outcome
m     1  
m     1
m     0
f     0
f     0
f     0

For binary response data (i.e. like this), it's common to use a logistic regression to model, among other things, the probability of outcome = 1 given sex. However, what if your input was sex = f. Then:

$$P(\text{outcome} = 1 | \text{sex} = f) = 0$$

because we have no training examples of this ever happening. In this way, we say that the data is linearly separable per @hxd1011's image above.

In fact, if you tried to fit a logistic regression you'd likely get an error because the MLE estimates tend to infinity (unless your computer software has a parameter which stops the algorithm, usually IRLS, from continuing to looks for a minimum). If you get data like this and want to use a logistic regression, you can look into penalized logistic regression. Here is nice write up about it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.