4
$\begingroup$

Three linear machine learning algorithms: Linear Regression, Logistic Regression and Linear Discriminant Analysis.

Five nonlinear algorithms: Classification and Regression Trees, Naive Bayes, K-Nearest Neighbors, Learning Vector Quantization and Support Vector Machines.

Can someone please explain for each of these algorithms specifically why are they linear or nonlinear?

Also what would a neural network be and why?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

To start, you are mixing classification and regression here, which complicates the answer a bit, but here is the extremely short version: For classification, the model is linear if you can plot all the n features in n-dimensional space, and there is a (n-1) dimensional "line" (or plane, or hyperplane), that separates (or mostly separates) different classes. So e.g., plot height and weight on x and y, and draw a straight line where most men are on one side and most women are on the other.

In regression, a linear model means that if you plotted all the features PLUS the outcome (numeric) variable, there is a line (or hyperplane) that roughly estimates the outcome. Think the standard line-of-best fit picture, e.g., predicting weight from height.

All other models are "non linear". This has two flavors. First, you have the same basic construct, but where the "line" doesn't have to be straight. In trees, the discriminating line is a stair-step shape. E.g., If you are over 6ft and over 250lb, there is a 90-whatever percent chance you are male... Neural nets and several other algos are similar, but with potentially very complex/curvy boundaries or "lines of best fit".

The second flavor of non linear models are non-parametric. K-nearest-neighbors is an example of this. It doesn't look for a discriminating line/curve at all, and instead just looks around at the classes of its nearest neighbors.

Hope that helps!

Improve this answer
$\endgroup$
4
  • $\begingroup$ And to clarify some terminology, "regression" as I'm using it above means the outcome is numeric, while "classification" means the outcome is binary or categorical. Confusingly, the word regression also applies to certain algorithms. Linear regression algorithm is numeric, but logistic regression is actually used for classification problems. $\endgroup$
    – Paul Fornia
    Commented Jan 6, 2021 at 16:01
  • 1
    $\begingroup$ This is misleading, even if technically correct when you are precise about your feature space. Remember that the basis functions of a linear regression can be nonlinear. $\endgroup$
    – Dave
    Commented Jan 6, 2021 at 17:19
  • $\begingroup$ Yes, of course, a linear model is only linear w.r.t. the features you feed in, so if you feed in non-linear transformations of a feature, then arguably linear regression is no longer linear. Is this what you are referring to, Dave? My entire answer is generally an oversimplification. E.g., you could argue with my use of the word "most", and technically non-parametric models still form a decision boundary. Just trying to get at a high-level answer. $\endgroup$
    – Paul Fornia
    Commented Jan 6, 2021 at 17:43
  • $\begingroup$ And if we're being really picky... I'm also technically conflating data from models. My description of the linear classification problem is actually a description of linearly-seperable DATA, which is well suited for linear models (which seek to find the separating line). You can (and should) always try both types of algorithms, because it's difficult to know in advance which type will provide a more useful model. $\endgroup$
    – Paul Fornia
    Commented Jan 6, 2021 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.