5
$\begingroup$

I am confused about the criteria which determines whether a model is linear or not. As far as I understand, the following statements are equivalent :

  1. A model is linear
  2. Output class label is a linear combination of model parameters w
  3. Decision boundary is linear [a hyperplane]

If the above understanding is correct, then how is a single layer perceptron with a non linear activation like sigmoid a linear model? From this post , I understand that it fulfils the 3rd statement that the decision boundary will be linear but the output is clearly not a linear combination of w.

Can anyone please clarify this? Am stuck on this for a while now, any help would be deeply appreciated!

$\endgroup$
2
  • $\begingroup$ "Output class label is a linear combination of model parameters w" I don't really understand what this is supposed to mean. Could you give an example of a model that fulfills this criterion? $\endgroup$
    – Firebug
    Feb 1 at 10:44
  • $\begingroup$ @Firebug what I meant was that the discriminant function would be linear in w ie in a linear model, prediction y = wTx where w is weight vector and x is the input vector $\endgroup$
    – TarS
    Feb 1 at 10:58

1 Answer 1

6
$\begingroup$

In statistics, we call a model linear when the outcome is a linear combination of the parameters, meaning that you can write $\hat y_i = \sum_j x_{ij}\hat\beta_j$ (second definition). We could have something like $x_{2j} = \left(x_{1j}\right)^2$ without breaking that, which is how polynomial regression qualifies as a linear model, even though the model could be squaring or cubing terms, leading to a highly nonlinear plot.

If you then throw an activation function on there to get $\hat y_j = \dfrac{1}{1 + \exp\left(-\sum_j x_{ij}\hat\beta_j\right)}$, then $\hat y$ no longer qualifies as being a linear combination of the $\beta$ parameters, and statistics would not consider this a linear model, with the caveat that the nonlinearity can be expressed in a way that qualifies it as a generalized linear model (which some people will consider linear enough to count as somewhat of an honorary linear model).

If the machine learning classification people want to consider a classifier to be linear if the decision boundary is a hyperplane, let them give one word multiple meanings. This is math. Words can have multiple distinct meanings in math. Just consider what "normal" can mean in math.

Overall, no, those statements are not equivalent. The second and third do not imply each other. Rather, both can be taken as definitions of "linear" depending on the context.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.