1
$\begingroup$

A lot of times I've read that with a Single Layer Perceptron (SLP), you can only learn linear functions. But what if we use a generalized linear model?

Imagine we only have one input feature $x_1$. The function the SLP would learn would be of the shape: $$ y = w_0 + w_1*x_1 $$ Here the SLP has 2 input neurons and one output neuron (we do regression). But what if we create a new feature $x_2$ which we generate from $x_1^2$. We simply add another input neuron to the SLP. The function the SLP would now learn looks like: $$ y = w_0 + w_1*x_1 + w_2*x_1^2 $$ This is a quadratic function and not a linear function anymore?

Can anyone explain this to me?

Can we say that with a SLP we can learn non-linear function if we define the base-functions (e.g. $x_1^2$) ourselves. In contrast, a MLP would learn these on its own? Thanks.

$\endgroup$
1
$\begingroup$

In your examples (generalized linear models, or using polynomial terms as predictors; note, these are not the same thing as your question seems to imply), the underlying model is still linear.

In a sense, you've added a pre-defined layer that turns your non-linear problem into a linear one before feeding it to your perceptron. This is the same way that these sorts of models are fit using traditional methods. The relative "magic" of generalized linear models, or polynomial regression, is that they are, in fact, linear models even though the relationships between predictors and response variables are nonlinear.

$\endgroup$
1
$\begingroup$

Just to add onto Bryan Krause's answer, you seem to misunderstand what the linear in linear model refers to. A regression model is called linear if the model parameters only appear as linear terms. In your example, $$ y=w_0+w_1x_1+w_2x_1^2 $$ the parameters are $w_0, w_1,$ and $w_2$ - and each parameter only occurs as a first-order term, regardless of the basis component it parameterizes. If it makes it easier to see, you can basically rewrite the equation by setting $x_2 = x_1^2$ and even though your feature $x_2$ is completely determined by the value of $x_1$, the underlying model hasn't changed.

Contrast your model above with this model (note that here $x_2 \neq x_1^2$): $$ y = w_0 + w_1^2x_1 + w_2x_1^2 + w_1w_2x_2 + w_3x_1x_2. $$

This model is non-linear because the parameters no longer only occur as linear factors - we have $w_1^2$ and $w_1w_2$ as coefficients of the basis terms $x_1$ and $x_2$, respectively. If you were to come up with an estimate for those two coefficients as functions of the data, they would now influence each other - you cannot write your function as a linear combination of the $w_i$ terms any longer. Of course, you could possibly come up with some kind of linearizing transformation of the second model, but that's another topic.

$\endgroup$
2
  • $\begingroup$ Nice! Really good answer ,thank you:). Is it the same for linear discriminant? Or is that linear in x? $\endgroup$ – user273484 Feb 12 '20 at 8:08
  • $\begingroup$ Yes, but LDA has some other assumptions (normality) which bring about the linearity. $\endgroup$ – Don Walpola Feb 12 '20 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.