2
$\begingroup$

A lot of times I've read that with a Single Layer Perceptron (SLP), you can only learn linear functions. But what if we use a generalized linear model?

Imagine we only have one input feature $x_1$. The function the SLP would learn would be of the shape: $$ y = w_0 + w_1*x_1 $$ Here the SLP has 2 input neurons and one output neuron (we do regression). But what if we create a new feature $x_2$ which we generate from $x_1^2$. We simply add another input neuron to the SLP. The function the SLP would now learn looks like: $$ y = w_0 + w_1*x_1 + w_2*x_1^2 $$ This is a quadratic function and not a linear function anymore?

Can anyone explain this to me?

Can we say that with a SLP we can learn non-linear function if we define the base-functions (e.g. $x_1^2$) ourselves. In contrast, a MLP would learn these on its own? Thanks.

$\endgroup$

3 Answers 3

3
$\begingroup$

Just to add onto Bryan Krause's answer, you seem to misunderstand what the linear in linear model refers to. A regression model is called linear if the model parameters only appear as linear terms. In your example, $$ y=w_0+w_1x_1+w_2x_1^2 $$ the parameters are $w_0, w_1,$ and $w_2$ - and each parameter only occurs as a first-order term, regardless of the basis component it parameterizes. If it makes it easier to see, you can basically rewrite the equation by setting $x_2 = x_1^2$ and even though your feature $x_2$ is completely determined by the value of $x_1$, the underlying model hasn't changed.

Contrast your model above with this model (note that here $x_2 \neq x_1^2$): $$ y = w_0 + w_1^2x_1 + w_2x_1^2 + w_1w_2x_2 + w_3x_1x_2. $$

This model is non-linear because the parameters no longer only occur as linear factors - we have $w_1^2$ and $w_1w_2$ as coefficients of the basis terms $x_1$ and $x_2$, respectively. If you were to come up with an estimate for those two coefficients as functions of the data, they would now influence each other - you cannot write your function as a linear combination of the $w_i$ terms any longer. Of course, you could possibly come up with some kind of linearizing transformation of the second model, but that's another topic.

$\endgroup$
2
  • $\begingroup$ Nice! Really good answer ,thank you:). Is it the same for linear discriminant? Or is that linear in x? $\endgroup$
    – user273484
    Feb 12, 2020 at 8:08
  • $\begingroup$ Yes, but LDA has some other assumptions (normality) which bring about the linearity. $\endgroup$ Feb 12, 2020 at 20:47
2
$\begingroup$

In your examples (generalized linear models, or using polynomial terms as predictors; note, these are not the same thing as your question seems to imply), the underlying model is still linear.

In a sense, you've added a pre-defined layer that turns your non-linear problem into a linear one before feeding it to your perceptron. This is the same way that these sorts of models are fit using traditional methods. The relative "magic" of generalized linear models, or polynomial regression, is that they are, in fact, linear models even though the relationships between predictors and response variables are nonlinear.

$\endgroup$
0
$\begingroup$

I stumbled on this question with the same thought process as the OP. The other answers clarify the underlying principles for thinking about linearity in terms of the model parameters and model inputs.

But if someone is interested in the solution (or the class of solutions) from neural networks history, similar to what the OP came up with, it falls under the category of Higher-Order Networks, for example, Sigma-Pi, Pi-Sigma etc, explained as an evolution here, where some of these networks use SLP (as the OP wondered).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.