3
$\begingroup$

I get that activation functions are what introduces non-linearities into a neural network model. But what is confusing is that the parameters we are estimating are still linear. Neural networks seem to be, just a stacking of multiple Generalized Linear Models in that regard. Where each "activation function" is just the equivalent of a link function between our linear predictors and eventually the data.

Edit: The definition of linear I'm referring to is the one that GLMs appeal to which refers to its parameters (Weights) being linear. It was first stated by McCullagh and Nelder (1982) suggesting that the parameters affect the distribution of y only through linear combination: wx + b and then mapped to the data with a link function. (Introduction to GLMs, linear model and Nonlinear vs. generalized linear model: How do you refer to logistic, Poisson, etc. regression?)

In the case of a Neural Network we are estimating linear parameters, and we are applying linear combinations with an activation or arguably equivalent "link function". So, unless somehow the composition of multiple GLMs stacked together doesn't qualify as a linear model anymore, it seems that this would make NN classify under a linear model category i.e. a stacking of multiple general linear models.

Improve this question
$\endgroup$
5
  • $\begingroup$ ANNs are clearly far from the probabilistic setting of linear models. It is true that each neuron is very much analogue to statistical linear models, but of course, a unical model made of a stack of many neurons (which is an ANN) is clearly not a linear model, nor in theory nor in practice. $\endgroup$
    – carlo
    Commented Nov 6, 2019 at 20:49
  • 1
    $\begingroup$ @Jose I think you'll be able to answer this question if you are a bit more precise in your meanings. When you describe a GLM as a "linear model," what definition of "linear" are you using? When you attempt to apply that same definition of "linear" to a neural network, does it fit the definition? Why or why not? $\endgroup$
    – Sycorax ♦
    Commented Nov 6, 2019 at 20:53
  • 1
    $\begingroup$ At best, they are piecewise-linear with respect to the input. They are not linear with respect to the inner weights however (in the sense of applying a link function on a linear combination), due to the stacking nature of layers. In a GLM, they are linear. $\endgroup$
    – Alex R.
    Commented Nov 6, 2019 at 23:37
  • $\begingroup$ Thanks @AlexR. This hints more to what I'm trying to find out, how exactly does the stacking nature of layers make the inner weights non-linear? $\endgroup$
    – Jose Miguel Cruz y Celis
    Commented Nov 7, 2019 at 4:55
  • $\begingroup$ It seems just logical failure from a misunderstanding of what those authors saying. GLM is not linear but "generalized" linear (or weakly nonlinear). In simple terms, Weights, predictors, and linear predictor have a linear relationship, but linear predictor and response have a nonlinear relationship in GLM because of the link function. There is a proof that NN with a single hidden layer (or "generalized" stacked GLMs) can approximate any smooth nonlinear function. Thus, NN is not a linear model, not a generalized linear model, not even a stacked GLMs. Some NNs may be "generalized" stacked GLMs $\endgroup$
    – Tae-Sung Shin
    Commented Jun 6, 2020 at 14:50

1 Answer 1

Reset to default
2
$\begingroup$

The non-linear activations (activation function) of each layer give the non-linear element. If you are not using the non-linear activation, you will get a "linear" model in some sense, the key is the activation function.

So the short answer is no neural networks are not linear models.

See here: http://neuralnetworksanddeeplearning.com/chap4.html

Improve this answer
$\endgroup$
2
  • $\begingroup$ I understand that, but at the same time the same could be said of Generalized Linear Models, which have a linear combination + a link function (which is non-linear), that maps the prediction to the data. Even then, they are still consider linear models, because of their parameters being linear. Why couldn't the same be said of NNs which also have linear params? $\endgroup$
    – Jose Miguel Cruz y Celis
    Commented Nov 6, 2019 at 23:07
  • $\begingroup$ Hi @JoseMiguelCruzyCelis. Because NNs have more than one layer. Play with this: playground.tensorflow.org/… $\endgroup$
    – Bill Chen
    Commented Nov 7, 2019 at 4:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.