Are neural networks linear models?

I get that activation functions are what introduces non-linearities into a neural network model. But what is confusing is that the parameters we are estimating are still linear. Neural networks seem to be, just a stacking of multiple Generalized Linear Models in that regard. Where each "activation function" is just the equivalent of a link function between our linear predictors and eventually the data.

Edit: The definition of linear I'm referring to is the one that GLMs appeal to which refers to its parameters (Weights) being linear. It was first stated by McCullagh and Nelder (1982) suggesting that the parameters affect the distribution of y only through linear combination: wx + b and then mapped to the data with a link function. (Introduction to GLMs, linear model and Nonlinear vs. generalized linear model: How do you refer to logistic, Poisson, etc. regression?)

In the case of a Neural Network we are estimating linear parameters, and we are applying linear combinations with an activation or arguably equivalent "link function". So, unless somehow the composition of multiple GLMs stacked together doesn't qualify as a linear model anymore, it seems that this would make NN classify under a linear model category i.e. a stacking of multiple general linear models.

• ANNs are clearly far from the probabilistic setting of linear models. It is true that each neuron is very much analogue to statistical linear models, but of course, a unical model made of a stack of many neurons (which is an ANN) is clearly not a linear model, nor in theory nor in practice. Nov 6 '19 at 20:49
• @Jose I think you'll be able to answer this question if you are a bit more precise in your meanings. When you describe a GLM as a "linear model," what definition of "linear" are you using? When you attempt to apply that same definition of "linear" to a neural network, does it fit the definition? Why or why not?
– Sycorax
Nov 6 '19 at 20:53
• At best, they are piecewise-linear with respect to the input. They are not linear with respect to the inner weights however (in the sense of applying a link function on a linear combination), due to the stacking nature of layers. In a GLM, they are linear. Nov 6 '19 at 23:37
• Thanks @AlexR. This hints more to what I'm trying to find out, how exactly does the stacking nature of layers make the inner weights non-linear? Nov 7 '19 at 4:55
• It seems just logical failure from a misunderstanding of what those authors saying. GLM is not linear but "generalized" linear (or weakly nonlinear). In simple terms, Weights, predictors, and linear predictor have a linear relationship, but linear predictor and response have a nonlinear relationship in GLM because of the link function. There is a proof that NN with a single hidden layer (or "generalized" stacked GLMs) can approximate any smooth nonlinear function. Thus, NN is not a linear model, not a generalized linear model, not even a stacked GLMs. Some NNs may be "generalized" stacked GLMs Jun 6 '20 at 14:50