Neural Network with linear activation function

I'm learning how to apply neural networks. In the example below, I use a linear activation function. Does this mean that the model becomes multivariate regression or is it an enhanced regression that covers discontinuous function due to layer and thresholds that control whether neurons fire up to the next ones • If you have "thresholds that control whether neurons fire," you're not using a linear activation function in the normal sense. What exactly are you using inside the network? – Dougal Aug 10 at 19:25
• This is my code parameters I apply via the nauralnet package in R nn <- neuralnet(consumption ~ ., data=scaled, hidden=c(3), algorithm = "rprop+", linear.output=TRUE, threshold=0.001) – nba2020 Aug 10 at 19:28
• Is the above example modeling interaction of input variables? How can I take that into account in the model design? – nba2020 Aug 10 at 19:30

Per the documentation, the call

nn <- neuralnet(consumption ~ ., data=scaled, hidden=c(3),
algorithm = "rprop+", linear.output=TRUE, threshold=0.001)

actually only uses a linear activation function for the final output node, the one labeled consumption in your figure. The interior hidden nodes use the default value of act.fct = "logistic", a logistic (also called sigmoid) activation function. Thus your model is capable of learning nonlinear functions.

There is no thresholding going on in your code, contrary to what you seem to think is happening. That would correspond to some different choice of activation function (perhaps a ReLU, among other options) that is zero if the input is below some threshold.

If you want to really use a linear activation function throughout, you can pass a linear function to act.fct. But then you'll just be learning a linear regression, as described by Dave's answer, with an unusual optimization scheme.

• Thank you so much for clarifying @Dougal! – nba2020 Aug 11 at 9:02
• Does the model also capture interactions between capacity and gasoline input variables? Or do I need a different architecture with e.g. 2 layers? – nba2020 Aug 11 at 9:07
• @nba2020 Yes, this model does handle interactions between capacity and gasoline (as seen by the units with lines coming in from both capacity and gasoline). It only handles relatively simple interactions, though; wider and/or deeper networks would be able to handle more complex interactions, at the cost of greater risk of overfitting. – Dougal Aug 12 at 17:08

"Multivariate regression" means that your response variable is a vector. "Multiple linear regression" is the term you mean, and using linear activation functions turns your neural network into a multiple linear regression. Your layers are linear transformations over and over, and the result of composing linear transformations is another linear transformation.

The power of a neural network is in its ability to figure out all kinds of nonlinear decision boundaries. Using ReLU activation functions, you can get two neurons to give a decision boundary in the shape of $$\vert x \vert$$. Using more neurons, you can approximate $$x^2$$ or $$x^7 - 13x^4+x^2$$ (or plenty else). Without the nonlinear activation function, you're only doing linear decision boundaries.

What you're doing with a threshold for neurons firing at all confuses me and, from the comments, others. Could you please clarify that?

• Since this is a regression, "decision boundaries" is a somewhat unusual phrase.... – Dougal Aug 10 at 19:41