In linear regression, assume we get the following equation : Y = 0.8X1+1.9X2+2.4X3+4X4.

We can interpret the linear equation: Keep the other predictors fix, one unit change in X1 will increase 0.8 units in Y in average.

Now, I use the neuralnet package in R and built a neural network model. Then, make a plot like the following picture.

plot(neural network)

From my perspective, all variables start with W such as W11,W21,W31 are the weight that previous variable or neural. The blue line and letters starts with b such as b11,b21,b31 are the bias weight.

Ignore the other coefficients I did not notice, just focus on the X1, can I write the equation of this neural network plot like:

Y = W31*W21*W11*X1 + W31*W21*W12*X1 + W31*W21*W13*X1 + W32*W21*W11*X1 + W32*W21*W12*X1 + W32*W21*W13*X1 + W32*W22*W11*X1 + W32*W22*W12*X1 + W32*W22*W13*X1

It looks quite ugly, but what I am curious is can I interpret the formula like the previous linear model ? Such as how Y changes if X1 increases one unit.

Or are there any other ways to interpret neural network plot (neuralnet) like this?


3 Answers 3


One of the issues when you introduce nonlinearities and interactions is that the change resulting in a change in a variable of interest depends on the starting value of that variable of interest and of the other variables in the model.

For instance, consider a model like $\hat y = x_1-x_2+x_1^2x_2$. If you want to know by how much $y$ changes upon changing $x_1$ by one unit, take the derivative.

$$ \dfrac{ \partial \hat y }{ \partial x_1 } = 1 + 2x_1x_2 $$

You cannot answer the question (with a single number) unless you know $x_1$ and $x_2$.

Neural networks are no different. We can take partial derivatives (making use of the chain rule) and interpret those derivatives as slopes just like normal. However, those slopes are likely to depend on the values of all variables (just like above), including the variable of interest.

Consequently, there is no simple interpretation like, “When $x_1$ increases one unit, our predicted $y$ increases by $\hat\beta_1$ units.”

If $A$ is an activation function, a simple neural network with two features and two neurons (with activation function $A$) in the hidden layer is:

$$ \hat y = \hat b_{2,1} + \hat w_{2,1}A\bigg(\hat b_{1,1}+\hat w_{1,1}x_1 + \hat b_{1,3}x_2\bigg) + \hat w_{2,2} A\bigg(\hat b_{1,2} + \hat w_{1,2}x_1 + \hat w_{1,4}x_2\bigg) $$

For a nonlinear activation function $A$, the partial derivatives with respect to $x_1$ and $x_2$ can involve both $x_1$ and $x_2$, meaning that you must know the point where you want to talk about changes in order to talk about changes.

  • $\begingroup$ Nice to meet you again! I understand what you are talking about now. The default activation function of neuralnet is 1/(1 + exp(-x)). What I am going to do is plug 1/(1 + exp(-x)) instead of A in your answer. $\endgroup$ Sep 21 at 10:23
  • 1
    $\begingroup$ That’s exactly what you would do, yes. $\endgroup$
    – Dave
    Sep 21 at 12:03
  • $\begingroup$ Hi Dave, I figure that my neural network model contains ten input variables and one hidden layers with 4 neurals in that layers. Can I conclude it is impossible to interpret the relationship directly between Y and X1 ? There are other variables such as X2 to X10. $\endgroup$ Sep 23 at 9:08
  • 1
    $\begingroup$ @nobodyishere It might be worth calculating $\dfrac{\partial\hat y}{\partial x_1}$ in the neural net I gave in my answer. $\endgroup$
    – Dave
    Sep 23 at 9:32
  • $\begingroup$ @nobodyishere notice that Dave's answer is local (it's the best linear approximation to the neural network output), so you are measuring the direction of change per unit of the input through an infinitesimal perturbation. It does not have the same property of the linear regression model you presented in your question (without interactions, polynomial terms). $\endgroup$
    – Firebug
    Sep 23 at 11:17

This is a nice question, that touches on some interesting points in the history of neural networks (which I can only briefly mention here).

First, what you say is absolutely right if and only if the nodes in your network have linear activation functions, e.g. $b_{1,1} = x_1w_1 + x_2w_2 + x_3w_3$.

However, as explained here, if your network only has linear activation functions, you can simplify the equations so that your multilayer network can be replaced with a simple network with no hidden layers where the input nodes connect directly to the outputs, and this simple network is basically just a linear regression model, $y = x_1b_1 + x_2b_2 + \dots + x_nb_n$.

This is why most useful neural networks use non-linear activation functions, and adding making the model non-linear means the equation you want can no longer be calculated: the effect of changing the value of x1 will depend on the values of all the other inputs.

What you can do, however, is compute the average effect of changing x1 in your training data quite easily: by modifying the values of x1 in your data and calculating the average change in y.

  • 1
    $\begingroup$ Very very clear explanation ! If our activation functions are non-linear, how can identify the interaction terms ? Like we can identify the interaction terms in linear regression, lm(Y~x1x2), if we know both x1 and x2 we can still make a decent interpretation. Is it possible in neural network plot ? $\endgroup$ Sep 21 at 3:11
  • $\begingroup$ If I understand your question, then yes, you can write out the non-linear formula for the change in y as you increase x1 from a given starting value, holding the other xs constant. This is what @Dave is talking about when he describes the partial derivative. As I've said, though, this is almost always too complicated to be usefully interpreted, so most people will instead explore changing the inputs to see what effect that has on the output. $\endgroup$
    – Eoin
    Sep 21 at 8:27
  • $\begingroup$ @Eoin It might be beneficial to give an example of the “average effect of changing $X_1$”. $\endgroup$
    – Dave
    Sep 21 at 17:32

You can interpret NN like that but it is pointless generally. The seeming utility in doing so for OLS is in decoupling of the partial derivative from other variables, I.e. $\partial/\partial x_1 f(x_1,\dots,x_2)=\beta_1$.

With a useful NN you won’t get this decoupling and the partial ends up being a nonlinear function of all inputs $\beta(x_1,\dots,x_2) $ ,which you may have thousands of. So, the sensitivity to the variables is not the constant and as such not a simple intuitive concept to start with. Consider also that if you add interaction and polynomial terms to the OLS the sensitivity becomes more complicated and less intuitive for a linear regression too.

Note, that your example is NN with linear activation in inner layers, not a useful contraption as it turns out.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.