Neural Networks (NNs) are tricky to play with when it comes to regression problems. Often the output unit of a neural network contains linear activation. However, many posts suggest using ReLU activation for the hidden layers (for instance, look at this question).

Now let's consider a problem $(X,y)$, where $X=(x_1,x_2)$ and $y \in \mathbb{R}$. Consider a NN with a single hidden layer with two neurons, each with ReLU activation. Let the weights be denoted as:

Inputs - Hidden unit$_1$: $w_{11}, w_{12}$

Inputs - Hidden unit$_2$: $w_{21}, w_{22}$

Hidden layer - Output: $v_1, v_2$.

(no bias is involved anywhere.)

The following computations are straightforward. $ReLU(z) = max(0,z)$ (in case you don't know)

$a_1 = ReLU(z_1) = ReLU(w_{11}x_1 + w_{12}x_2) = either~0~or~w_{11}x_1 + w_{12}x_2$

$a_2 = ReLU(z_2) = ReLU(w_{21}x_1 + w_{22}x_2) = either~0~or~w_{21}x_1 + w_{22}x_2$

In the end, the output from the network will be one of the following four:

  1. $\hat{y} = 0$; when both $a_{1,2} = 0$
  2. $\hat{y} = v_2(w_{21}x_1 + w_{22}x_2)$; when $a_1=0~and~a_2 \neq 0$
  3. $\hat{y} = v_1(w_{11}x_1 + w_{12}x_2)$; when $a_1 \neq 0~and~a_2=0$
  4. $\hat{y} = v_1(w_{11}x_1 + w_{12}x_2) + v_2(w_{21}x_1 + w_{22}x_2)$; when $a_{1,2} \neq 0$

In all the cases above, $\hat{y}$ can be written as $\hat{y} = U_1x_1 + U_2x_2$, where $U = vw$.

So, does this mean that using ReLU activation in the hidden layers of a deep network, the network would become a linear perception (or linear regression)?


1 Answer 1


No. Because there is no almost linear, there's only linear or nonlinear. Otherwise, I'd be saying that ReLU is a linear function "because it is always linear with a slope 0 or 1."

ReLU is a nonlinear function with non zero convexity. If you step back you'll see a world of difference between a line and ReLU. ReLU is cousin of a highly convex function. enter image description here Yes, its convexity is concentrated in a small region but that's also what makes it so useful too: your weights "locate" convexities in the domain, helping the NN approximate the shapes so well.

I'd also say that having a protractor is not the same as having only a ruler. You can draw complex surface much faster with a protractor than with only a ruler, because to make a simple angle with only a ruler would require quite a bit of trigonometry while with a protractor it's just a quick single operation.

enter image description here

  • $\begingroup$ Great answer. But, what is wrong with my mathematical reasoning in my question? $\endgroup$
    – Coder
    Commented Jun 15, 2020 at 0:33
  • 3
    $\begingroup$ you are dealing with a piece wise linear function, and looking at each linear piece in isolation - that's what wrong. you can take any continuous function and approximate it with a piece wise linear function, using small pieces. so in your framework of thought what NN accomplishes is that it finds the appropriate knot points and picks the right slopes for each piece. that's quite a bit and that's what a simple linear regression can't do out of the box. $\endgroup$
    – Aksakal
    Commented Jun 15, 2020 at 0:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.