# Does use of ReLU activation in hidden layers a neural network for regression make it expensive (stacked) linear regression?

Neural Networks (NNs) are tricky to play with when it comes to regression problems. Often the output unit of a neural network contains linear activation. However, many posts suggest using ReLU activation for the hidden layers (for instance, look at this question).

Now let's consider a problem $$(X,y)$$, where $$X=(x_1,x_2)$$ and $$y \in \mathbb{R}$$. Consider a NN with a single hidden layer with two neurons, each with ReLU activation. Let the weights be denoted as:

Inputs - Hidden unit$$_1$$: $$w_{11}, w_{12}$$

Inputs - Hidden unit$$_2$$: $$w_{21}, w_{22}$$

Hidden layer - Output: $$v_1, v_2$$.

(no bias is involved anywhere.)

The following computations are straightforward. $$ReLU(z) = max(0,z)$$ (in case you don't know)

$$a_1 = ReLU(z_1) = ReLU(w_{11}x_1 + w_{12}x_2) = either~0~or~w_{11}x_1 + w_{12}x_2$$

$$a_2 = ReLU(z_2) = ReLU(w_{21}x_1 + w_{22}x_2) = either~0~or~w_{21}x_1 + w_{22}x_2$$

In the end, the output from the network will be one of the following four:

1. $$\hat{y} = 0$$; when both $$a_{1,2} = 0$$
2. $$\hat{y} = v_2(w_{21}x_1 + w_{22}x_2)$$; when $$a_1=0~and~a_2 \neq 0$$
3. $$\hat{y} = v_1(w_{11}x_1 + w_{12}x_2)$$; when $$a_1 \neq 0~and~a_2=0$$
4. $$\hat{y} = v_1(w_{11}x_1 + w_{12}x_2) + v_2(w_{21}x_1 + w_{22}x_2)$$; when $$a_{1,2} \neq 0$$

In all the cases above, $$\hat{y}$$ can be written as $$\hat{y} = U_1x_1 + U_2x_2$$, where $$U = vw$$.

So, does this mean that using ReLU activation in the hidden layers of a deep network, the network would become a linear perception (or linear regression)?

ReLU is a nonlinear function with non zero convexity. If you step back you'll see a world of difference between a line and ReLU. ReLU is cousin of a highly convex function. Yes, its convexity is concentrated in a small region but that's also what makes it so useful too: your weights "locate" convexities in the domain, helping the NN approximate the shapes so well. 