Neural Networks (NNs) are tricky to play with when it comes to regression problems. Often the output unit of a neural network contains linear activation. However, many posts suggest using ReLU activation for the hidden layers (for instance, look at this question).
Now let's consider a problem $(X,y)$, where $X=(x_1,x_2)$ and $y \in \mathbb{R}$. Consider a NN with a single hidden layer with two neurons, each with ReLU activation. Let the weights be denoted as:
Inputs - Hidden unit$_1$: $w_{11}, w_{12}$
Inputs - Hidden unit$_2$: $w_{21}, w_{22}$
Hidden layer - Output: $v_1, v_2$.
(no bias is involved anywhere.)
The following computations are straightforward. $ReLU(z) = max(0,z)$ (in case you don't know)
$a_1 = ReLU(z_1) = ReLU(w_{11}x_1 + w_{12}x_2) = either~0~or~w_{11}x_1 + w_{12}x_2$
$a_2 = ReLU(z_2) = ReLU(w_{21}x_1 + w_{22}x_2) = either~0~or~w_{21}x_1 + w_{22}x_2$
In the end, the output from the network will be one of the following four:
- $\hat{y} = 0$; when both $a_{1,2} = 0$
- $\hat{y} = v_2(w_{21}x_1 + w_{22}x_2)$; when $a_1=0~and~a_2 \neq 0$
- $\hat{y} = v_1(w_{11}x_1 + w_{12}x_2)$; when $a_1 \neq 0~and~a_2=0$
- $\hat{y} = v_1(w_{11}x_1 + w_{12}x_2) + v_2(w_{21}x_1 + w_{22}x_2)$; when $a_{1,2} \neq 0$
In all the cases above, $\hat{y}$ can be written as $\hat{y} = U_1x_1 + U_2x_2$, where $U = vw$.
So, does this mean that using ReLU activation in the hidden layers of a deep network, the network would become a linear perception (or linear regression)?
