0
$\begingroup$

Consider a Neural Network and let $L$ be it's last layer or the output layer. Also suppose we're doing Binary Classification, hence the activation function for the last layer is $\sigma$, ie. $g^{[L]}(.)=\sigma(.)$. $$\sigma(z^{[L]})=a^{[L]}= \begin{cases} 1, & \text{if}\ a^{[L]}\geq 0.5 \Rightarrow z^{[L]}\geq 0\\ 0, & \text{if}\ a^{[L]}< 0.5 \Rightarrow z^{[L]}< 0 \end{cases}$$

$$z^{[L]}=W^{[L]}a^{[L-1]}+b^{[L]}$$ here $a^{[L-1]}$ is a value composed of non-linear activation functions or $a^{[L-1]}=f(\vec{x}), \vec{x}=[x_1,x_2,...,x_n]$, the inputs, and hence using the above formulae for $z^{[L]}$, $Z^{[L]}$ or the Decision Boundary is non-linear! (Note: this is Capital Z which is used to represent for all the training examples, on the contrary Small z is used for only one training example)

Now consider Batch Normalisation, suppose we're using $\tanh$ activation function for the hidden layers. Consider two cases, first, the Parameters $\beta^{[l]}$ and $\gamma^{[l]}$ for most layers are such that most elements of $Z^{[l]}$ is lies in the non-linear region, so $Z^{[L]}$ or the Decision Boundary will be more non-linear than the second case where $\beta^{[l]}$ and $\gamma^{[l]}$ for most layers are such that most elements of $Z^{[l]}$ lies in the linear part of $\tanh$ then the decision boundary will be less non-linear.

Thus Batch-Normalisation makes the Decision Boundary more non-linear, or our model could learn more complex stuff.

Is the above reasoning correct?

Any help is highly appreciated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.