Consider a Neural Network and let $L$ be it's last layer or the output layer. Also suppose we're doing Binary Classification, hence the activation function for the last layer is $\sigma$, ie. $g^{[L]}(.)=\sigma(.)$. $$\sigma(z^{[L]})=a^{[L]}= \begin{cases} 1, & \text{if}\ a^{[L]}\geq 0.5 \Rightarrow z^{[L]}\geq 0\\ 0, & \text{if}\ a^{[L]}< 0.5 \Rightarrow z^{[L]}< 0 \end{cases}$$

$$z^{[L]}=W^{[L]}a^{[L-1]}+b^{[L]}$$ here $a^{[L-1]}$ is a value composed of non-linear activation functions or $a^{[L-1]}=f(\vec{x}), \vec{x}=[x_1,x_2,...,x_n]$, the inputs, and hence using the above formulae for $z^{[L]}$, $Z^{[L]}$ or the Decision Boundary is non-linear! (Note: this is Capital Z which is used to represent for all the training examples, on the contrary Small z is used for only one training example)

Now consider Batch Normalisation, suppose we're using $\tanh$ activation function for the hidden layers. Consider two cases, first, the Parameters $\beta^{[l]}$ and $\gamma^{[l]}$ for most layers are such that most elements of $Z^{[l]}$ is lies in the non-linear region, so $Z^{[L]}$ or the Decision Boundary will be more non-linear than the second case where $\beta^{[l]}$ and $\gamma^{[l]}$ for most layers are such that most elements of $Z^{[l]}$ lies in the linear part of $\tanh$ then the decision boundary will be less non-linear.

Thus Batch-Normalisation makes the Decision Boundary more non-linear, or our model could learn more complex stuff.

Is the above reasoning correct?

Any help is highly appreciated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.