In chapter 8 section 8.7.1 it tries to explain batch normalization. In the second paragraph of that section it tells us to consider the simple example:
$$ \hat{y} = x w_1 ... w_i ... w_l $$
and then claims:
The output $\hat y$ is a linear function of the input x, but a nonlinear function of the weights $w_i$.
which I believe is incorrect, however, I wanted to make sure I was not wrong myself. I will argue here why I think its wrong.
Recall the definition of a linear function to be $ f(x+y) = f(x) + f(y) $. Now lets consider the function they wrote and see if it obeys that property. First it clearly obeys it with respect to x:
$$ \hat{y}(x) + \hat{y}(y) = x w_1 ... w_i ... w_l + y w_1 ... w_i ... w_l $$
then by factoring out $w_1 ... w_i ... w_l$ we get:
$$ \hat{y}(x) + \hat{y}(y) = ( x + y ) w_1 ... w_i ... w_l = \hat{y}(x + y) $$
One can do nearly an identical proof but with respect to $w_i$:
$$\hat{y}(w_i) + \hat{y}(w'_i) = x w_1 ... w_i ... w_l + x w_1 ... w'_i ... w_l $$
but instead by factoring everything first from the left and then from the right. Similarly to why $abc+ab'c = a(bc + b'c) = a(b + b')c$ is true. Do that and we get:
$$\hat{y}(w_i) + \hat{y}(w'_i) = (x w_1 ... )(w_i ... w_l + w'_i ... w_l) = (x w_1 ... )(w_i + w'_i)( ... w_l) = \hat{y}(w_i + w'_i)$$
which yields the desired result (that $\hat y$ linear wrt to $w_i$).
From the above argument I can't see why they'd say its non-linear. Maybe I have a misunderstanding what they are trying to say? Or is there a small mistake on the draft of the book? If its not to much to ask, can a potential answer try to address how or in what waywhy is my proof wrongis wrong?
