Ian Goodfellow: Input/Parameter linearity in DNNs Ian Goodfellow says at 18:20 in https://www.youtube.com/watch?v=CIfsB_EYsVI : “The input of the model to the output of the model is close to being linear or piecewise linear with relatively few pieces. The mapping of the parameters of the model to the output is highly non linear. So the parameters have highly non linear interactions and thats what makes training much harder. Thats why optimising parameters is much harder than optimising inputs.”
I don't understand how the inputs are linear and parameters non linear. Can someone pls explain this. He goes on to show an image to illustrate this as well.
 A: Welcome to StackExchange!
To understand what he is saying, think about what neural networks are designed for: They are function approximators.  That is, you are trying to model a true function $f : X \rightarrow Y$ using some representation of the function.  Linear functions take this form $f(X) = a \; X + b$ ($a$ and $b$ are parameters to be learnt from data), and neural networks take on a more complex form: Each layer $i$ computes a transformation $f_i$, and the final output is a composition of outputs from each layer $i$: $f = f_n \circ f_{n-1} \ldots \circ f_1$.  For more information, check this nice chapter: Visual 'proof' that neural networks are good function approximators.
What Ian is saying in the video is that a composition of ReLU functions at each layer (that have worked so well in deep learning empirically) is a nice representation of piecewise linear functions.  This is because the composition of piecewise linear functions is also a piecewise linear function.  Which means that you could say that the output of a deep neural network is a specific linear function as long as the input lies within a small region.  You can see this visually here:
https://cs.stanford.edu/people/karpathy/convnetjs/demo/regression.html
Use this simple network definition:
layer_defs = [];
layer_defs.push({type:'input', out_sx:1, out_sy:1, out_depth:1});
layer_defs.push({type:'fc', num_neurons:20, activation:'relu'});
layer_defs.push({type:'regression', num_neurons:1});

net = new convnetjs.Net();
net.makeLayers(layer_defs);

trainer = new convnetjs.SGDTrainer(net, {learning_rate:0.01, momentum:0.0, batch_size:1, l2_decay:0.001});

You can also see that Ian's observation will not hold true for other functions (pick sigmoid and see for yourself).

Now you can see what he meant when he said the non-linearity is because of the parameters of the network.
