Consider a feed-forward neural net with multiple input vectors $x_1, x_2, ..., x_N$ whose final output $y$ is differentiable with respect to those inputs. Assume the learned representations of each input are combined to produce the final output $y$ by addition.

Suppose we remove the non-linear activations (and any other non-linear function applied between layers) so that the output becomes linear with respect to the inputs. Then the output has the form $y=b+\sum_{i=1}^Nx_iW_i$ for tensors $b$ and $W_i, i\in\{1, ..., N\}$, and $dy/dx_i = W_i^T$ if I remember my matrix calculus.

Can the non-linear activations or functions, and in particular RELU or softmax, change this behavior so that $dy/dx_i$ involves some $W_{j\neq i}$ or $x_{j\neq i}$?

up vote 1 down vote accepted

You can make the problem really simple: let $y = f(w_1x_1 + w_2x_2)$ for scalars $w_i, x_i$ and nonlinear function $f$. The derivative with respect to $x_1$ is $w_1f'(w_1x_1 + w_2x_2)$, which depends on everything.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.