0
$\begingroup$

EDIT:

Following on the comment of Sycorax, I am assuming that equation (4) is an "immediate consequence" of assuming the relative linearity of $f$ under the "regime" of our inputs. I'm still interested in justifying the assumptions of the preceding paragraph, and ideally in seeing some discussion of why we assume the linear regime. (Forgive my possible misinterpretation, but if we're reaching saturation, aren't we stepping into the non-linear part? Or is the point that we're forcing ourselves out of the linear part, which is undesirable?)

I made the decision to try to push through the paper "Understanding the difficulty of training deep feedforward neural networks". (The paper is given as a reference in "Hands-On Machine Learning," in the chapter on training, and given its brevity and my math background I thought I'd try it on as an exercise.) I'm getting hung up at section 4.2.

Chiefly, I can't understand the setup being introduced, the main part I was hoping to follow closely. The authors make mention of a ``linear regime" or linear activations, which confuses me given the focus on non-linear activation functions of the previous section; is their $f$ meant to be taken as linear? How, in particular, do equations $(4)$ and $(5)$ follow? (The paragraph preceding them as justification is very unclear to me.) Is the cost function fully generic, in this case?

(Thank you for any clarifications! :) )

Improve this question
$\endgroup$
2
  • $\begingroup$ For some definition of approximately, there is an interval where the activation function is approximately linear. For relu, this is the positive part. For tanh, this is “near 0,” etc. (And the asymptotes of tanh are also approximately linear, as is the 0 part of relu.) $\endgroup$
    – Sycorax ♦
    Commented Jan 30 at 1:19
  • $\begingroup$ @Sycorax Thus, if we're in the "linear regime" still with our inputs, $f'$ is approximately constant? (That is, is your comment to justify equation $(4)$?) $\endgroup$
    – Chris
    Commented Jan 30 at 1:35

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.