I see many sources claim this result, however, I was not able to find a proof for it. I think this should be given in some paper or book. Can someone point me to some resources, or even better, show me the proof? Thanks in advance!

To clarify my question, for a given dataset of $\{x_i, y_i\}$, where $x_i$ represents the data point, and $y_i$ represents the labels, I want to show that using the loss function

$$\mathcal{L} = \frac{1}{N}\sum_{i=1}^{N}(f_\theta(x_i) - y_i)^2$$

makes $f_\theta(x_i)$ learn faster using gradient descent than when using the loss function

$$\mathcal{L} = \frac{1}{N}\sum_{i=1}^{N}|f_\theta(x_i) - y_i|.$$

Faster is referring to the number of iterations required to achieve

$$\frac{1}{N}\sum_{i=1}^{N}|f_\theta(x_i) - y_i| \leq \epsilon$$

with $\epsilon$ being a loss threshold. In both cases, the neural network will be initialized with the same weights.

  • 2
    $\begingroup$ Converges faster to what? Using those two regularizations can lead to different results, if you don't use any regularization, you would also get a different one, so what we are convergint to? $\endgroup$
    – Tim
    Sep 23 at 14:28
  • $\begingroup$ I updated my question. Does it make more sense now? Please let me know if any part of it is still confusing. $\endgroup$ Sep 23 at 17:44
  • 1
    $\begingroup$ It looks like your question must be predicated on some particular learning algorithm. Otherwise there is a mathematical solution in both cases that exists without regard to any sense of "speed" of "learning." Indeed, to what is "faster" referring? Number of iterations in some iterative procedure; total computational burden to find an acceptable solution; the value of $N$ needed to achieve an acceptable prediction or set of estimates; or perhaps something else? $\endgroup$
    – whuber
    Sep 23 at 17:57
  • $\begingroup$ @whuber does it make more sense now? $\endgroup$ Sep 23 at 18:02
  • $\begingroup$ That helps. But there are many versions of gradient descent; the rate depends partly on the starting value; and your termination condition is tailored to one of the loss functions, which suggests a direct comparison might not be meaningful. $\endgroup$
    – whuber
    Sep 23 at 18:06


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.