Timeline for Why second order SGD convergence methods are unpopular for deep learning?
Current License: CC BY-SA 4.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 29, 2022 at 4:21 | history | edited | Jarek Duda | CC BY-SA 4.0 |
Added 2D example of SGD augumented with 2nd order information from sequence of gradients
|
| Mar 24, 2021 at 18:32 | comment | added | user2961927 | A minor reason. If an NN (or most NNs) is activated by ReLUs, second order derivative seems to have no impact | |
| Nov 29, 2020 at 11:49 | comment | added | Jarek Duda | Just recorded SGD overview lecture for the dropbox.com/s/54v8cwqyp7uvddk/SGD.pdf slides : youtu.be/ZSnYtPINcug | |
| Aug 21, 2020 at 20:44 | answer | added | Amir Gholami | timeline score: 11 | |
| Aug 15, 2019 at 7:40 | history | edited | Jarek Duda | CC BY-SA 4.0 |
Added crucial saddle-free Newton, and minimial 2nd order methods
|
| Apr 23, 2019 at 8:09 | comment | added | Jarek Duda | @Sycorax, thanks, the answer below is about K-FAC problems, pure Newton has its own (like Hessian size, its estimation and inversion, saddle attraction), but there are also cheap possibilities for 2nd order methods, like just momentum method plus 3 additional updated averages to model parabola in its single direction for smarter choice of step size: stats.stackexchange.com/questions/404545/… | |
| Apr 22, 2019 at 15:43 | comment | added | Sycorax♦ | Related: stats.stackexchange.com/questions/253632/… | |
| Apr 21, 2019 at 4:49 | history | edited | Jarek Duda | CC BY-SA 4.0 |
added 118 characters in body
|
| Feb 26, 2019 at 12:44 | vote | accept | Jarek Duda | ||
| Feb 24, 2019 at 21:00 | history | tweeted | twitter.com/StackStats/status/1099776119342010368 | ||
| Feb 24, 2019 at 11:48 | answer | added | DeltaIV | timeline score: 28 | |
| Feb 24, 2019 at 11:09 | history | edited | Jarek Duda | CC BY-SA 4.0 |
added 24 characters in body
|
| Feb 24, 2019 at 8:57 | history | asked | Jarek Duda | CC BY-SA 4.0 |