Timeline for Why are neural networks smooth functions?
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 25, 2020 at 16:01 | comment | added | Neal | ReLU neural nets with integer weights are tropical rational maps, i.e. piecewise linear. cf arxiv.org/pdf/1805.07091.pdf | |
| Jun 24, 2020 at 9:42 | vote | accept | Sean | ||
| Jun 24, 2020 at 9:41 | vote | accept | Sean | ||
| Jun 24, 2020 at 9:42 | |||||
| Jun 23, 2020 at 19:45 | comment | added | Danica | They indeed do mean smooth in the parameters, although this barely changes your discussion here: "Unlike methods like CART and MARS, neural networks are smooth functions of real-valued parameters. This facilitates the development of Bayesian inference for these models." And in the book, they only mention sigmoid or Gaussian RBF activations. (The most recent second edition was written in 2008, a few years before ReLUs became popular.) | |
| Jun 23, 2020 at 19:30 | comment | added | user20160 | Note: I have interpreted the quote as referring to smoothness w.r.t. the input. However, it's possible they're referring to smoothness w.r.t. the parameters. I'll try to edit later to address this possibility | |
| Jun 23, 2020 at 19:25 | history | answered | user20160 | CC BY-SA 4.0 |