Timeline for What are the main theorems in Machine (Deep) Learning?
Current License: CC BY-SA 3.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 15, 2018 at 20:28 | history | edited | Tobias Windisch | CC BY-SA 3.0 |
added 132 characters in body
|
| Jan 13, 2018 at 8:52 | comment | added | user188529 | Just placed a bounty of 100 reps on the questions. I will choose the most comprehensive answer with supporting references, just mentioning in case you want to add something more to your answer! | |
| Jan 7, 2018 at 11:44 | comment | added | DeltaIV | Yes, I interpreted that in the sense of "approximating". My point is that even if you know that you can in theory approximate any function (on a bounded hypercube) with a one hidden layer NN, in practice it's useless in many cases. Another example: Gaussian Processes with the squared exponential kernel have the universal approximation property, but they haven't eliminated all the other regression methods, also because of the fact that for some problems the number of samples needed for accurate approximation grows exponentially. | |
| Jan 7, 2018 at 9:00 | comment | added | Tobias Windisch | @DeltaIV: There is a typo in the last sentence of my previous comment: the word "learn" should actually be "approximate" (otherwise, my statement about "overfitting" would make no sense). Thank you for the hint! | |
| Jan 6, 2018 at 22:00 | comment | added | DeltaIV | It doesn't say exactly that. It only says that there exists a neural network with one hidden layer which can represent $f$, but it doesn't tell you anything about how $N$ grows with $m$, for example, or with some measure of complexity of $f$ (for example its total variation). It doesn't tell you if you can $learn$ the weights of your network, given data. You will find that in a lot of interesting cases $N$ is exponentially larger for one hidden layer networks than for multilayer (deep) networks. Which is why no one uses one hidden layer networks for ImageNet or for Kaggle. | |
| Jan 6, 2018 at 19:25 | history | made wiki | Post Made Community Wiki by whuber♦ | ||
| Jan 6, 2018 at 18:43 | comment | added | Tobias Windisch | @Olivier: I totally agree. But even though this theorem is not exclusively devoted to neural networks, I still find it statement, its rigorous proof, and its implications interesting. For instance, it says that as long as you are using an activation function that has the properties stated above, the approximative capability of the network is the same (roughly speaking). Or, it says that neural networks are prune to overfitting as you can learn a lot already with one hidden layer. | |
| Jan 6, 2018 at 17:44 | comment | added | Olivier | This theorem is a bit uninteresting as it is not particular to neural nets. Many other classes of functions share similar (and sometimes stronger) approximation properties. See for instance the Stone-Weierstrass theorem. A more interesting result would be the consistency of neural net regression in a general framework. Also, there must be known bounds on the mean generalization error in terms of the complexity of the net and the size of the training sample. | |
| Jan 6, 2018 at 17:17 | history | answered | Tobias Windisch | CC BY-SA 3.0 |