Shouldn't a deep enough linear neural network (with activation functions) be able to learn anything, even recurrent and spatial patterns? I could just flattened timeseries data or image data and pass it as input to a fully connected feed forward network. Why wouldn't it be able to learn from that data. This would be inefficient but would this be possible? So Hypothetically I could replace an RNN or a CNN with a deep enough enough fully connected feed forward network?
8
-
3$\begingroup$ Isn't this a universal approximation theorem? $\endgroup$– DaveCommented Feb 28 at 18:02
-
1$\begingroup$ Maybe some things are not learnable from data? E.g. concepts like Gödel's incompleteness theorems $\endgroup$– Sextus EmpiricusCommented Feb 28 at 18:05
-
2$\begingroup$ Yes, but does Universal Approximation Theorem also apply to functions that model temporal or pictorial data? So Hypothetically I could replace an RNN or a CNN with a deep enough enough fully connected feed forward network? $\endgroup$– Zohaib HamduleCommented Feb 28 at 18:05
-
2$\begingroup$ Yes, it just becomes hard to train. Also note that you can directly represent a convolutional layer as a matrix multiplication ai.stackexchange.com/a/21874/41576 $\endgroup$– Ggjj11Commented Feb 28 at 18:15
-
1$\begingroup$ It doesn't even have to be deep! One layer is enough if you have a buncha neurons $\endgroup$– John MaddenCommented Feb 28 at 18:23
|
Show 3 more comments