12
votes
$\sin(x)$ is a counterexample to the universal approximation theorem
The classical (Cybenko) universal approximation theorem has a condition about the function being approximated on a compact space.
On the real line, the Heine-Borel theorem says that compacts sets are ...
- 35.7k
11
votes
If dropout is going to remove neurons, why are those neurons built?
To add to @frank's answer, the reason using dropout is not the same as training a smaller network is that the neurons that are dropped out are randomly selected each time the weights are updated. So ...
- 1,107
7
votes
If dropout is going to remove neurons, why are those neurons built?
The neurons are only dropped temporarily during training. They are not dropped from the network altogether. It is just that it turns out that we get better weights if we randomly set them to zero, ...
- 8,249
2
votes
Accepted
$\sin(x)$ is a counterexample to the universal approximation theorem
A ReLU network is ultimately a piecewise-linear continuous function. Each neuron in the first hidden layer is just a shifted and scaled ReLU. Taking a linear combination of those produces a ...
- 3,231
1
vote
Accepted
A question on computational complexity of a numerical differentiation (equation (5.77)) in Bishop's Pattern Recognition and Machine Learning
its talking about units of 1 full computation ie calculating an output for a given input.
To do the numerical differentiation
for each input feature you shift each feature by +/- epsilon (leaving ...
- 4,538
1
vote
Why Deep Learning needs to be performed in Graphical representations?
Same could have been said about image data, but CNNs were born. Models that exploit the structure of the underlying data are likely to be more successful than generic methods.
- 53.2k
Only top scored, non community-wiki answers of a minimum length are eligible