Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I was wondering how NN could possibly resolve specific problems (let's say trigonometry).
So, in order to compute some more complex data, why not "propose" our NN several sorts of neurons to use (or not), such as arithmetic or calculus functions, instead of using only the "classic" neuron model (weighted sum + ReLu).
Imagine, for example, sin/cos as activation. Or making possible to divide some inputs by other ones.
And, after all, are we forced to limit the neuron ouputs to [-1; 1] ?
Neural Networks with two layers and enough neurons can theoretically approximiate every function. The reason of doing it deeper - meaning more layers - is that it became clear that those networks learn better and with less data. Those networks start to build more abstract filters with each added layer.
The whole point of doing backpropgation and deep learning is that those networks are able to learn feautres themself. For example, a CNN can learn to detect edges which are then further abstracted to more complex shape finally allwoing it to detect shapes that resamble cats.
If we would use layers of neurons that you propose we are back to feature engineering ourself. We might find that that actually helps for some problems. But the idea of the idea of software 2.0, ML, DL and other buzzwords is that we create an infrastructure (the model) and feed it with data. Given enough capacity of the model and enough qualtiy data it should learn the representation of the problem itself.
Regarding the [-1,1]: Weights can be limited to that. That‘s actually what happens when you eg use tanh as activation function. It has been shown that the saturation at around 0 and 1 for tanh is not beneficial. Therefore ReLu was adopted in many architectures. However, batch and layer normalization have shown to improve performance significantly. The reason therefore is mainly due to nummerical reasons. Important to realize is that only the reltive scale of the weights are important. Weights literally weigh the importcane of inputs. To produce a result assigning the two weights 2 and 4 to two neurons can have the same informatauon as assinging 0.5 and 1. Just a matter of scale. Nummerically speaking it is however cumbersome to handle big numbers. You can imagine for example a CNN that treats RGB images. The pixel values could be something common like 0-255 or 0-1. They could also lay between 0-10^6. For a network trained on 8 bit images with values 0-255 I would need to scale images with other images. Keep in mind, I ommited a few details in my explanation and examples to keep it simple. Also I’m somewhat cheating and try to explain the importance of weights by talking about input images. I hope the idea is, however, clear. Afterall you could use a handcraftet „neuron layer“ that scales images into the proper range.
