# What exactly is neural network learning?

I am a newbie to ML. I was reading about neural networks and was confused about the learning part of the network.

Say there are some classes of images $c_1$ to $c_{k}$ and a neural network is trying to learn to classify a given image into these classes. Further assume that it achieved a very good accuracy(99.999%, whatever).

So what exactly did it learn? Did it learn the a very good approximation to the distributions of each of $c_i$? If yes, then why do we need GANs to generate new data instead of just sampling (somehow) from this network? If not, without having any idea about the distributions of $c_i$, how is the network able to distinguish the images?

Neural network learns a set of weights, so that when your data is multiplied by them and passed through some non-linear functions, they are able to predict the target variable as actually as possible.

You can argue that this is a low-level answer to your question, but at least it does not give you the false sense of what they can learn.

Neural networks can learn some reasonable things about the reality, but nothing prohibits them from learning crazy interpretations of the data they "seen". It is often the case that neural networks overfit, i.e. learn bogus features, that "work" for making predictions on training set, but do not generalize to new data. To give example, neural network trained to detect wolfs on images, learns to predict that if picture contains white spots (of snow), there must be a wolf, because all the wolfs it has ever seen in training set, were photographed in winter.

Why can't the ordinary neural networks generate new data like GANs? First of all, they are deterministic functions and you cannot take samples from deterministic functions. You can overcome this by using Bayesian neural networks that treat parameters as random variables, so that you can samples of them. Still, even in such case, if the network was not explicitly trained to learn accurate representations of the data, it does not have to be the case that if network was learned on images, you would be able to generate any meaningful images using it's weights. For example, neural network that detects cars may learn to detect wheels on the pictures, while image containing only the wheels is far from realistic image of the car.

Neural networks, like other machine learning algorithms, learn to detect patterns in your data, such that enable them to make predictions. Those can be any kind of patterns. They are more like lazy students on an exam, that when discovered that "A" answers in multiple-choice questions test are usually correct, so instead of learning to exam, they just always answer "A", without even reading the questions.

An artificial neural network (ANN) learns in your (supervised) classification problem, just like any other ML model, a function $$h_{model}$$ which approximates the 'true' unknown function $$h_{data}$$. These functions map an input image to the set containing $$c_1$$ to $$c_k$$.

In the bigger picture it is not much different from the way logistic regression learns this: the weights are fit to minimize a loss function e.g. using randomized gradient descent. Just that the ANN has multiple connected units doing this instead of doing just 'one' logistic regression.

It sometimes helps to just view ANNs as interconnected linear regressors with a non-linear activation function instead of anything else.

Neural Networks is the algorithm that tries to mimic the brain. It is designed similar to neurons of the brain(neurons are the cells in the brain). Each neurons in the brain act as a computational unit which receives input through input wires, then performs computation and output them through output wire.

Neural Networks have different layers namely input layer, hidden layer and output layer. Input layer is where we pass out input such as features of the example. Hidden layers are used to extract the implicit information of the features (i.e) the input and calculate the activation values. It is difficult to say what exactly each hidden layer is doing. Whenever we train the neural network, all its learning are stored in the weights(also called parameters). So after training whenever we use a test set, it will predict as it has previously learned all stored all its learning in the weights. (Weights are calculated by back-propagation)

• Neural networks try mimicking the brain to the same degree as helicopter tries mimicking birds...
– Tim
Nov 8, 2019 at 23:33
• And also, this does not really answer the question. Dec 11, 2019 at 8:18