I have been working in deep learning and clearly know how to train a deep NN on a large image dataset that has been labelled before.
The more in depth I go, the more I want to know the intuition behind what the weights of the neural network represent once trained.
I read this statement in this paper from Google on Bayesian Neural Networks - https://arxiv.org/pdf/1505.05424v2.pdf
It says
Inputs x are mapped onto the parameters of a distribution on Y by several successive layers of linear transformation (given by w) interleaved with element-wise non-linear transforms.
Intuitively if you imagine that you have a large set of images each labelled as either a dog or a cat (binary classification), once you train the neural network, the above statement says that for every input from the training set the neural network, once trained, has estimated the parameters of a binomial distribution (a coin toss is a binomial distribution with one parameter p indicating the prob of getting heads when the coin is tossed).
In the above dataset case, the neural network once trained on training images, has estimated the probability with which our nature would have chosen the label dog (considering dog is like heads in a coin toss), to generate any input image.
So, during the labelling of the training images, we set the probability with which nature would have selected "dog" as a label as 1, and the probability with which nature would have selected "cat" as a label as 0, for all dog images in our training data.
Similarly, we set the probability with which nature would have selected "dog" as a label as 0, and the probability with which nature would have selected "cat" as a label as 1, for all cat images in our training data.
Next, we know in a plain feedforward neural network, once trained, the weights are frozen. So this means for all training images we wont be able to get an output probability for dog label as 1 for all dog training images and neither will we be able to get an output probability for cat label as 1 for all cat images.
The objective therefore is to find a single footprint of weights that will maximize the likelihood of the dog label once we pass all the dog images in the training dataset through the trained network, and similarly maximize the likelihood of the cat label across all cat labelled images.
If you observe above, my understanding is that this is not a generative kind of a model which otherwise would have looked at how nature would have generated the training images once it selects a label as either dog or cat for every image. Instead this a discriminatory approach of given an image how do I predict the probabilities between the labels.
Just wanted to confirm if this right or wrong and what parts of this explanation does need corrections.
