Is this intuition behind understanding what the weights of a plain feedforward neural network are representing correct? I have been working in deep learning and clearly know how to train a deep NN on a large image dataset that has been labelled before. 
The more in depth I go, the more I want to know the intuition behind what the weights of the neural network represent once trained.
I read this statement in this paper from Google on Bayesian Neural Networks - https://arxiv.org/pdf/1505.05424v2.pdf
It says 

Inputs x are mapped onto the parameters of a distribution on Y by
  several successive layers of linear transformation (given by w)
  interleaved with element-wise non-linear transforms.

Intuitively if you imagine that you have a large set of images each labelled as either a dog or a cat (binary classification), once you train the neural network, the above statement says that for every input from the training set the neural network, once trained, has estimated the parameters of a binomial distribution (a coin toss is a binomial distribution with one parameter p indicating the prob of getting heads when the coin is tossed). 
In the above dataset case, the neural network once trained on training images, has estimated the probability with which our nature would have chosen the label dog (considering dog is like heads in a coin toss), to generate any input image. 
So, during the labelling of the training images, we set the probability with which nature would have selected "dog" as a label as 1, and the probability with which nature would have selected "cat" as a label as 0, for all dog images in our training data. 
Similarly, we set the probability with which nature would have selected "dog" as a label as 0, and the probability with which nature would have selected "cat" as a label as 1, for all cat images in our training data.
Next, we know in a plain feedforward neural network, once trained, the weights are frozen. So this means for all training images we wont be able to get an output probability for dog label as 1 for all dog training images and neither will we be able to get an output probability for cat label as 1 for all cat images. 
The objective therefore is to find a single footprint of weights that will maximize the likelihood of the dog label once we pass all the dog images in the training dataset through the trained network, and similarly maximize the likelihood of the cat label across all cat labelled images.
If you observe above, my understanding is that this is not a generative kind of a model which otherwise would have looked at how nature would have generated the training images once it selects a label as either dog or cat for every image. Instead this a discriminatory approach of given an image how do I predict the probabilities between the labels.
Just wanted to confirm if this right or wrong and what parts of this explanation does need corrections.
 A: It looks good. Minor comments:

Next, we know in a plain feedforward neural network, once trained, the weights are frozen. So this means for all training images we wont be able to get an output probability for dog label as 1 for all dog training images and neither will we be able to get an output probability for cat label as 1 for all cat images.

I agree with this, although there is nothing mathematically wrong with getting an output probability of 1. It can happen. It just doesn't usually happen in practice.
The other thing to note is that treating the output of (non-Bayesian) neural networks as probability distributions is a rather heuristic (albeit common) thing to do. For instance, see this answer.

If you observe above, my understanding is that this is not a generative kind of a model which otherwise would have looked at how nature would have generated the training images once it selects a label as either dog or cat for every image. Instead this a discriminatory approach of given an image how do I predict the probabilities between the labels.

Yes, it is a discriminative approach, only caring about $p(y|x)$, rather than the more general (but harder to learn) $p(x,y)$.
