With Sigmoid activation and Softmax normalization with cross entropy, are we fitting distributions?

Question

Let's consider I have a multi layer neural network that is doing multi class classification. So each input sample belongs to one on N classes.

Now, lets say the last layer has Sigmoid activation units. This is followed by a Softmax normalization and then a cross entropy loss.

What is the neural network intuitively trying to do with training? Is is trying to fit the distribution of each of the classes around the input training samples that are labelled with the corresponding class?

As in, if we draw a set of points on a line and then we try to fit a gaussian distribution around the points such that the likelihood of all the points is maximum - is this something the neural network is doing for each of the class?

If so, and if neural networks are basically trying to estimate the parameters of the distribution of each of the classes around the training samples of each of the classes, then why don't we just use a Sigmoid activation and a cross entropy loss? Why do we need Softmax and what is the real intuition behind this please?

I have encountered a lot of web pages that lists a lot of equations but unfortunately none of them have managed to explain this clearly.

Or is my understanding that the neural network trying to fit the distribution of each of the classes around the training examples wrong in the first place?

I would appreciate if you throw some light on the intuition behind the role of logistic regression here.

The answer will be highly appreciated!

Answer 1

Is the NN trying to estimating the distribution of the classes?

No, you are not estimating the parameters of a distribution. The idea is similar to the logistic regression. The logistic regression provides the probability the dependent variable is one or zero using a generalized linear model. In the case of the logistic distribution, the dependent variable $y$ has a bernoulli distribution and you are not in fact estimating it.

Why do we not use the sigmoid function?

You may use a sigmoid in a binary classification problem.

You may also use a sigmoid curve in a different fashion (some people actually do that). You may estimate $N$ different models where each one is a classifier for each class. I mean classifier 1 predicts the probability that a given input belongs to class 1 and not to any other class, classifier 2 predicts the probability that a given input belongs to class 2 and not to any other class and so on.

...

I am not sure for me if you understand that the softmax function is N dimensional and it is normalized. So you can interpret each output as the probability associated with that class.

Answer 2

The reason you need softmax is because your last layer of outputs has more than one neuron.

Suppose, you have a sigmoid activation in the last layer but it has two neurons. How are you going to plug this into a loss function? What is your scalar prediction?

So, with more than neuron you need a way to aggregate the outputs of the neurons into one number, a scalar, your prediction for an input sample. You can use softmax to do it. Conceptually, you can think of a softmax as an ultimate true last layer with a sigmoid activation, it accepts outputs of your last layer as inputs, and produces one number on the output (activation). So, the softmax is a sigmoid you want.

Not only it is a sigmoid, it's also a multinomial logit: $\frac{e^{a_i}}{\sum_ie^{a_i}}$. In other words,using softmax is exactly what you suggested: use a sigmoid and loss function.