0
$\begingroup$

In multinomial classification, I'm using soft-max activation function for all non-linear units and ANN has 'k' number of output nodes for 'k' number of classes. Each of the 'k' output nodes present in output layer is connected to all the weights in preceding layer, kind of like the one shown below.

enter image description here

So, if the first output node intends to pull the weights in it's favor, it will change all the weights that precede this layer and the other sets will also pull which usually contradicts to the direction in which the first one was pulling. It seems more like a tug of war with single set of weights. So, do we need a separate set of weights(,which includes weights for every node of every layer) for each of the output classes or is there a different form of architecture present?

Improve this question
$\endgroup$
1
  • $\begingroup$ Your intuition is somewhat right if you consider a single example in isolation. However, the network needs many many examples to learn its weights. Moreover, the learning procedure always tends to minimise the total classification error in expectation, so the "tug-of-war" will always produce a better solution than the one you are currently at (it may not be the optimal, however). $\endgroup$
    – Vimal
    Commented Sep 8, 2015 at 21:56

1 Answer 1

Reset to default
0
$\begingroup$

I think you don't have a clear concept of what a neural network does.

The output nodes are mapped to the hidden layer (in this case) by a separate vector of weights for every node, thus the matrix-nature of the algorithm.

The fact that one node is pulling the weights in its favor does not contradict that the other are doing the same with ITS RELEVANT mappings (its own vector of weights). That's what makes difficult to visualize the concept of weight optimization/gradient descent, because it is happening in a (probably high) number of different dimensions, while we are used to understand only three spatial ones.

Hope to have helped.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.