Let us assume that I have the following neural network architecture:

Input-Layer: 12 nodes
1st Hidden Layer: 9 nodes
2nd Hidden Layer: 6 nodes
Output Layer: 3 nodes

Can I use Softmax activation function on the output layer for the above architecture? If so, how? Because, in the Softmax formula how will I get the numerator properly if the number of nodes in the last hidden layer and the output layer are not same?

In the architecture above, I will get 6 weighted inputs $(x_i)$ at the output layer. Then the softmax output for $j$th output node is: $$\frac{e^{x_j}}{S}$$, where, $$S=\sum{e^{x_i}}$$

But this will only work if the range of $i$ and $j$ are same. That is only possible if the last hidden layer and the output layer have the same number of nodes. Have I understood this correctly?

  • 1
    $\begingroup$ No, you have 3 softmax activation functions at the output each having 6 weights coming in from the previous layer. $\endgroup$ – Cagdas Ozgenc May 26 '17 at 16:53
  • $\begingroup$ You will get 6 * 3 weights at the output layer in total, just like Cagdas Ozgenc said. Not 6. $\endgroup$ – Thomas Wagenaar May 26 '17 at 17:34
  • $\begingroup$ Yes, I know that. Actually I meant 6 inputs per output-layer-node. I did that because I need to find the output for each output-layer node (for my program) rather than the whole output layer at the same time. $\endgroup$ – user1637645 May 26 '17 at 18:39

Have you done multi-class logistic regression? Because the same "problem" arises. Logistic regression is a two-layer (no hidden layer) neural network. It can have as many inputs as you want, regardless of the number of classes of the target variable. So I will explain the answer in the context of multi-class logistic regression, and all you need to do to translate this to your problem is replace "input features" with "last hidden layer" in your network.

In logistic regression you have $D$ input features, $X_0, X_1, \dots, X_{D-1}.$ For convenience, I'm including $X_0$ as the "bias term", so $X_0=1$ all the time. I'm just doing this so I don't have to describe the bias separately. (Bias terms appear in hidden layers of NNs, so the same thing would apply.)

Suppose the target variable, $t,$ is a categorical variable with $k$ classes. Then we use a one-of-k encoding for $t.$ (That is, we represent it with $k$ binary variables, where one of the variable values is $1$ and the rest are $0.$) Then the non-activated output is,

$$ A = W^T X, $$ where $W$ is a $D \times k$ matrix representing the weight vectors for each class, and $X$ is a $D \times 1$ vector representing a single observation. Thus $A$ is a $k \times 1$ vector. The output, $Y,$ which represents our predicted probabilities is, $$ Y = \text{softmax}(A), $$ meaning $$ Y_i = \frac{e^{A_i}}{\sum\limits_{j=1}^{k} e^{A_j}}. $$ $Y_i$ is the probability that the $i$th binary variable of $t$ is one.

Notice that the dimensions of $X$ is irrelevant here, because $W$ converts a $D \times 1$ vector $X$ into a $k \times 1$ vector $A.$ Thus there is no hard constraint on the number of input features you can apply to multi-class logistic regression. This is exactly analogous to the last hidden layer in a multi-class classification neural network.

  • $\begingroup$ You said A is a k x 1 vector. If D>k, how can I get A(j) (in the denominator of your last equation) if j>k? $\endgroup$ – user1637645 May 26 '17 at 18:37
  • $\begingroup$ You said yourself that $A$ is a $k \times 1$ vector. So how can you have an $A_j$ for $j>k$? $\endgroup$ – Bridgeburners May 26 '17 at 18:48
  • $\begingroup$ Maybe I am not understanding the last equation correctly. In the last equation you wrote: Y(i) = $\frac{e^{A_i}}{\sum{e^{A_j}}}$. Suppose if $D=12$ and $k=3$, how can I sum $A_j$ from $j=0$ to $11$? if A is a $k \times 1$ vector containing $k$ numbers, so, suppose $A = \{ 2.5, 6.3, 8.4 \}$, then what will be the denominator?? $e^2.5 + e^6.3 + e^8.4 + (?) + (?) + (?) ... + (?)$ ? How can I get the sum of the remaining $9$ elements which do not exist? $\endgroup$ – user1637645 May 26 '17 at 19:22
  • $\begingroup$ Oh I'm sorry, I made a typo. In the denominator I wrote $\sum_{j=0}^{D-1} e^{A_j}$ when I should have wrote $\sum_{j=1}^{k} e^{A_j}.$ Maybe that's what caused your confusion. I edited it. You sum over the values of $A$, which has $k$ elements. $\endgroup$ – Bridgeburners May 26 '17 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.