0
$\begingroup$

Let us assume that I have the following neural network architecture:

Input-Layer: 12 nodes
1st Hidden Layer: 9 nodes
2nd Hidden Layer: 6 nodes
Output Layer: 3 nodes

Can I use Softmax activation function on the output layer for the above architecture? If so, how? Because, in the Softmax formula how will I get the numerator properly if the number of nodes in the last hidden layer and the output layer are not same?

In the architecture above, I will get 6 weighted inputs $(x_i)$ at the output layer. Then the softmax output for $j$th output node is: $$\frac{e^{x_j}}{S}$$, where, $$S=\sum{e^{x_i}}$$

But this will only work if the range of $i$ and $j$ are same. That is only possible if the last hidden layer and the output layer have the same number of nodes. Have I understood this correctly?

$\endgroup$
3
  • 1
    $\begingroup$ No, you have 3 softmax activation functions at the output each having 6 weights coming in from the previous layer. $\endgroup$ May 26, 2017 at 16:53
  • $\begingroup$ You will get 6 * 3 weights at the output layer in total, just like Cagdas Ozgenc said. Not 6. $\endgroup$ May 26, 2017 at 17:34
  • $\begingroup$ Yes, I know that. Actually I meant 6 inputs per output-layer-node. I did that because I need to find the output for each output-layer node (for my program) rather than the whole output layer at the same time. $\endgroup$ May 26, 2017 at 18:39

1 Answer 1

1
$\begingroup$

Have you done multi-class logistic regression? Because the same "problem" arises. Logistic regression is a two-layer (no hidden layer) neural network. It can have as many inputs as you want, regardless of the number of classes of the target variable. So I will explain the answer in the context of multi-class logistic regression, and all you need to do to translate this to your problem is replace "input features" with "last hidden layer" in your network.

In logistic regression you have $D$ input features, $X_0, X_1, \dots, X_{D-1}.$ For convenience, I'm including $X_0$ as the "bias term", so $X_0=1$ all the time. I'm just doing this so I don't have to describe the bias separately. (Bias terms appear in hidden layers of NNs, so the same thing would apply.)

Suppose the target variable, $t,$ is a categorical variable with $k$ classes. Then we use a one-of-k encoding for $t.$ (That is, we represent it with $k$ binary variables, where one of the variable values is $1$ and the rest are $0.$) Then the non-activated output is,

$$ A = W^T X, $$ where $W$ is a $D \times k$ matrix representing the weight vectors for each class, and $X$ is a $D \times 1$ vector representing a single observation. Thus $A$ is a $k \times 1$ vector. The output, $Y,$ which represents our predicted probabilities is, $$ Y = \text{softmax}(A), $$ meaning $$ Y_i = \frac{e^{A_i}}{\sum\limits_{j=1}^{k} e^{A_j}}. $$ $Y_i$ is the probability that the $i$th binary variable of $t$ is one.

Notice that the dimensions of $X$ is irrelevant here, because $W$ converts a $D \times 1$ vector $X$ into a $k \times 1$ vector $A.$ Thus there is no hard constraint on the number of input features you can apply to multi-class logistic regression. This is exactly analogous to the last hidden layer in a multi-class classification neural network.

$\endgroup$
4
  • $\begingroup$ You said A is a k x 1 vector. If D>k, how can I get A(j) (in the denominator of your last equation) if j>k? $\endgroup$ May 26, 2017 at 18:37
  • $\begingroup$ You said yourself that $A$ is a $k \times 1$ vector. So how can you have an $A_j$ for $j>k$? $\endgroup$ May 26, 2017 at 18:48
  • $\begingroup$ Maybe I am not understanding the last equation correctly. In the last equation you wrote: Y(i) = $\frac{e^{A_i}}{\sum{e^{A_j}}}$. Suppose if $D=12$ and $k=3$, how can I sum $A_j$ from $j=0$ to $11$? if A is a $k \times 1$ vector containing $k$ numbers, so, suppose $A = \{ 2.5, 6.3, 8.4 \}$, then what will be the denominator?? $e^2.5 + e^6.3 + e^8.4 + (?) + (?) + (?) ... + (?)$ ? How can I get the sum of the remaining $9$ elements which do not exist? $\endgroup$ May 26, 2017 at 19:22
  • $\begingroup$ Oh I'm sorry, I made a typo. In the denominator I wrote $\sum_{j=0}^{D-1} e^{A_j}$ when I should have wrote $\sum_{j=1}^{k} e^{A_j}.$ Maybe that's what caused your confusion. I edited it. You sum over the values of $A$, which has $k$ elements. $\endgroup$ May 26, 2017 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.