1
$\begingroup$

I found this interesting paper regarding deep unsupervised clustering and am looking to mimic some of the things done, however there is one thing that is not clear to me. In the paper, they use a softmax layer to get the probabilities of the latent vectors belonging to a certain cluster, however there is one thing that I seem to miss with using softmax.

Let's say that we wanted to use a sigmoid activation function for our encoding layers as described in the papers. If I understand correctly, doing this causes the outputs to be between 0 and 1, meaning that the latent vectors will be vectors with entries between 0 and 1. The problem with this is that with values between 1 and 0, the softmax function by its nature cannot give a probability, for $k=2$ (where $k$ is the number of clusters), greater than $$\frac{e}{e + \frac{1}{e}} = 0.88079707797...$$ The problem here is that it seems to me like the softmax function should be able to 'reach' values much closer to 0 and 1 as these are probabilities.

My original thought was that I need to normalize the data in some way, but I am not sure if this is necessary here or where I would normalize the data in the network.

I also know that softmax is supposed to get log probabilities and 'undo' them by exponentiating, and because I don't think my program uses logs anywhere, I think the issue may be here.

I think there also could be some fundamental misunderstanding I have with this whole problem, so if anybody could explain why this is happening and what I can do to avoid this problem I would be very thankful.

Improve this question
$\endgroup$
3
  • $\begingroup$ where did they say they used sigmoid activation? Typically, relu is used with an (unactivated) linear layer before softmax. $\endgroup$
    – shimao
    Sep 5, 2018 at 19:21
  • $\begingroup$ They don't say they used any particular activation which was part of my confusion. Using ReLU makes a lot more sense because its range is not restricted, but I guess this must be what they used then? $\endgroup$
    – Alerra
    Sep 5, 2018 at 19:26
  • $\begingroup$ yes, they probably used relu. also, regardless of the activation function, softmax is applied after a linear layer so is not affected by the choice activation function. $\endgroup$
    – shimao
    Sep 5, 2018 at 19:28

1 Answer 1

Reset to default
1
$\begingroup$

The expression you wrote is for the softmax function applied directly to outputs from a sigmoidal layer. But, a softmax output layer applies an affine transformation to its inputs before running them through the softmax function.

Suppose we we have a softmax output layer with $k$ units, and the output of the previous layer is $x$. The output of the $i$th unit (with weights $w_i$ and bias $b_i$) is:

$$\sigma_i(x) = \frac{\exp(w_i \cdot x + b_i)}{\sum_{j=1}^k \exp(w_j \cdot x + b_j)}$$

So, the weights and biases can compensate for the range of $x$.

Improve this answer
$\endgroup$
1
  • $\begingroup$ So it sounds like to me that simply having relu activation before applying softmax will give you this type of transformation, correct? And if that is the case, then would it be okay to have a sigmoidal activation layer, followed by a layer with relu activation, and then softmax? $\endgroup$
    – Alerra
    Sep 6, 2018 at 11:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.