1
$\begingroup$

I am using Tensorflow for multi-label classification of Audio. The dataset I am using is made up of 10 different classes, and to each sample of audio correspond two labels. In other words, the number of overlapping sounds/classes per file is always 2. So the vector of labels "y" would be for example:

y = [0, 0, 0, 0, 0, 1, 0, 1, 0, 0]

I am using 3 convolutional layers , 1 FC layer and 1 output layer made up of 10 logistic units. And as cost function I am using the binary cross entropy for each unit and then averaging the result.

I don't know if this actually has any consequences but all my inputs are negative values.

For some reason(s), I guess mainly as a consequence of the initialization of the parameters (weights) of the network and the logistic units in the output layer, the network tends to predict mainly zeros. I guess after some training this trend will start to dissapear, however, I wonder if there is a smart way of doing something so the network does not predict those many zeros and predicts more ones, which I believe would make the cost higher and the gradient stronger (not 100% sure about this one though) and therefore training would be faster.

I use Xavier initialization for the weights of all layers, as an example:

W2 = tf.get_variable("W", [5,5,24,48], initializer=tf.contrib.layers.xavier_initializer())

Maybe using tf.truncated_normal_initializer instead could help?

$\endgroup$
1
$\begingroup$

I use sigmoid when there are an arbitrary number of possible labels. In your case, you know you have exactly two labels. I would instead use softmax and divide the true label by two, for example [0,...,0,.5,.5,0,...].

This will force your network to at least sum to 1, something you don’t get with sigmoid.

During inference, multiply the output by two if you want it to remain more interpretable as individual probabilities.

$\endgroup$
  • $\begingroup$ Thanks for the comment @kbrose. But if I use softmax, am I not then modelling a probability distribution that would give me as prediction 0.5 and 0.5 when two classes are present, when in fact the probabilities of those two classes should be 1 and 1 ? $\endgroup$ – sdiabr May 23 '18 at 15:24
  • $\begingroup$ Softmax is usually used to force a probability distribution, but the only thing really special about it is you force the output of a layer to sum to a specific value. In your case, you can just re-interpret it as forcing your network to make two decisions instead of one. As I suggested, you could always multiply output by 2 during inference so it appears more interpretable. $\endgroup$ – kbrose May 23 '18 at 18:56
  • $\begingroup$ Okey, I will give it a try then. Thanks. But using sigmoids should in theory also work right? Even if you are not forcing it to sum 1 in the outputs, it would end up doing that, ideally. I am not 100% sure though about what you say about multiplying the output by 2 during inference. I've read (towardsdatascience.com/…) that the softmax does some sort of weighted average. Are you sure that multiplying the output by two is a fair thing to do? $\endgroup$ – sdiabr May 23 '18 at 19:18
  • $\begingroup$ Yes, sigmoids should work in theory. Using just fully connected layers on images also work in theory, though, but no one does that except as a toy project on MNIST. A lot of my experience in this field has been that adding constraints to your model that best reflect the data-at-hand tends to give better results faster. $\endgroup$ – kbrose May 23 '18 at 19:23
  • $\begingroup$ All softmax does is force all outputs across the given layer to sum to 1 in a way that gradients can still flow through pretty well. I don't really know how to answer "are you sure that ...". It's all just numbers that we're choosing to interpret in different ways. I feel confident in my ability to manipulate these numbers in ways that may make them slightly more interpretable. Can you say why you think doing such a multiplication would be invalid? $\endgroup$ – kbrose May 23 '18 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.