1
$\begingroup$

I have a multilabel classification on audio files and I'm troubled about the architecture. First of all, I would like my model to output the probabilities of each label which in my case are all independent (don't need to sum up to 1).

So I have constructed a CNN that consists of :

  • 3 convolutional layers
  • 1 fully connected and
  • output layer

Regarding the activation functions of each layer I chose ReLu for the 3 convolutional and the fully connected and sigmoid for the output. The loss function is also chosen as sigmoid_cross_entropy_with_logits (I'm using tensorflow).

The problem is that the produced output is not a probability but simply 0 or 1 and this is actually normal as ReLu outputs positive values which are not upperbounded while the sigmoid is flat for values higher than 5.

enter image description here

Also the weights and the bias I use as sampled from the normal distribution.

What should I do ? My thoughts so far are:

  • Change the activation of the convolutional and fully connected layers so that they produce bounded values to feed into the sigmoid.
  • Sample weights and biases from another distribution other than normal so that when multiplied with layer's output will give relatively small values.

Some pieces of the corresponding code:

Weights and biases initialization

weights = {
    'wc1': tf.Variable(tf.random_normal([10, 10, 1, 128])),
    'wc2': tf.Variable(tf.random_normal([10, 10, 128, 284])),
    'wc3': tf.Variable(tf.random_normal([10, 10, 284, 768])),
    'wd1': tf.Variable(tf.random_normal([10*10*768, 2048])),
    'out': tf.Variable(tf.random_normal([2048, n_classes]))
}    
biases = {
    'bc1': tf.Variable(tf.random_normal([128])),
    'bc2': tf.Variable(tf.random_normal([284])),
    'bc3': tf.Variable(tf.random_normal([768])),
    'bd1': tf.Variable(tf.random_normal([2048])),
    'out': tf.Variable(tf.random_normal([n_classes]))
}

CNN definition

def conv_net(x, weights, biases, dropout):
    x = tf.reshape(x, shape=[-1, 120, 120, 1])
    # 1st Convolution Layer
    conv1 = conv2d(x, weights['wc1'], biases['bc1'])
    # Max Pooling (down-sampling)
    conv1 = maxpool2d(conv1, k=6)

    # 2nd Convolution Layer
    conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
    # Max Pooling (down-sampling)
    conv2 = maxpool2d(conv2, k=2)

    # 3rd Convolution Layer (without maxpooling)
    conv3 = conv2d(conv2, weights['wc3'], biases['bc3'])

    # Fully connected layer
    # Reshape conv2 output to fit fully connected layer input
    fc1 = tf.reshape(conv3, [-1, weights['wd1'].get_shape().as_list()[0]])
    print (fc1.get_shape().as_list())
    fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
    print (fc1.get_shape().as_list())
    fc1 = tf.nn.relu(fc1)
    print (fc1.get_shape().as_list())
    # Apply Dropout
    fc1 = tf.nn.dropout(fc1, dropout)

    # Output, class prediction
    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
    out = tf.nn.sigmoid(out)
    return out
$\endgroup$
1
$\begingroup$

Couple of comments:

  1. If you use the loss function which already has the sigmoid in it, you do not need to use a sigmoid at the output.

  2. sigmoid_cross_entropy_with_logits is a loss function for binary outputs, so it tries to solve binary classification problems. what you are looking for is not binary classification, but rather a regression type of problem.

  3. Is your training data labeled like you describe your output to be? Are your training labels binary or also probabilities like you describe? If they are binary, then you are expecting sth from the network, for which you do not supply the ground truth. If your labels are probabilities like you describe, then I would try to solve the problem as a regression problem with an activation function at the output which maps to [0,1] for each channel.

$\endgroup$
7
  • $\begingroup$ 1. What activation should I use instead? 2. Regarding the appropriateness of the loss function I read the documentation here ( tensorflow.org/api_docs/python/tf/nn/… ) and thought that was the right choice. I am actually looking for the equivalent of softmax for a multilabel classification task 3. My labels are in binary format. The main reason I want to output posterior probabilities is that I want to use them in order to construct the ROC and compute AUC for model evaluation. $\endgroup$ – Mewtwo Aug 25 '17 at 9:43
  • $\begingroup$ 1. Nothing. The loss function passes it through a sigmoid itself. 2. The problem is due to the sigmoid function. Your training compares values after the last sigmoid with binary ground truth labels, and reduces that error, however it does not try to match the values before the sigmoid, i.e. there is not a probabilistic relation there. $\endgroup$ – user166243 Aug 25 '17 at 9:58
  • $\begingroup$ What is exactly the problem with taking the output of the sigmoid as probabilities? You say they are always 0 or 1, but that shows that the network has a very high accuracy. A prob. can also be 0 or 1, right? $\endgroup$ – user166243 Aug 25 '17 at 10:12
  • $\begingroup$ Removing the sigmoid from the output layers looks good. But then the network will output positive values in the (0, +00) which are not interpretable as probabilities. How can I fix this? As for your second comment, 0 and 1 are indeed "valid" probabilities. The problem is that when high values ( > 5 ) are being fed to the sigmoid the output is always 1. Intuitively, I feel that it is not so good to "squeeze" the initial variability of the values outputted from the network in those greater than 5 and those lower than 5. It is somehow like segmenting the positive numbers into (0,5) and (5,+00) $\endgroup$ – Mewtwo Aug 25 '17 at 10:21
  • $\begingroup$ "Intuitively, I feel that it is not so good to "squeeze" the initial variability of the values outputted from the network" -> but this is exactly what you are trying to achieve. If the network is certain of sth, it will make the prob. go to 1 (or 0), else it will keep it somewhere in between. $\endgroup$ – user166243 Aug 25 '17 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.