CNN architecture for multilabel classification on audio files

Question

I have a multilabel classification on audio files and I'm troubled about the architecture. First of all, I would like my model to output the probabilities of each label which in my case are all independent (don't need to sum up to 1).

So I have constructed a CNN that consists of :

• 3 convolutional layers
• 1 fully connected and
• output layer

Regarding the activation functions of each layer I chose ReLu for the 3 convolutional and the fully connected and sigmoid for the output. The loss function is also chosen as sigmoid_cross_entropy_with_logits (I'm using tensorflow).

The problem is that the produced output is not a probability but simply 0 or 1 and this is actually normal as ReLu outputs positive values which are not upperbounded while the sigmoid is flat for values higher than 5.

Also the weights and the bias I use as sampled from the normal distribution.

What should I do ? My thoughts so far are:

• Change the activation of the convolutional and fully connected layers so that they produce bounded values to feed into the sigmoid.
• Sample weights and biases from another distribution other than normal so that when multiplied with layer's output will give relatively small values.

Some pieces of the corresponding code:

Weights and biases initialization

weights = {
    'wc1': tf.Variable(tf.random_normal([10, 10, 1, 128])),
    'wc2': tf.Variable(tf.random_normal([10, 10, 128, 284])),
    'wc3': tf.Variable(tf.random_normal([10, 10, 284, 768])),
    'wd1': tf.Variable(tf.random_normal([10*10*768, 2048])),
    'out': tf.Variable(tf.random_normal([2048, n_classes]))
}    
biases = {
    'bc1': tf.Variable(tf.random_normal([128])),
    'bc2': tf.Variable(tf.random_normal([284])),
    'bc3': tf.Variable(tf.random_normal([768])),
    'bd1': tf.Variable(tf.random_normal([2048])),
    'out': tf.Variable(tf.random_normal([n_classes]))
}

CNN definition

def conv_net(x, weights, biases, dropout):
    x = tf.reshape(x, shape=[-1, 120, 120, 1])
    # 1st Convolution Layer
    conv1 = conv2d(x, weights['wc1'], biases['bc1'])
    # Max Pooling (down-sampling)
    conv1 = maxpool2d(conv1, k=6)

    # 2nd Convolution Layer
    conv2 = conv2d(conv1, weights['wc2'], biases['bc2'])
    # Max Pooling (down-sampling)
    conv2 = maxpool2d(conv2, k=2)

    # 3rd Convolution Layer (without maxpooling)
    conv3 = conv2d(conv2, weights['wc3'], biases['bc3'])

    # Fully connected layer
    # Reshape conv2 output to fit fully connected layer input
    fc1 = tf.reshape(conv3, [-1, weights['wd1'].get_shape().as_list()[0]])
    print (fc1.get_shape().as_list())
    fc1 = tf.add(tf.matmul(fc1, weights['wd1']), biases['bd1'])
    print (fc1.get_shape().as_list())
    fc1 = tf.nn.relu(fc1)
    print (fc1.get_shape().as_list())
    # Apply Dropout
    fc1 = tf.nn.dropout(fc1, dropout)

    # Output, class prediction
    out = tf.add(tf.matmul(fc1, weights['out']), biases['out'])
    out = tf.nn.sigmoid(out)
    return out

Answer 1

Couple of comments:

1. If you use the loss function which already has the sigmoid in it, you do not need to use a sigmoid at the output.

2. sigmoid_cross_entropy_with_logits is a loss function for binary outputs, so it tries to solve binary classification problems. what you are looking for is not binary classification, but rather a regression type of problem.

3. Is your training data labeled like you describe your output to be? Are your training labels binary or also probabilities like you describe? If they are binary, then you are expecting sth from the network, for which you do not supply the ground truth. If your labels are probabilities like you describe, then I would try to solve the problem as a regression problem with an activation function at the output which maps to [0,1] for each channel.