Why would a dropout neural network fail to classify a class?

Question

I built a multi-layer neural net (ReLU for hidden layer, softmax on logistic activation) that classifies to 3-class labels. Then I tried to add drop-out to it but the results are much worse. I found out the reason being that after about a few epochs the network no longer recognized one of the class.

// epoch # 8:
F1: 0.547721, precision 0.704396, recall 0.448061, accuracy 0.345785
precisions: [ 0.11338404  0.19483106  0.91521546]
recalls: [ 0.43370881  0.52922429  0.16718137]
    class[0] is predicted as class[0]: 72
    class[0] is predicted as class[1]: 86
    class[0] is predicted as class[2]: 8
    class[1] is predicted as class[0]: 149
    class[1] is predicted as class[1]: 181
    class[1] is predicted as class[2]: 12
    class[2] is predicted as class[0]: 414
    class[2] is predicted as class[1]: 662
    class[2] is predicted as class[2]: 216

// epoch # 9 and later became something like:
F1: 0.087709, precision 0.047347, recall 0.594508, accuracy 0.232189
precisions: [ 0.09353582  0.2037937   0.        ]
recalls: [ 0.33130534  0.72220111  0.        ]
    class[0] is predicted as class[0]: 55
    class[0] is predicted as class[1]: 111
    class[0] is predicted as class[2]: 0
    class[1] is predicted as class[0]: 95
    class[1] is predicted as class[1]: 247
    class[1] is predicted as class[2]: 0
    class[2] is predicted as class[0]: 438
    class[2] is predicted as class[1]: 854
    class[2] is predicted as class[2]: 0

whereas the non-dropout one would recognized class 2:

F1: 0.578016, precision 0.694886, recall 0.494799, accuracy 0.460616
precisions: [ 0.1372982   0.13861043  0.9137756 ]
recalls: [ 0.72284802  0.1637379   0.36919219]
    class[0] is predicted as class[0]: 120
    class[0] is predicted as class[1]: 41
    class[0] is predicted as class[2]: 5
    class[1] is predicted as class[0]: 246
    class[1] is predicted as class[1]: 56
    class[1] is predicted as class[2]: 40
    class[2] is predicted as class[0]: 508
    class[2] is predicted as class[1]: 307
    class[2] is predicted as class[2]: 477

(Background: In this data model, 70% of samples belongs to class 2 and I normalize the training set by generating more class 0 and 1 samples to balance the 3 classes.)

What would cause drop-out network to behave this way?

Answer 1

1. Are you accidentally applying dropout during test phase?

   • If not, are you remembering to divide all unit activations by the dropout ratio during test phase?

2. Are you accidentally applying dropout to the output layer too?

3. Are you accidentally not changing the subset of units that are dropped out with each training iteration?

A combination of 2 and 3 could be your problem, if class 2 is dropped-out and never un-dropped out.