Restricting the output of a model didn't improve the loss value (MSE) of the model evaluation

Question

There is a deep model for prediction.

The outputs are some numbers between 0 and 80. (In the dataset the outputs are 0-80)

The model Loss value is 70 and I would like to reduce it.

I printed the outputs after evaluating the model by test values and some of the predicted values are more than 80 or less than 0.

I decided to set up the final layer to predict just in 0-80 in the training step, therefore I set a lambda layer after final Dense layer to clip output values.

The codes:

def relu_advanced(x):
    return K.relu(x, max_value=80)

def createModel4():
    model = models.Sequential()
    model.add(Conv2D(256,(3, 3),
                     activation='relu',
                     input_shape=(320,20,1), padding='same'))
    model.add(MaxPooling2D((2, 2)))

    model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
    model.add(MaxPooling2D((2, 2)))

    model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
    model.add(MaxPooling2D((2,2)))

    model.add(Flatten())
    #model.add(Dense(5*320, activation= 'relu'))
    model.add(Dense(5*320))
    model.add(Lambda(relu_advanced))



    model.summary()
    return model

I tested the model with and without the relu_advanced and unfortunately, the Loss value (MSE) is increased with advanced_relu!

While there is no value much than 80 or less than zero, I don't know what may happen that the Loss (MSE) is increased?

Thank you

Answer 1

ReLU's are great for some purposes, but it's important to remember that they will set your gradient to zero if the values input to the ReLU are outside the allowed bounds. This is particularly problematic at the final layer since no other layers will be able to update on that iteration on that training example (the zero gradient stays zero during all of backprop).

You may have better luck using a properly scaled squashing function, e.g. sigmoid(x) * 80. This will ensure you have a gradient everywhere, albeit a much smaller gradient near 0 and 80 than at 40.