# The accuracy decreased when I used L2 regularization method [closed]

I am training a 6 layer Deep Neural Network with:

<bound method Module.parameters of Model2(
(layer1): Linear(in_features=4800, out_features=8000, bias=True)
(layer2): Linear(in_features=8000, out_features=5000, bias=True)
(layer3): Linear(in_features=5000, out_features=2000, bias=True)
(layer4): Linear(in_features=2000, out_features=200, bias=True)
(layer5): Linear(in_features=200, out_features=20, bias=True)
(layer6): Linear(in_features=20, out_features=52, bias=True)


Inputs are images in size 60 * 80. I am using relu activation function, Cross-Entropy for loss function, and Stochastic Gradient Descent. I used the weight-decay parameter(i.e. L2 regularization method). I don't face overfitting but the accuracy was 61 % and now is 26%!

Can anyone explain the reason?

(I assigned hyperparameter of regularization to 0.01. I changed it but no improvement in the accuracy!)

• 1) How does your loss function perform with and without regularization? Accuracy has some issues as a performance metric. 2) In-sample or out-of-sample accuracy? – Dave Jul 6 at 11:29
• @Dave I didn't get it. What do you mean by loss function performance? I am just comparing the accuracy of model on the train and test data(to check overfitting) and the accuracy of the model. Also, I checked the parameters of my network, weights decreased but they were not near zero. – Maryam Jul 6 at 11:33
• @Dave It is out-sample accuracy that I mentioned in the post. – Maryam Jul 6 at 11:35
• Evaluate the cross-entropy loss on your out-of-sample data. It is possible for loss to decrease while accuracy also decreases. Accuracy turns out to be a surprisingly bad performance metric, despite how common it is. – Dave Jul 6 at 11:37
• @Dave For 10 epochs my losses are:epoch 1 3.109, epoch 2, 2.585, epoch 3, 2.469 , epoch 4, 2.306, epoch 5, 2.267, epoch 6, 2.161, epoch 7, 2.103, epoch 8, 2.043, epoch 9, 2.047, epoch 10, 1.996, – Maryam Jul 6 at 11:49

1. Accuracy is a surprisingly bad performance metric. If you evaluate your model using a so-called proper scoring rule like cross-entropy loss or Brier score, you may find that the out-of-sample performance improves even though accuracy decreases. This is because accuracy relies on a threshold, and the threshold that gives the beat accuracy might not even be $$0.50$$ (For binary classification).