So I'm trying to implement a neural network using only numpy module in Python. The problem I'm facing is related to the correct implementation of the regularization through weight decay, and also the momentum in stochastic gradient descent.
When I only perform sgd without any weight decay and momentum my neural net performs well on the application scenario I'm facing. If I try to add either of these two terms, my model performs really badly. So I suppose that I'm not understanding them well.
Since my model works fine with sgd, let us assume that my model do it's job in forward and back propagation. The problematic aspect arise in the implementation of the parameters update for a layer of my net.
The method for the parameters update is the following:
def update_params(self, n_samples): weight_decay_term = (self.learning_rate * self.weight_decay / n_samples) * self.W weight_velocity = self.momentum * self.weight_velocity + self.dW self.W = self.W - weight_decay_term - self.learning_rate * weight_velocity self.weight_velocity = np.copy(weight_velocity) bias_velocity = self.momentum * self.bias_velocity + self.db self.b = self.b - self.learning_rate * bias_velocity self.bias_velocity = np.copy(bias_velocity)
where 'n_samples' is the size of the mini-batch.
For the weight decay formula I followed the guideline described in: https://github.com/limberc/deeplearning.ai/blob/master/COURSE%202%20Improving%20Deep%20Neural%20Networks%20Hyperparameter%20tuning%2C%20Regularization%20and%20Optimization/week%2001/week%2001%20Setting%20up%20your%20Machine%20Learning%20Application.pptx
For the momentum formula I followed the guideline described in: https://cs230.stanford.edu/files/C2M2.pdf
I also tried the formula described in: Neural Networks: weight change momentum and weight decay without any success.
None of these solutions worked, meaning that setting for example:
self.learning_rate = 0.01 self.momentum = 0.9 self.weight_decay = 0.1
my model performs really badly. I suppose it is related to my understanding of the implementation details of weight decay and momentum, but I really can't wrap my head around this problem. If I set the momentum and weight_decay parameters to zero I get acceptable results with my model, which are similar to those that I obtain with a pytorch model with the same architecture.
I tried to instantiate a pytorch multy layer perceptron with the same architecture that I tried with my model, and used as optimizer:
torch_optimizer = torch.optim.SGD(torch_model.parameters(), lr=0.01, momentum=0.9, weight_decay=0.1)
and the torch net performs greatly on my application scenario.
So, can you spot the problem on my implementation of the update parameters method? Thank you for reading