I am currently trying to learn how MLE in a poisson regression context works. As such I am trying to compute a poisson regression from scratch using numpy. Furthermore, I try to solve the MLE using gradient descent. However, the loss I am computing stays constant and my guess is that my representation (matrix representation) for the gradient is not correct.
Following an example I found here: https://web.stanford.edu/class/archive/stats/stats200/stats200.1172/Lecture27.pdf
This is how I implemented it using numpy where X is the matrix of covariates and weights are the coefficients I am trying to learn:
y_pred = np.exp(np.dot(X, weights)) gradient = np.dot(X.T,(y - y_pred))
This is my full code:
n_samples, n_features = X.shape weights = np.zeros(n_features) def forward(X, weights): return np.exp(np.dot(X, weights)) def gradient(y, y_pred, X): gradient = np.dot(X.T,(y - y_pred)) return gradient lr = 0.01 n_iter = 3000 for i in range(n_iter): # predicting y_pred = forward(X, weights) # computing loss loss = np.sqrt(np.mean((y-y_pred)**2)) if i % 250 == 0: print(loss) # calculating gradient and updating weights calc_gradient = gradient(y, y_pred, X) weights -= lr * calc_gradient
Any help on what I am doing wrong here is highly appreciated!