I am trying to implement a solution to Ridge regression in Python using Stochastic gradient descent as the solver. My code for SGD is as follows:
def fit(self, X, Y): # Convert to data frame in case X is numpy matrix X = pd.DataFrame(X) # Prepend a column of 1s to the data for the intercept X.insert(0, 'intercept', np.array([1.0]*X.shape)) # Find dimensions of train m, d = X.shape # Initialize weights to random beta = self.initializeRandomWeights(d) beta_prev = None epochs = 0 prev_error = None while (epochs < self.nb_epochs): print("## Epoch: " + str(epochs)) indices = range(0, m) shuffle(indices) for i in indices: # Pick a training example from a randomly shuffled set beta_prev = beta xi = X.iloc[i] errori = sum(beta*xi) - Y[i] # Error[i] = sum(beta*x) - y = error of ith training example gradient_vector = xi*errori + self.l*beta_prev beta = beta_prev - self.alpha*gradient_vector epochs += 1
The data I'm testing this on is not normalized and my implementation always ends up with all the weights being Infinity, even though I initialize the weights vector to low values. Only when I set the learning rate alpha to a very small value ~1e-8, the algorithm ends up with valid values of the weights vector.
My understanding is that normalizing/scaling input features only helps reduce convergence time. But the algorithm should not fail to converge as a whole if the features are not normalized. Is my understanding correct?