In the Stanford ML course, we were taught to find good values for the lambda parameters of ridge/lasso by iterating for various lambda values on several cross-validation sets and picking the values which correspond to the hypothesis with the minimum CV error.
The problem is, I am playing with a big data set (which might not even be well suited for logistic regression (??): Internet Ads set) and I can't use the method described above because, during optimization, the cost stays stable (changes only around the 8th decimal place between iterations) and doesn't seem to converge.
I need good values for the regularization in order to converge, but I can't converge without the aforementioned good values.
Any suggestions? Should I move on to using SVM, or is this data set solvable with logistic regression?
NB: I am doing this for learning purposes, so I'm much more interested in explanations why my approach is bad than I am in black-box libraries which will give me a solution.
EDIT: Some relevant code snippets (occasionally pseudocode-ish, for clarity). The usual notations apply.
The function used to compute the cost:
def computeCost(theta, X, y): global iter iter += 1 if iter > 10: raise TooManyIterationsException(iter) # Because the cost doesn't converge, I force interruption in order to jump to another combination of lambda values m = y.size h = sigmoid(X.dot(theta.T)) J = y.T.dot(log(h)) + (1.0 - y.T).dot(numpy.log(1.0 - h)) J_reg2 = theta[1:]**2 J_reg1 = theta[1:] cost = (-1.0 / m) * (J.sum()) + LAMBDA2 * J_reg2.sum() + LAMBDA1 * J_reg1.sum() print "Cost: ", cost return cost
initial_thetas = numpy.zeros((len(train_X), 1)) myargs = (train_X, train_y) for LAMBDA1 in [0.01, 0.02, 0.04, ..., 10]: for LAMBDA2 in my_range[0.01, 0.02, 0.04, ..., 10]: try: iter = 0 theta = scipy.optimize.fmin_bfgs(computeCost, x0=initial_thetas, args=myargs) except TooManyIterationsException as e: print '\n'
A typical output looks like this:
EDITED AGAIN: Evolution of thetas!