Why is L-BFGS optimization faster when binary features have been standardized? I have a question related to standardization of binary features within a regularized logistic regression. Suppose that you have a model where all the features are binary (the result of applying the hashing trick) so, it's not necessary to apply it.
The problem comes when I try to train the model using LBFGS, why does the standardized system converge faster? (For example, the standardized version takes 9 minutes and 37 iterations while non-standardized takes 7 hours and 4992 iterations.) What is the intuition here? The optimization function is similar in both cases, isn't it? I don't know if it's related to LBFGS or it occurs with all the optimizers, but I don't get why standardization makes it faster. 
In addition, if I apply standardization to my training database, should I apply it in prediction time? What if I can't apply this standardization in prediction time for technical reasons (or design restrictions)?
 A: Comments from a colleague:

It would be good to look at the mean and variance of the features.
  Actually, the mean probably isn't all that important, but I would bet
  dimes to dollars that some of the features have VERY small variance,
  resulting in a ill-conditioned problem.  This presents a problem for
  any solver, but it's especially hard for first-order methods like
  BFGS.  Recall that the first iteration of BFGS is gradient descent;
  the next few iterations aren't all that much better.  In a full BFGS
  implementation, the solver would eventually build a pretty decent
  estimate of the Hessian, but L-BFGS never quite gets there because the
  Hessian estimate is only based on m gradients, with m typically pretty
  small.
A second-order method would certainly help, but it's probably too
  expensive.  On the other hand, preconditioning is cheap and easy.  I
  had more than one professor tell me that you should ALWAYS
  precondition your problems.  Even if all you do is use the diagonal of
  your matrix as a preconditioner, it's worth it.  Because it's cheap
  and easy and it can really pay off.  I think this is an excellent
  demonstration of that.

