I'm doing machine learning with a training set, validation set and test set.
I train with the L_BFGS algorithm. The training converges all the time. I have the default accuracy from scipy, which is quite high.
Then I have a regularization parameter that I optimize on the validation set. I do this with grid search. For efficiency reasons, the way I implemented is that after each iteration during validation, I start from the weights used for the previous training.
Therefore, I don't start each training with zero or random weights. I do this because I think the training algorithm finds the minimum faster this way, because it has a good guess.
Now I have this result which I don't understand.
Doing grid search [0, 40] set param to 0 training, converged! measure validation error set param 40 training, converged! measure validation error Best param is 40, lowest valid err: -8916, training error:-35274
Now I do the same thing but only with param 40
Doing grid search  set param 40 training, converged! measure validation error Best param is 40, lowest valid err: -5214, training error:-41428
So in the second case, I started training with param 40 with weights all zero. In the first case, I started training with weights that came from training with param 0.
If I used LBFGS with high accuracy, shouldn't it give me the same result with param 40 in both cases? How come the training and validation errors are so different? If I don't get the same result, is it likely that I have a bug in my code?
As an explanation, I was thinking that LBFGS gets stuck in a local minimum based on the starting weights, but I'm not sure. If that's the case, how do I prevent this? Am I supposed to start from some random weights every time? When can I be relatively sure that LBFGS has indeed found a global minimum?