# Why do I get different results each time I run my algorithm?

I'm doing machine learning with a training set, validation set and test set.

I train with the L_BFGS algorithm. The training converges all the time. I have the default accuracy from scipy, which is quite high.

Then I have a regularization parameter that I optimize on the validation set. I do this with grid search. For efficiency reasons, the way I implemented is that after each iteration during validation, I start from the weights used for the previous training.

Therefore, I don't start each training with zero or random weights. I do this because I think the training algorithm finds the minimum faster this way, because it has a good guess.

Now I have this result which I don't understand.

Doing grid search [0, 40]
set param to 0
training, converged!
measure validation error
set param 40
training, converged!
measure validation error
Best param is 40, lowest valid err: -8916, training error:-35274


Now I do the same thing but only with param 40

Doing grid search [40]
set param 40
training, converged!
measure validation error
Best param is 40, lowest valid err: -5214, training error:-41428


So in the second case, I started training with param 40 with weights all zero. In the first case, I started training with weights that came from training with param 0.

If I used LBFGS with high accuracy, shouldn't it give me the same result with param 40 in both cases? How come the training and validation errors are so different? If I don't get the same result, is it likely that I have a bug in my code?

As an explanation, I was thinking that LBFGS gets stuck in a local minimum based on the starting weights, but I'm not sure. If that's the case, how do I prevent this? Am I supposed to start from some random weights every time? When can I be relatively sure that LBFGS has indeed found a global minimum?