I run modified non-negative matrix factorization (NMF) and tune the regularization weight from 1e5 to 1e13.

The table below shows errors from calculating cost function for 25 iterations of all weights.

At weight 1e4, the objective function is monotonically decreased. But when I increase the weight more and more, the error goes down and gradually up. What is happened here? By the way, I think I got the best result is at weight 5e10 where the error is not monotonically decreased. In this case, which weight I should choose?