I run modified non-negative matrix factorization (NMF) and tune the regularization weight from `1e5` to `1e13`. 

The table below shows errors from calculating cost function for 25 iterations of all weights. 

[![enter image description here][1]][1]

At weight `1e4`, the objective function is monotonically decreased. But when I increase the weight more and more, the error goes down and gradually up. What is happened here?
By the way, I think I got the best result is at weight `5e10` where the error is not monotonically decreased. In this case, which weight I should choose?

  [1]: https://i.sstatic.net/3otF1.png