# Why do increasing regularization weights make objective function not monotonically decrease?

I run modified non-negative matrix factorization (NMF) and tune the regularization weight from 1e5 to 1e13.

The table below shows errors from calculating cost function for 25 iterations of all weights.

At weight 1e4, the objective function is monotonically decreased. But when I increase the weight more and more, the error profile of 25 iterations goes down and gradually up. What is happened here? By the way, I think I got the best result is at weight 5e10 but the error profile is not monotonically decreased. In this case, which weight I should choose?