The tricky thing of manually implement optimization algorithm is that, even there are some errors, such as wrong gradient, the algorithm still can work in some way, i.e., decrease the objective, and even find the optimal parameters.
I am manually implementing gradient boosting algorithm (gradient descent), can I checking the correct implementation for gradient boosting algorithm by looking at if the loss is monotonically decreasing?
For example, I am plotting objective function for $200$ iterations in left subplot, and plotting the
diff(L_trace)>0 in the right subplot.
There are 2 cases in right sub-plot where the objective is not monotonically decreasing, so, can we know something wrong with the algorithm?