Can I checking the correct implementation for gradient descent algorithm by looking at if the loss is monotonically decreasing? The tricky thing of manually implement optimization algorithm is that, even there are some errors, such as wrong gradient, the algorithm still can work in some way, i.e., decrease the objective, and even find the optimal parameters.
I am manually implementing gradient boosting algorithm (gradient descent), can I checking the correct implementation for gradient boosting algorithm by looking at if the loss is monotonically decreasing?
For example, I am plotting objective function for $200$ iterations in left subplot, and plotting the diff(L_trace)>0 in the right subplot.

There are 2 cases in right sub-plot where the objective is not monotonically decreasing, so, can we know something wrong with the algorithm?
 A: There is something "wrong" with the algorithm, gradient descent (its very name is an example of false advertising). That doesn't mean there is a mistake in your implementation.
Gradient descent, i.e., steepest descent without use of line search or trust regions, need not converge, and can actually diverge, even on a strictly convex function, even in one dimension.
For example, apply gradient descent to the function $f(x) = x^4$. Its gradient is $4x^3$.  Apply gradient descent with a learning rate of $1.0$; this is the classic steepest descent, implemented without the protection afforded by line search or trust regions.  
Start at $x=1$. The gradient is $4$, so the next point is $-3$. Its gradient is $-108$, so the next point is $105$. its gradient is $4630500$, so the next point is $-4630395$.   Hmm, this seems to be diverging.  On a nice strictly convex function. Gradient descent has actually achieved monotonic increase in the objective function.
So yes, there is no guarantee gradient descent will descend on every iteration.  And even with a smaller learning rate, it may not necessarily be small enough to ensure descent.
A: When I was learning SGD, this[1] left me with a strong impression and I quote Leon Bottou as possible/alternative answer/solution to your question,
"During the last twenty years, I have often been approached for advice in setting the learning rates $\gamma_t$ of some rebellious stochastic gradient descent program. My advice is to forget about the learning rates and check that the gradients are computed correctly. This reply is biased because people who compute the gradients correctly quickly find that setting small enough learning rates is easy. Those who ask usually have incorrect gradients. Carefully checking each line of the gradient computation code is the wrong way to check the gradients. Use finite differences...." (page 8, [1])
[1] Bottou, L. (2012). Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade (pp. 421-436). Springer Berlin Heidelberg.
A: Checking your gradient against Finite differences method aka numerical differentiation is a good way to find errors.
See this: https://en.wikipedia.org/wiki/Numerical_differentiation
