I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python implementation. Because I want to try to predict these extreme values, excluding these data points is not an option. Some of the built-in objective and evaluation metrics commonly used for regression like RMSE and even MAE can be overly sensitive to outliers, and I think it's impacting my results.
In addition to RMSLE (which I believe should be less sensitive to these sorts of extreme values), I would like to experiment with custom objective and evaluation functions. I saw this article that I think can help me get started:
However, I don't really understand how to calculate the gradient or the hessian given an objective function. Here is my current understanding of why these functions are necessary (and please someone correct me if I'm misunderstanding).
(1) The gradient is roughly analogous (or maybe it is) to the first derivative. My understanding is that this function helps the algorithm determine if a local minimum or maximum is being approached (i.e., if the first derivative is 0, then a local min/max has been achieved).
(2) The hessian is roughly analogous to the second derivative. My understanding is that this function helps the algorithm determine if the local min/max is a minimum or a maximum (i.e., if the second derivative is positive, then a local min has been achieved).
(3) However, based on what I'm reading online, the gradient and hessian seem to be based on partial derivatives.
What I don't understand is how to go from this:
I feel like if I have a better understanding of how the gradient and hessian were calculated for RMSLE, I'll be able to do this for other objective functions too. I've tried looking up articles about the gradient and hessian online, but I'm having a very hard time figuring out how to calculate them for RMSLE.
Can anyone help walk me through how to calculate these, or point me to some online resources that can help me learn how to calculate the gradient and hessian given a particular objective function?