Timeline for Ordinal vs multinominal classification in XGboost: differences in one-hot encoding
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 18 at 23:52 | comment | added | usεr11852 |
Agreed, we need only the diagonal, it was more that we need (some of the) Hessian information and simple gradient information won't save us. I have not come across that derivation in particular but if you can derive it youself you might want to consider an auto-diff approach. It will be slower than a true derivation but it can also work.
|
|
| May 18 at 11:29 | comment | added | Royi |
Is there a place where the gradient and hessian of the proportional odds is derived for XGBoost? Also I think XGBoost require only the diagonal of the Hessian.
|
|
| Nov 20, 2023 at 9:06 | history | edited | usεr11852 | CC BY-SA 4.0 |
Clarifications based on questions
|
| Nov 20, 2023 at 9:00 | comment | added | usεr11852 | OK, anyway I will edit this post such it includes the clarifications for point 1. Let's keep them separately so it is clear what is asked and answered. | |
| Nov 20, 2023 at 8:27 | comment | added | deblue | hey, no sorry, it's another post actually this one (I mixed them up): stats.stackexchange.com/questions/631367/…. I describe there what problems I have with implementing the proportional odds assumptions. | |
| Nov 19, 2023 at 21:09 | comment | added | usεr11852 | Then our response is an indicator variable that is 1 if the target variable for the i-th observation is less than or equal to the j-th ordinal value, and 0 otherwise. So when we have something like $y_{ij} \log(\hat{p}_{ij})$ in our loss we represent the contribution to the loss from the correct predictions as this part is maximized when the predicted probability is equal to $1$ for the correct ordinal value. ($\hat{p}_{ij}$ being the predicted cumulative probability of the j-th ordinal value for the i-th observation.) | |
| Nov 19, 2023 at 21:01 | comment | added | usεr11852 |
(1). There are two options, without or with thresholding: A. We directly pick the class/label with the highest predicted probability (something akin to np.argmax(predicted_probs, axis=1) in Python) B. We apply thresholds to our predicted probability ranges to map them to ordinal values. (something akin to : np.argmax(predicted_probs > thresholds, axis=1) in Python). (2) Apologies, I don't see any loss in your post; (in any case) we can see this as a binary problem where we predict the cumulative probability of the j-th ordinal value for the i-th observation. (cont.)
|
|
| Nov 19, 2023 at 9:34 | comment | added | deblue | Ye, so can we you please expand on two points: (1) 'Then we can directly change the predicted probabilities to ordinal values', how exactly? and (2) What I am trying to figure out is how to implement the loss, gradient and Hessian given the proportional odds loss. In the post, I've written what I think is the loss (loglikelihood, cross-entropy) given the proportional odds assumption, but for this, I think you need to treat this as a binary problem, no? Can you please expand? Thanks. | |
| Nov 19, 2023 at 1:30 | history | edited | usεr11852 | CC BY-SA 4.0 |
added 72 characters in body
|
| Nov 19, 2023 at 1:13 | history | answered | usεr11852 | CC BY-SA 4.0 |