1
$\begingroup$

I am dealing with very right-skewed distribution of y which seems to follow a tweedie distribution. and I found out it would give a higher performance to change lgbm's loss function to a "tweedie".

As far as I know, a boosting model is non-parametric which means it has no assumption regarding a distribution of data and at the same time, gradient boosting has a loss function which varies on the target distribution.

At this point, I kinda felt contradictory since it has no assumption of distribution but still its loss function depends on the target distribution..

Doesn't it have a direct connection to whether an algorithm is parametric or not that its loss funtion depends on a distribution of target variable?

Improve this question
$\endgroup$
6
  • $\begingroup$ Loss functions optimize errors, differences between true values and observed values. Errors can have a distribution which is often different from data distribution y. Non parametric models syntagma refer to data distribution, with no connection to errors distribution $\endgroup$
    – rapaio
    Aug 26, 2021 at 8:04
  • $\begingroup$ @rapaio Thanks. Well then do you mean by your comment that it was just by accident that setting the loss function to tweedie loss which is close to y's distribution? if not, could you explain more of your comment please? I am a little confused. $\endgroup$
    – Kosney
    Aug 26, 2021 at 14:18
  • $\begingroup$ The loss/risk function is just a criteria to minimize. It has nothing to do with distributional assumptions. Yes, the least squares risk can be derived from the gaussian error model, but it is globally minimized by the conditional mean under zero assumptions. The same is true for logistic, poisson, gamma, etc. Logistic is useful because it ensures the conditional mean estimate is bounded in [0,1]. Poisson risk is useful if you know the conditional mean is nonnegative. All these risk functions allow for different parameterizations/link functions for the conditional mean. $\endgroup$
    – user327671
    Aug 27, 2021 at 0:03
  • $\begingroup$ @LarsvanderLaan Thanks for detailed comments! So, the distributional assumptions are not things like that setting a loss function with a little bit of pre-knowledge of the target distribution(or should I say a conditional mean to be more technical?) $\endgroup$
    – Kosney
    Aug 27, 2021 at 5:32
  • $\begingroup$ I would think of the different loss functions as providing different "link functions", i.e. the transformation that maps a function in the optimization space to the conditional mean space (expit for binomial, exp for poisson, identity for gaussian, something for tweedle, etc). The true regression function might have a more simple representation with different link functions. All these losses are nonparametric but xgboost for example might perform better for different losses depending on the situation. $\endgroup$
    – user327671
    Aug 27, 2021 at 17:55

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.