1
$\begingroup$

Doing my thesis I came up on a problem where I can't find the answer on. I have a dataset with only categorical predictors with sometimes many levels and a numeric outcome variable bounded between zero and 1. The outcome variable is a given discount percent variable. It is not normally distributed with more then two modes:

enter image description here

In this case I used 5 predictor variables and Nrows around 850.000 I tried several different algorithms ( betareg, lm, anova, random forest, svr ) But every time this residuals vs fitted plot comes up:

enter image description here

Also when I use a smaller training you will see this downwards trend.

My question is how to interpet this plot and is there a way to deal with this since all my predictor variables are categoric?

I try to find out witch variables of a 200 variables wide dataset have a impact on the given discounts.

Thanks,

$\endgroup$
  • $\begingroup$ You have non-constant variance in residuals, model behaves more badly where your model predicts bigger discount. $\endgroup$ – Analyst Jun 25 '18 at 19:38
  • 1
    $\begingroup$ If you use beta regression, then don't worry about residuals being non-normally distributed. That's an assumption of standard linear regression with a Gaussian (identity) link function. $\endgroup$ – Mark White Aug 6 '18 at 18:07
1
$\begingroup$

If your response is bounded between 0 and 1, you don't conform to the assumptions of a linear regression, and it is thus not surprising that you don't get normal residuals.

You can model 0/1 proportional data in a number of ways. A common approach is a beta regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.