# How to improve linear regression model with extremely right skewed predictors and large number of zero values as observations

I have the dataframe as:

> head(df,20)
Field.1    Complexity RQT.1 RQT.2 RQT.3 EQT.1 EQT.2 EQT.3 Outcome
1    Application 1     M    48    13     1    1594   945    50     832
2    Application 2     C     3     1     0     0     0     0       0
3    Application 3     C     1    31     2     0     0     0       0
4    Application 4     C     0     1     0     0     0     0       0
5    Application 5     M    11     5     0     0     0     0       0
6    Application 6     C     3     0     0     1     0     0      18
7    Application 6     C    48     8     0   346    21     0      60
8    Application 6     C     2     0     0    24     0     0       1
9    Application 6     C     0     1     0     0    22     0       0
10   Application 7     C     1     4     0    25    94     0       6
11   Application 8     M     0     2     0     0    44     0       1
12   Application 6     C     2     2     0    45   201     0       7
13   Application 6     C     0     1     0     0    10     0       2
14   Application 9     M     0     6     0     0    37     0       0
15  Application 10     C   138    22     0  2438   497     0    3122
16   Application 4     C     1    17     1     1   250    31      70
17   Application 4     C     0     1     0     0    25     0       1
18  Application 11     M     5    11     0    71    55     0      26
19   Application 4     C     0    22     0     0   184     0      25
20   Application 4     C     0     1     0     0     1     0       0

str(df)  # dim of 442 x 9
## 'data.frame':    442 obs. of  9 variables:
##  $Field.1 : Factor w/ 49 levels "Application 1",..: 1 12 23 34 45 46 46 46 46 47 ... ##$ Complexity: Factor w/ 3 levels "C","M","S": 2 1 1 1 2 1 1 1 1 1 ...
##  $RQT.1 : int 48 3 1 0 11 3 48 2 0 1 ... ##$ RQT.2     : int  13 1 31 1 5 0 8 0 1 4 ...
##  $RQT.3 : int 1 0 2 0 0 0 0 0 0 0 ... ##$ EQT.1     : int  1594 0 0 0 0 1 346 24 0 25 ...
##  $EQT.2 : int 945 0 0 0 0 0 21 0 22 94 ... ##$ EQT.3     : int  50 0 0 0 0 0 0 0 0 0 ...
##  \$ Outcome   : int  832 0 0 0 0 18 60 1 0 6 ...


Now this dataset seems to be too imbalanced.....all the numeric var are highly right skewed and also there are lot of zero values (by the way these are not to be treated as missing)

library(e1071)
sapply(df.num,skewness)
##   RQT.1   RQT.2   RQT.3   EQT.1   EQT.2   EQT.3 Outcome
##     5.2     4.1     6.8     9.6     7.5    11.4     9.9

colSums(df.num == 0)# no of rows having Zero values
##   RQT.1   RQT.2   RQT.3   EQT.1   EQT.2   EQT.3 Outcome
##     191     130     402     213     147     404     135


Lets see the scatter plot:

Lets see the distribution of outcome var:

As we can see the var is extremely rt skewed....with many zero values

Now i run the linear regression first with no transformation.(Convert the Complexity var into a factor and then into 2 dummies with C as reference level...

## Call:
## lm(formula = Outcome ~ . - Field.1, data = train.df)
## Residuals:
##    Min     1Q Median     3Q    Max
## -685.1   -8.6   12.8   31.7 1606.1
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)
## (Intercept) -41.3617    10.4143   -3.97  8.9e-05 ***
## Complexity2  19.6875    16.2356    1.21   0.2262
## Complexity3  35.1605    27.5110    1.28   0.2022
## RQT.1         1.4156     0.5366    2.64   0.0088 **
## RQT.2         0.4816     1.2024    0.40   0.6891
## RQT.3        18.6362    10.3355    1.80   0.0724 .
## EQT.1         0.4675     0.0348   13.44  < 2e-16 ***
## EQT.2         0.4258     0.0791    5.38  1.5e-07 ***
## EQT.3        -2.2936     1.0362   -2.21   0.0276 *
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## Residual standard error: 128 on 300 degrees of freedom
## Multiple R-squared:  0.809,  Adjusted R-squared:  0.804
## F-statistic:  159 on 8 and 300 DF,  p-value: <2e-16


Assumptions of normality/hetroskedasticity..etc NOT met....

Then I went for log tranformations:

  Call:
lm(formula = log(Outcome + 1) ~ Complexity + log(RQT.1 + 0.01) +
log(RQT.3 + 0.002) + log(EQT.1 + 1) + log(EQT.2 + 1) + log(EQT.3 +
0.001) + -Field.1, data = train.df)
Residuals:
Min     1Q Median     3Q    Max
-3.638 -0.731 -0.056  0.488  3.658
Coefficients:
Estimate Std. Error     t value     Pr(>|t|)
(Intercept)         0.32730    0.27749    1.18     0.24
Complexity2         0.14346    0.14468    0.99     0.32
Complexity3        -0.05001    0.24592   -0.20     0.84
log(RQT.1 + 0.01)  -0.00916    0.04517   -0.20     0.84
log(RQT.3 + 0.002)  0.12981    0.11122    1.17     0.24
log(EQT.1 + 1)      0.51327    0.05996    8.56  5.9e-16 ***
log(EQT.2 + 1)      0.35306    0.03464   10.19  < 2e-16 ***
log(EQT.3 + 0.001) -0.04204    0.08396   -0.50     0.62
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.1 on 301 degrees of freedom
Multiple R-squared:  0.611, Adjusted R-squared:  0.602
F-statistic: 67.6 on 7 and 301 DF,  p-value: <2e-16


The explanatory power of log tranformed var has reduced drastically.Also only heteroscedasticity assumption was satified.......

So my dilemma is how can i improve the model so that i get better predictions....should i go for quantile/polynomial or ridge/lasso regression to tackle the imbalanced nature of the data or maybe convert the outcome var into categorical var (bin it into say 3 levels).....??

## migrated from stackoverflow.comNov 14 '16 at 8:41

This question came from our site for professional and enthusiast programmers.

• How are you checking the the normality assumption has not been met? The assumption has nothing to do with the distribution of your predictors, but rather the distribution of the residuals – Simon Nov 14 '16 at 4:45
• One option is to bin the outcome as you say, but what is the nature of that variable? What is it measuring? – Simon Nov 14 '16 at 4:46
• @simon.....i m referring to normality of residual only..... – Nishant Nov 14 '16 at 5:19
• No to binning, yes to the question regarding what the outcome measures. – Roland Nov 14 '16 at 6:06
• Is the outcome a count? – mdewey Nov 14 '16 at 12:31