1
$\begingroup$

I have a model which has lot of dummy variables representing seasonalities, day of week, day pf month, month of year, week of year etc. My dependent variable is total number of movie tickets that will be sold for a show. The independent variables are the number of days left before the show, seasonality dummy variables (day of week, month of year, holiday), price, tickets sold till date, movie rating. Also, please note that movie hall's capacity is fixed. That is, it can host maximum of x number of people only. I am creating a linear regression solution.

This is a multivariate dataset. For every date, there are 90 duplicate rows, representing days before the show. So, for 1 Jan 2016 there are 90 records. There is a 'lead_time' variable which gives me number of days before the show. So for 1 Jan 2016, if lead_time has value 5, it means it will have tickets sold until 5 days before the show date. In the dependent variable, total tickets sold, I will have the same value 90 times.

I have added some polynomial features and interaction variables to account for non-linearity.

I used Breusch-Pagan Test to test for heteroskedasticity and it appears to be significant.

BP Test

 bptest(fit.lm)

    studentized Breusch-Pagan test

data:  fit.lm
BP = 10941, df = 98, p-value < 2.2e-16

When i try to test for variables which is responsible for multi-collinearity, i am not getting any direction, as none of the variables have VIF>10.

VIF output

DAAG::vif(fit.lm)
              poly(LeadTime, 5)1               poly(LeadTime, 5)2 
                               2.6327                                1.4248 
              poly(LeadTime, 5)3               poly(LeadTime, 5)4 
                               1.1465                                1.0832 
              poly(LeadTime, 5)5                      ConfPricePerTicket 
                               1.0649                                2.9649 
                    Cml_CanceledTickets              poly(ConfirmedTickets, 2)1 
                               9.2778                                2.6670 
             poly(ConfirmedTickets, 2)2                           UnsoldTickets 
                               1.5045                                3.2374 
           poly(CancPricePerTicket, 2)1            poly(CancPricePerTicket, 2)2 
                               2.9172                                1.1571 
                          IMDB_Rating                          
                               2.1780                                       MOY.2                                 MOY.6 
                               8.0781                                2.0724 
                                DOM.3                                 DOM.7 
                               1.2178                                1.2113 
                               DOM.11                                DOM.15 
                               1.2714                                1.2733 
                               DOM.23                                DOM.31 
                               1.2022                                1.3208 
                                WOY.5                                 WOY.9 
                               1.7233                                2.1015 
                               WOY.13                                WOY.17 
                               1.1295                                1.1275 
                               WOY.41                                WOY.45 
                               1.1816                                1.2130 
                               WOY.53                                 DOW.3 
                               1.3630                                5.2812 
                                DOM.2                                 DOM.6 
                               1.2208                                1.2102 
                               DOM.10                                DOM.14 
                               1.2221                                1.2741 
                               DOM.18                                DOM.26 
                               1.2013                                1.2247 
                               DOM.30                                 WOY.4 
                               1.3222                                1.2616 
                                WOY.8                                WOY.12 
                               3.3542                                1.1042 
                               WOY.16                                WOY.20 
                               1.1382                                1.1003 
                               WOY.24                                WOY.40 
                               1.5572                                1.1708 
                               WOY.44                                WOY.48 
                               1.2445                                1.2173 
                               WOY.52                                MOY.12 
                               1.6721                                2.6124 
                                DOM.5                                DOM.13 
                               1.2106                                1.2656 
                               DOM.17                                DOM.25 
                               1.2612                                1.2255 
                               DOM.29                                 WOY.3 
                               1.2491                                1.2516 
                                WOY.7                                WOY.11 
                               3.4246                                1.1394 
                               WOY.15                                WOY.19 
                               1.1239                                1.0931 
                               WOY.23                                WOY.39 
                               1.5247                                1.2063 
                               WOY.43                                WOY.47 
                               1.2176                                1.2237 
                               WOY.51                                 DOM.4 
                               1.5773                                1.2293 
                               DOM.12                                DOM.16 
                               1.2589                                1.2770 
                               DOM.20                                DOM.24 
                               1.1986                                1.1987 
                               DOM.28                                 WOY.2 
                               1.2521                                1.2243 
                                WOY.6                                WOY.10 
                               3.3613                                1.1349 
                               WOY.14                                WOY.18 
                               1.1027                                1.1011 
                               WOY.22                                WOY.42 
                               1.1107                                1.2115 
                               WOY.46               Cml_CanceledTickets:DOW.2 
                               1.2785                                1.2743 
                  DOW.3:LeadTime               Cml_CanceledTickets:DOW.3 
                               5.1767                                1.6566 
                  LeadTime:DOW.4               Cml_CanceledTickets:DOW.4 
                               1.3877                                1.3173 
                  LeadTime:DOW.5                   LeadTime:DOW.6 
                               1.3651                                1.3908 
              Cml_CanceledTickets:DOW.6                   LeadTime:DOW.7 
                               1.4448                                1.3758 
                 DOW.7:ConfirmedTickets Cml_ConfirmedTickets:CanRevenuePerShow 
                               1.2916                                3.4255 
  CanRevenuePerShow:CancPricePerTicket    ConfPricePerTicket:Cml_CanceledTickets 
                               5.1226                                5.2405 
      Cml_CanceledTickets:LeadTime          LeadTime:ConfirmedTickets 
                               2.7752                                1.6774 

Regression output.

> summary(fit.lm)

Call:
lm(formula = sqrt(Tot_TicketSold) ~ poly(LeadTime, 5) + 
    ConfPricePerTicket + Cml_CanceledTicket + poly(ConfirmedTicket, 
    2) + UnsoldTicket + poly(CancPricePerShow, 2) + IMDB_Rating
+ MOY.2 + MOY.6 + 
    DOM.3 + DOM.7 + DOM.11 + DOM.15 + DOM.23 + DOM.31 + WOY.5 + 
    WOY.9 + WOY.13 + WOY.17 + WOY.41 + WOY.45 + WOY.53 + DOW.3 + 
    DOM.2 + DOM.6 + DOM.10 + DOM.14 + DOM.18 + DOM.26 + DOM.30 + 
    WOY.4 + WOY.8 + WOY.12 + WOY.16 + WOY.20 + WOY.24 + WOY.40 + 
    WOY.44 + WOY.48 + WOY.52 + MOY.12 + DOM.5 + DOM.13 + DOM.17 + 
    DOM.25 + DOM.29 + WOY.3 + WOY.7 + WOY.11 + WOY.15 + WOY.19 + 
    WOY.23 + WOY.39 + WOY.43 + WOY.47 + WOY.51 + DOM.4 + DOM.12 + 
    DOM.16 + DOM.20 + DOM.24 + DOM.28 + WOY.2 + WOY.6 + WOY.10 + 
    WOY.14 + WOY.18 + WOY.22 + WOY.42 + WOY.46 + DOW.2:Cml_CanceledTickets + 
    DOW.3:LeadTime + DOW.3:Cml_CanceledTickets + DOW.4:LeadTime + 
    DOW.4:Cml_CanceledTickets + DOW.5:LeadTime + DOW.6:LeadTime + 
    DOW.6:Cml_CanceledTickets + DOW.7:LeadTime + DOW.7:ConfirmedTickets + 
    Cml_ConfirmedTickets:CanRevenuePerShow + CanRevenuePerShow:CancPricePerTicket + 
    ConfPricePerTicket:Cml_CanceledTickets + LeadTime:Cml_CanceledTickets + 
    LeadTime:ConfirmedTickets, data = train_data)

Residuals:
    Min      1Q  Median      3Q     Max 
-3.9799 -0.7983 -0.0397  0.6973  7.3669 

Coefficients:
                                        Estimate Std. Error  t value Pr(>|t|)    
(Intercept)                            2.002e+01  4.472e-01   44.768  < 2e-16 ***
poly(LeadTime, 5)1                2.059e+01  2.058e+00   10.004  < 2e-16 ***
poly(LeadTime, 5)2               -3.223e+01  1.514e+00  -21.291  < 2e-16 ***
poly(LeadTime, 5)3                1.201e+01  1.358e+00    8.841  < 2e-16 ***
poly(LeadTime, 5)4               -7.591e+00  1.320e+00   -5.751 8.93e-09 ***
poly(LeadTime, 5)5                6.311e+00  1.309e+00    4.822 1.43e-06 ***
ConfPricePerTicket                       2.434e-02  2.414e-03   10.082  < 2e-16 ***
Cml_CanceledTicket                     -2.441e-02  3.327e-03   -7.337 2.21e-13 ***
poly(ConfirmedTicket, 2)1              -1.466e+01  2.071e+00   -7.077 1.49e-12 ***
poly(ConfirmedTicket, 2)2              -5.880e+00  1.556e+00   -3.779 0.000157 ***
UnsoldTicket                           -5.091e-02  6.085e-04  -83.662  < 2e-16 ***
poly(CancPricePerTicket, 2)1            -9.686e-01  2.166e+00   -0.447 0.654803    
poly(CancPricePerTicket, 2)2            -5.411e+00  1.364e+00   -3.966 7.32e-05 ***  
IMDB_Rating                          1.859e-01  2.441e-02    7.614 2.71e-14 ***
MOY.2                                  9.187e-02  5.256e-02    1.748 0.080483 .  
MOY.6                                  3.599e-01  3.584e-02   10.042  < 2e-16 ***
DOM.3                                 -2.672e-01  3.258e-02   -8.202 2.41e-16 ***
DOM.7                                  1.559e-01  3.327e-02    4.686 2.79e-06 ***
DOM.11                                 1.106e-01  3.409e-02    3.243 0.001182 ** 
DOM.15                                 9.440e-01  3.411e-02   27.676  < 2e-16 ***
DOM.23                                 2.173e-01  3.314e-02    6.557 5.54e-11 ***
DOM.31                                 3.335e-01  4.455e-02    7.486 7.22e-14 ***
WOY.5                                 -3.387e+00  4.719e-02  -71.773  < 2e-16 ***
WOY.9                                 -2.287e+00  5.211e-02  -43.877  < 2e-16 ***
WOY.13                                -3.037e+00  5.372e-02  -56.539  < 2e-16 ***
WOY.17                                 3.459e+00  5.367e-02   64.442  < 2e-16 ***
WOY.41                                -7.566e-01  3.908e-02  -19.363  < 2e-16 ***
WOY.45                                -1.918e+00  3.959e-02  -48.452  < 2e-16 ***
WOY.53                                 7.761e-01  7.787e-02    9.967  < 2e-16 ***
DOW.3                                 -2.619e-01  3.520e-02   -7.439 1.03e-13 ***
DOM.2                                 -2.686e-01  3.262e-02   -8.233  < 2e-16 ***
DOM.6                                 -3.594e-01  3.325e-02  -10.809  < 2e-16 ***
DOM.10                                 7.569e-02  3.426e-02    2.209 0.027145 *  
DOM.14                                 1.390e+00  3.412e-02   40.740  < 2e-16 ***
DOM.18                                 4.393e-01  3.313e-02   13.258  < 2e-16 ***
DOM.26                                 1.947e-01  3.345e-02    5.819 5.96e-09 ***
DOM.30                                -3.835e-01  3.658e-02  -10.484  < 2e-16 ***
WOY.4                                 -4.082e+00  4.038e-02 -101.089  < 2e-16 ***
WOY.8                                 -2.144e+00  6.584e-02  -32.566  < 2e-16 ***
WOY.12                                -2.773e+00  5.311e-02  -52.208  < 2e-16 ***
WOY.16                                 3.674e+00  5.392e-02   68.143  < 2e-16 ***
WOY.20                                -2.441e-01  5.302e-02   -4.604 4.16e-06 ***
WOY.24                                -6.706e-01  6.307e-02  -10.633  < 2e-16 ***
WOY.40                                -2.239e+00  3.890e-02  -57.555  < 2e-16 ***
WOY.44                                -2.752e+00  4.010e-02  -68.625  < 2e-16 ***
WOY.48                                -1.475e+00  3.966e-02  -37.194  < 2e-16 ***
WOY.52                                -7.096e-01  4.649e-02  -15.265  < 2e-16 ***
MOY.12                                -1.595e+00  2.879e-02  -55.395  < 2e-16 ***
DOM.5                                 -4.717e-01  3.326e-02  -14.181  < 2e-16 ***
DOM.13                                 9.993e-01  3.401e-02   29.384  < 2e-16 ***
DOM.17                                 1.043e+00  3.395e-02   30.715  < 2e-16 ***
DOM.25                                 2.394e-01  3.346e-02    7.154 8.56e-13 ***
DOM.29                                -2.049e-01  3.463e-02   -5.916 3.33e-09 ***
WOY.3                                 -3.514e+00  4.022e-02  -87.373  < 2e-16 ***
WOY.7                                 -1.906e+00  6.652e-02  -28.649  < 2e-16 ***
WOY.11                                -3.299e+00  5.395e-02  -61.144  < 2e-16 ***
WOY.15                                -2.180e-01  5.358e-02   -4.068 4.75e-05 ***
WOY.19                                 2.336e+00  5.284e-02   44.199  < 2e-16 ***
WOY.23                                -4.309e-01  6.241e-02   -6.904 5.12e-12 ***
WOY.39                                -1.385e+00  3.948e-02  -35.081  < 2e-16 ***
WOY.43                                -3.746e+00  3.967e-02  -94.447  < 2e-16 ***
WOY.47                                -2.232e+00  3.977e-02  -56.135  < 2e-16 ***
WOY.51                                 4.526e-01  4.515e-02   10.024  < 2e-16 ***
DOM.4                                 -2.360e-01  3.274e-02   -7.208 5.75e-13 ***
DOM.12                                 4.651e-01  3.392e-02   13.713  < 2e-16 ***
DOM.16                                 9.722e-01  3.416e-02   28.460  < 2e-16 ***
DOM.20                                -2.088e-01  3.309e-02   -6.309 2.82e-10 ***
DOM.24                                 3.294e-01  3.310e-02    9.953  < 2e-16 ***
DOM.28                                -1.455e-01  3.382e-02   -4.300 1.71e-05 ***
WOY.2                                 -3.579e+00  4.124e-02  -86.779  < 2e-16 ***
WOY.6                                 -2.672e+00  6.591e-02  -40.543  < 2e-16 ***
WOY.10                                -1.633e+00  4.757e-02  -34.336  < 2e-16 ***
WOY.14                                -1.979e+00  5.308e-02  -37.283  < 2e-16 ***
WOY.18                                 3.261e+00  5.304e-02   61.478  < 2e-16 ***
WOY.22                                -7.811e-01  5.327e-02  -14.663  < 2e-16 ***
WOY.42                                -3.150e+00  3.957e-02  -79.609  < 2e-16 ***
WOY.46                                -2.646e+00  4.065e-02  -65.088  < 2e-16 ***
Cml_CanceledTicket:DOW.2                6.343e-03  3.509e-03    1.808 0.070658 .  
DOW.3:LeadTime                    7.647e-03  6.550e-04   11.675  < 2e-16 ***
Cml_CanceledTicket:DOW.3                8.908e-03  3.982e-03    2.237 0.025298 *  
LeadTime:DOW.4                    4.314e-03  3.391e-04   12.722  < 2e-16 ***
Cml_CanceledTicket:DOW.4                7.537e-03  3.348e-03    2.251 0.024389 *  
LeadTime:DOW.5                    8.495e-03  3.364e-04   25.256  < 2e-16 ***
LeadTime:DOW.6                    1.721e-02  3.395e-04   50.679  < 2e-16 ***
Cml_CanceledTicket:DOW.6                6.295e-03  2.917e-03    2.158 0.030896 *  
LeadTime:DOW.7                    2.352e-02  3.394e-04   69.302  < 2e-16 ***
DOW.7:ConfirmedTicket                   3.825e-02  6.695e-03    5.713 1.11e-08 ***
Cml_ConfirmedTicket:CanRevenuePerShow -1.782e-05  3.414e-06   -5.221 1.78e-07 ***
CanRevenuePerShow:CancPricePerTicket    4.698e-06  1.524e-06    3.084 0.002045 ** 
ConfPricePerTicket:Cml_CanceledTicket    -1.430e-03  2.663e-04   -5.370 7.90e-08 ***
Cml_CanceledTicket:LeadTime        2.375e-03  1.026e-04   23.152  < 2e-16 ***
LeadTime:ConfirmedTicket           6.860e-04  1.325e-04    5.177 2.26e-07 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.268 on 55593 degrees of freedom
Multiple R-squared:  0.7473,    Adjusted R-squared:  0.7468 
F-statistic:  1677 on 98 and 55593 DF,  p-value: < 2.2e-16

Residual Vs. Fitted values plot

enter image description here

$\endgroup$
  • $\begingroup$ I haven't really absorbed all your question, but there's no reason heteroskedasticity and collinearity (vif) should be related. What exactly is the question? $\endgroup$ – Peter Ellis Sep 18 '16 at 5:52
  • $\begingroup$ What are the ways to detect the direction from where heteroskedasticity is coming from? I guess through VIF test? Thats what i did and i took care of variables where vif was greater than 10. $\endgroup$ – Enthusiast Sep 18 '16 at 5:58
  • $\begingroup$ First things first: "My dependent variable is total number of movie tickets that will be sold for a show." means that you model a count variable and should use a GLM, i.e., Poisson regression. Then, high order polynomials are usually not sensible in regression. If you have strong non-linearities a Generalized Additive Model (GAM) would be a more sensible and robust way to proceed. $\endgroup$ – Roland Sep 19 '16 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.