1
$\begingroup$

I am trying to predict the time (in seconds) a process would take based on the computer's disk throughput (measured by dd and reported in MiB/s) and the amount of user data that needs to be processed. I have collected data with several levels of user data (64, 124, 256, 512, 1024, and 2048 MiB) on several Amazon EC2 instance/computer types. I have around 2500 observations.

The time (ec2$elapsed_time) and the throughput (ec2$dd_128k) look like this:

enter image description here

Because I do not control the throughput of the disks that I get assigned per instance, there are some gaps in the histogram. So I decided to use binning for the throughput data. I grouped it in 8 categories (bin #5 has no observations):

enter image description here

  • Bin 1: <= 15
  • Bin 2: 15.01 - 35
  • Bin 3: 35.01 - 55
  • Bin 4: 55.01 - 75
  • Bin 5: 75.01 - 95
  • Bin 6: 95.01 - 115
  • Bin 7: 115.01 - 135
  • Bin 8: 135+

To visualize the trend, I plotted the mean for each bin and grouped it by the amount of user data:

enter image description here

At first, I attempted to use a linear regression but the model could not explain the variation in the data properly and given that the data did not seem linear, I figured it was not appropriate. I was looking into using a GLM with a gamma distribution.

Using the script below, I attempted to fit a Gamma GLM to the data. Where dd_128k_bin is a categorical value (1-8) and mem_factor is also categorical (64, 128, 256, 512, 2048).

model <- glm(elapsed_time ~ dd_128k_bin * mem_factor, data=ec2, family = "Gamma")

This is what I get in the summary:

Call:
glm(formula = elapsed_time ~ dd_128k_bin * mem_factor, family = "Gamma", 
    data = ec2)

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-1.84829  -0.22591  -0.00416   0.10916   1.63584  

Coefficients:
                            Estimate Std. Error t value Pr(>|t|)    
(Intercept)                   6.4516     0.5198  12.413  < 2e-16 ***
dd_128k_bin2                 -2.1475     0.5747  -3.737 0.000191 ***
dd_128k_bin3                 -1.6414     0.5659  -2.901 0.003757 ** 
dd_128k_bin4                 -0.5071     0.5887  -0.861 0.389128    
dd_128k_bin6                 -2.5555     0.5652  -4.522 6.42e-06 ***
dd_128k_bin7                 -1.6285     0.5879  -2.770 0.005648 ** 
dd_128k_bin8                 -1.7198     0.6446  -2.668 0.007678 ** 
mem_factor128                -1.4433     0.6580  -2.193 0.028366 *  
mem_factor256                -2.9428     0.5916  -4.974 7.01e-07 ***
mem_factor512                -6.0697     0.5207 -11.658  < 2e-16 ***
mem_factor1024               -6.4028     0.5198 -12.319  < 2e-16 ***
mem_factor2048               -6.4349     0.5198 -12.381  < 2e-16 ***
dd_128k_bin2:mem_factor128    0.7405     0.7315   1.012 0.311487    
dd_128k_bin3:mem_factor128    0.5700     0.7187   0.793 0.427787    
dd_128k_bin4:mem_factor128    0.2581     0.7473   0.345 0.729791    
dd_128k_bin6:mem_factor128    0.8439     0.7194   1.173 0.240872    
dd_128k_bin7:mem_factor128    0.6524     0.7491   0.871 0.383924    
dd_128k_bin8:mem_factor128    0.7655     0.8276   0.925 0.355103    
dd_128k_bin2:mem_factor256    1.4049     0.6595   2.130 0.033257 *  
dd_128k_bin3:mem_factor256    0.9708     0.6462   1.502 0.133122    
dd_128k_bin4:mem_factor256    0.3132     0.6710   0.467 0.640676    
dd_128k_bin6:mem_factor256    1.4496     0.6466   2.242 0.025047 *  
dd_128k_bin7:mem_factor256    0.9485     0.6719   1.412 0.158179    
dd_128k_bin8:mem_factor256    0.9939     0.7387   1.346 0.178572    
dd_128k_bin2:mem_factor512    3.1667     0.5810   5.450 5.52e-08 ***
dd_128k_bin3:mem_factor512    2.3968     0.5692   4.211 2.63e-05 ***
dd_128k_bin4:mem_factor512    2.0393     0.5962   3.420 0.000635 ***
dd_128k_bin6:mem_factor512    3.7534     0.5731   6.549 7.01e-11 ***
dd_128k_bin7:mem_factor512    3.0910     0.5980   5.169 2.54e-07 ***
dd_128k_bin8:mem_factor512    3.2402     0.6632   4.885 1.10e-06 ***
dd_128k_bin2:mem_factor1024   2.1949     0.5747   3.819 0.000137 ***
dd_128k_bin3:mem_factor1024   1.7589     0.5659   3.108 0.001906 ** 
dd_128k_bin4:mem_factor1024   0.7333     0.5889   1.245 0.213159    
dd_128k_bin6:mem_factor1024   3.4384     0.5677   6.057 1.60e-09 ***
dd_128k_bin7:mem_factor1024   2.6521     0.5911   4.487 7.56e-06 ***
dd_128k_bin8:mem_factor1024   2.7587     0.6505   4.241 2.31e-05 ***
dd_128k_bin2:mem_factor2048   2.1517     0.5747   3.744 0.000185 ***
dd_128k_bin3:mem_factor2048   1.6706     0.5659   2.952 0.003185 ** 
dd_128k_bin4:mem_factor2048   0.5531     0.5887   0.939 0.347606    
dd_128k_bin6:mem_factor2048   3.0518     0.5659   5.393 7.60e-08 ***
dd_128k_bin7:mem_factor2048   2.1888     0.5888   3.717 0.000206 ***
dd_128k_bin8:mem_factor2048   2.2907     0.6463   3.544 0.000401 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for Gamma family taken to be 0.1947056)

    Null deviance: 8942.38  on 2519  degrees of freedom
Residual deviance:  661.38  on 2478  degrees of freedom
AIC: 2578.1

Number of Fisher Scoring iterations: 6

And these are the plots associated with the model: enter image description here

Using the RsqGLM() function, I obtained an R^2 of 0.95 with the CoxSnell method.

I am concerned about the assumptions that need to be satisfied. For linear models, you are usually concerned with the Normal Q-Q plot following the drawn line closely. In this case, since the data is not normal, it does not. So I am wondering if this model is appropriate even when it's R^2 value seems appropriate. Is there anything else that I should consider to validate whether this model is appropriate?


Edit: I hadn't considered that binning could introduce bias. I checked the model without binning and these are the results:

model <- glm(elapsed_time ~ dd_128k * mem_factor, data=ec2, family = "Gamma")

R^2 = 0.93

glm(formula = elapsed_time ~ dd_128k * mem_factor, family = "Gamma", 
    data = ec2)

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-1.60095  -0.42446   0.01102   0.16540   1.70529  

Coefficients:
                        Estimate Std. Error t value Pr(>|t|)    
(Intercept)             5.304060   0.299458  17.712  < 2e-16 ***
dd_128k                -0.006009   0.003406  -1.764 0.077843 .  
mem_factor128          -1.032953   0.387206  -2.668 0.007686 ** 
mem_factor256          -2.155936   0.348180  -6.192 6.92e-10 ***
mem_factor512          -4.866027   0.305453 -15.931  < 2e-16 ***
mem_factor1024         -5.361043   0.299505 -17.900  < 2e-16 ***
mem_factor2048         -5.320488   0.299461 -17.767  < 2e-16 ***
dd_128k:mem_factor128   0.002454   0.004428   0.554 0.579522    
dd_128k:mem_factor256   0.002412   0.003960   0.609 0.542493    
dd_128k:mem_factor512   0.019934   0.003572   5.580 2.66e-08 ***
dd_128k:mem_factor1024  0.012081   0.003413   3.540 0.000408 ***
dd_128k:mem_factor2048  0.007703   0.003407   2.261 0.023833 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for Gamma family taken to be 0.3468551)

    Null deviance: 8942.38  on 2519  degrees of freedom
Residual deviance:  875.08  on 2508  degrees of freedom
AIC: 3259

Number of Fisher Scoring iterations: 6

enter image description here

$\endgroup$
3
  • $\begingroup$ I don't understand the value of doing the binning; few things will be worth what it costs you in bias etc. $\endgroup$
    – Glen_b
    Commented Sep 10, 2021 at 6:53
  • $\begingroup$ @Glen_b thanks for your suggestion! I made an edit to include the non-binning model. $\endgroup$ Commented Sep 10, 2021 at 16:39
  • $\begingroup$ That's likely to be helpful. There's some structure in the residuals (beyond the obvious lack of fit in the mean and spread). The binning maybe helpful in terms of identifying models. I'd take logs on the y-axis on your first plot to see what's going on. I would also tend to look at a log link rather than the canonical link (although the Gamma's canonical link has a useful interpretation -- 1/time is speed). $\endgroup$
    – Glen_b
    Commented Sep 11, 2021 at 0:41

1 Answer 1

1
$\begingroup$

The most important aspect of a GLM family that determines it suitability for a modelling a particular response variable is whether it captures the mean-variance relationship correctly. Your elapsed data is strictly positive and highly right skew and so is likely to show a strongly increasing mean-variance relationship.

Plotting standardized residuals vs fitted values is a good way to check the mean-variance relationship. You want to check that the spread of the standardized residuals doesn't show a trend with the size of the fitted values. In your data, there is no obvious trend in terms of absolute residual so the mean-variance relationship assumed by the gamma family seems ok. In your plot, the vertical spread of the standardized residuals is about the same for small fitted values as for large fitted values, which is what you want to see. The gamma family assumes a constant coefficient of variation (standard deviation proportional to the mean), an assumption that is often appropriate for positive quantitative data.

The other diagnostic plots are less important. With 2500 observations, the GLM statistical tests are not very sensitive the exact form of the gamma probability distribution. However, if you would like to make a q-q plot of the residuals you should first transform to normality as follows:

library(statmod)
r <- qresid(model)
qqnorm(r)
qqline(r)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.