3
$\begingroup$

I'm trying to determine if the exponential distribution is a good model for a data set that I'm exploring. It doesn't have to be precise. I'm using the data for capacity planning (if it's a good fit) and for my own learning. I found a data set that I have ready access to and I wanted to test to see it fit a known distribution.

To test if it is a good fit, I've generated 2 plots from a data set with about 1.5M entries.

In the first plot, I take the data and generate an empirical CDF and plot the log of the CCDF and the linear model using geom_smooth. I'm doing this because if it is exponential (y ~= e^−λx), then the log of both sides gives log y ~= -λx which should be a straight line with slope -λ.

data <- sort(data) 
valid <- data[1:(length(data)*0.96)]  
ecdf <- (1:length(valid))/length(valid)
df <- data.frame(cbind(x=valid, y=ecdf))  

# if this is exponential, get the slope and the x intercept
lm(log(1-y)~x, data=df, subset=seq(1,length(valid)-1))

ggplot(aes(x=x, y=log(1-y)), data=df[1:length(valid)-1,]) +
    geom_line(color="blue") +
    geom_smooth(method="lm", color="red", size=0.3) +
    labs(x="Data", y="CCDF of Data Log(y)", title="Line of Best Fit on Log(CCDF) of Data")

Here is the result. It mostly fits except at the tail. Should I conclude that the exponential distribution is a (mostly) good fit?

enter image description here

Edit: here is the summary data from the model generated by lm for the Log(CCDF) plot:

Residuals:
    Min      1Q  Median      3Q     Max 
-9.7550 -0.0750  0.0044  0.1171  0.1328 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept)  1.239e-01  2.388e-04   518.8   <2e-16 ***
x           -1.868e-03  2.989e-07 -6248.6   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.1907 on 1473365 degrees of freedom
Multiple R-squared: 0.9636,     Adjusted R-squared: 0.9636 
F-statistic: 3.904e+07 on 1 and 1473365 DF,  p-value: < 2.2e-16 

Next I attempted to generate a Q-Q plot by generating a data set that is the same size as the empirical data set (about 1.5M) using a function that generates random numbers from an exponential distribution. From the lm above I used the x output and switched the sign as my lambda. I sorted both of them and plotted the sampled data as x and the empirical data as y.

Here is the result. Again, it's not quite x=y. Should I conclude that the exponential distribution is a good enough fit?

enter image description here

Edit: here is the summary data for the model from the Q-Q plot as generated by lm.

Residuals:
    Min      1Q  Median      3Q     Max 
-49.256  -9.214  -0.069  10.060 268.769 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -1.757e+01  4.086e-02  -429.9   <2e-16 ***
x            1.264e+00  6.078e-05 20803.7   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 32.82 on 1488715 degrees of freedom
Multiple R-squared: 0.9966,     Adjusted R-squared: 0.9966 
F-statistic: 4.328e+08 on 1 and 1488715 DF,  p-value: < 2.2e-16 

Thanks everyone for all of the helpful comments!

$\endgroup$
  • $\begingroup$ It looks like a very good fit apart from the tail, but without standard errors or actual data on the plots, there's no basis on which to assert that the impression of lack of fit is much of an issue. $\endgroup$ – Glen_b -Reinstate Monica May 20 '13 at 22:37
  • $\begingroup$ Thanks for the comment Glen_b. I'm working on the standard error. $\endgroup$ – drsnyder May 21 '13 at 15:16
  • 1
    $\begingroup$ Glen_b, I added the standard errors as generated by lm. $\endgroup$ – drsnyder May 21 '13 at 15:51
  • $\begingroup$ Thanks, but rather than standard errors of coefficients, I meant some sense of 'typical distance' on the plot. One might anticipate that even a pretty good model for the data would tend to deviate more in the tails, and in the case of an exponential, mostly on the right tail. Nonetheless I do get some sense that perhaps the appearance of poor fit in the extreme upper tail is real (the sample size is quite large); the question now is how much that impacts whatever you want to do with it. $\endgroup$ – Glen_b -Reinstate Monica May 21 '13 at 22:32
  • $\begingroup$ However, with those sort of sample sizes I can't think of many questions that can't be answered by operating with the empirical distribution, $\hat F_n$ or in some cases a kernel or logspline smooth of the density, rather than some assumed $F$, perhaps with a few fitted parameters. $\endgroup$ – Glen_b -Reinstate Monica May 21 '13 at 22:36
2
$\begingroup$

It depends on what you want to use this for. I can easily imagine situations in capacity planning where you would be most interested in extreme occurrences, as these peak events are what strains capacity most. If that is the case, then your tail behaviour would be a problem. I can also imagine other situations where the system is somewhat flexible so that they can absorb short peaks, in which case you would be more interested in typical behaviour. In that case your model may work.

$\endgroup$
  • $\begingroup$ Maarten, each data point is the size of a text document so the peaks aren't an issue. Since they are so few they can easily be absorbed. $\endgroup$ – drsnyder May 21 '13 at 15:15
  • $\begingroup$ I don't think "good fit?" is easy to answer without knowing purpose (Maarten's point) and without knowing what other distributions you tried. Did you look at the gamma distribution as a more general alternative, for example? Note that recasting a distribution fit problem as a regression problem is a little dubious: points on the distribution function are hardly independent. But the graphical evidence is clearly pertinent. As you have systematic departure from the exponential, two-parameter distributions should give closer fits, but you must judge whether that's worthwhile. $\endgroup$ – Nick Cox May 21 '13 at 16:01
  • $\begingroup$ Nick, thanks for the comments. As I mentioned in the question, it doesn't have to be precise. I'm trying to take a little bit of what I learned and apply it to a data set that I have access to. I don't think this is necessarily recasting the a distribution fit problem a regression problem (though I can see it looks that way wit the additional error data). I'm simply trying to use the few tools I know to try and determine if the exponential distribution is a good choice for modeling the data. I only tried the exponential distribution. I'll take a look at the gamma distribution. Thanks again. $\endgroup$ – drsnyder May 21 '13 at 17:18
  • $\begingroup$ You used lm(); that's regression, is it not? No harm done so long as you don't believe the P-values. But the exponential can just be fitted by calculating the mean directly. The gamma is trickier to fit, and maximum likelihood and various short-cut approximations might give slightly different results. $\endgroup$ – Nick Cox May 21 '13 at 17:38
1
$\begingroup$

I know this is not really a direct answer to your question, but I was looking for a qq-plot function for the exponential distribution in R and found this. It may be helpful for other people browsing this page with the goal to find a qq-plot function for exponential distribution.

"qqexp" <-  function(y, line=FALSE, ...) { 
    y <- y[!is.na(y)]
    n <- length(y)
    x <- qexp(c(1:n)/(n+1))
    m <- mean(y)
    if (any(range(y)<0)) stop("Data contain negative values")
    ylim <- c(0,max(y))
    qqplot(x, y, xlab="Exponential plotting position",ylim=ylim,ylab="Ordered sample", ...)
    if (line) abline(0,m,lty=2)
    invisible()
  }
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.