1
$\begingroup$

I have a frequency table with the following CSV data.

df <- read.csv('data.csv')

I convert it to a vector like so:

vec <- rep(df$x, df$frequency)

Then try to fit a distribution to it:

library(fitdistrplus)

plotdist(vec, histo = TRUE, demp = TRUE)

enter image description here

descdist(vec, boot = 100)

enter image description here

fit <- fitdist(vec, "gamma", method="mme")
plot(fit)

enter image description here

Is there another distribution which may suit this dataset better?

I tried a Pareto distribution, but couldn't get a fit: enter image description here

$\endgroup$
  • $\begingroup$ Do you have any additional information as to what the data represents? A few distributions come to mind that have a similar shape, but it might be better to start with what the data actually stands for. $\endgroup$ – Emil Apr 27 '18 at 23:21
  • $\begingroup$ the x values represent the time it takes for devices to first register on a computer network $\endgroup$ – skunkwerk Apr 28 '18 at 0:47
  • 1
    $\begingroup$ The fact that ${X = 0}$ is by far your most frequent outcome should rule out the Gamma distribution, since the support of that is $(0,\infty)$. $\endgroup$ – Emil Apr 28 '18 at 18:26
  • $\begingroup$ Additionally, if the values are supposed to be the time for the event to happen, then from what I saw in your data, $X$ only takes integer values, meaning your variable is not continuous, in which case definitely not Gamma, as indicated also by the QQ-plot (for larger values, there is significant deviation). $\endgroup$ – Emil Apr 28 '18 at 18:58
2
$\begingroup$

After some digging around, I would say there are indications that your data might follow what's called a power law, but I wouldn't sign off on it without additional inspection. The two approaches I used that lead me to suspect this particular distribution are a Pareto QQ-plot and a log-log plot.

  1. As written in the link, when comparing the logarithm of the sample data $X$ to the quantiles of an Exponential distribution with mean equal to 1, $Exp(1)$, then if the points "asymptotically" converge to a straight line, this indicates a power law distribution:

qqplot_full

The catch here is that your most frequent observation is indeed $X=0$, and so the log transform of that brings its own difficulties in seeing the behavior of your data apropos this method. If however we employ this same method in the case where we omit the zero cases from the data, then the picture becomes significantly different:

qqplot_truncated

I would therefore recommend that if you are indeed binning the data into discrete buckets, at least start with $X=1$ as the minimum value.

  1. The other method is the log-log plot, which plots the log of the data vs the log of their observed frequencies:

Log-log

The rule of thumb here is that if, for large values in the x-axis, we start observing a straight line, then this could point to a power law distribution. To reiterate the article in the link above, the catch here is that this particular method requires very large amounts of data to provide reliable results, only works with binned data (which just happens to be the case here), and of course, whether or not it starts resembling a straight line from $log(x) \approx 5.2$ and after is probably open to interpretation.

One further method you can try for identifying the underlying distribution consists of the residual quantile functions. Unfortunately, I have not worked with this type of transformation in the past to be able to provide more information. Hope this helps.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.