Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

i've sampled a real world process, network ping times. The "round-trip-time" is measured in milliseconds. Results are plotted in a histogram:

alt text

Ping times have a minimum value, but a long upper tail.

i want to know what statistical distribution this is, and how to estimate its parameters.

Even though the distribution is not a normal distribution, i can still show what i am trying to achieve.

The normal distribution uses the function:

alt text

with the two parameters

  • μ (mean)
  • σ2  (variance)

Parameter estimation

The formulas for estimating the two parameters are:

alt text

Applying these formulas against the data i have in Excel, i get:

  • μ = 10.9558 (mean)
  • σ2  = 67.4578 (variance)

With these parameters i can plot the "normal" distribution over top my sampled data:

alt text

Obviously it's not a normal distribution. A normal distribution has an infinite top and bottom tail, and is symmetrical. This distribution is not symmetrical.


  • What principles would i apply, what flowchart, would i apply to determine what kind of distribution this is?
  • Given that the distribution has no negative tail, and long positive tail: what distributions match that?
  • Is there a reference that matches distributions to the observations you're taking?

And cutting to the chase, what is the formula for this distribution, and what are the formulas to estimate its parameters?


i want to get the distribution so i can get the "average" value, as well as the "spread": alt text

i am actually plotting the histrogram in software, and i want to overlay the theoretical distribution:

alt text

Note: Cross-posted from math.stackexchange.com


Update: 160,000 samples:

enter image description here

Months and months, and countless sampling sessions, all give the same distribution. There must be a mathematical representation.


Harvey suggested putting the data on a log scale. Here's the probability density on a log scale:

enter image description here

Tags: sampling, statistics, parameter-estimation, normal-distribution


It's not an answer, but an addendum to the question. Here's the distribution buckets. i think the more adventurous person might like to paste them into excel (or whatever program you know) and can discover the distribution.

The values are normalized

Time    Value
53.5    1.86885613545469E-5
54.5    0.00396197500716395
55.5    0.0299702228922418
56.5    0.0506460012708222
57.5    0.0625879919763777
58.5    0.069683415770654
59.5    0.0729476844872482
60.5    0.0508017392821101
61.5    0.032667605247748
62.5    0.025080049337802
63.5    0.0224138145845533
64.5    0.019703973188144
65.5    0.0183895443728742
66.5    0.0172059354870862
67.5    0.0162839664602619
68.5    0.0151688822994406
69.5    0.0142780608748739
70.5    0.0136924859524314
71.5    0.0132751080821798
72.5    0.0121849420031646
73.5    0.0119419907055555
74.5    0.0117114984488494
75.5    0.0105528076448675
76.5    0.0104219877153857
77.5    0.00964952717939773
78.5    0.00879608287754009
79.5    0.00836624596638551
80.5    0.00813575370967943
81.5    0.00760001495084908
82.5    0.00766853967581576
83.5    0.00722624372375815
84.5    0.00692099722163388
85.5    0.00679017729215205
86.5    0.00672788208763689
87.5    0.00667804592402477
88.5    0.00670919352628235
89.5    0.00683378393531266
90.5    0.00612361860383988
91.5    0.00630427469693383
92.5    0.00621706141061261
93.5    0.00596788059255199
94.5    0.00573115881539439
95.5    0.0052950923837883
96.5    0.00490886211579433
97.5    0.00505214108617919
98.5    0.0045413204091549
99.5    0.00467214033863673
100.5   0.00439181191831853
101.5   0.00439804143877004
102.5   0.00432951671380337
103.5   0.00419869678432154
104.5   0.00410525397754881
105.5   0.00440427095922156
106.5   0.00439804143877004
107.5   0.00408656541619426
108.5   0.0040616473343882
109.5   0.00389345028219728
110.5   0.00392459788445485
111.5   0.0038249255572306
112.5   0.00405541781393668
113.5   0.00393705692535789
114.5   0.00391213884355182
115.5   0.00401804069122759
116.5   0.0039432864458094
117.5   0.00365672850503968
118.5   0.00381869603677909
119.5   0.00365672850503968
120.5   0.00340131816652754
121.5   0.00328918679840026
122.5   0.00317082590982146
123.5   0.00344492480968815
124.5   0.00315213734846692
125.5   0.00324558015523965
126.5   0.00277213660092446
127.5   0.00298394029627599
128.5   0.00315213734846692
129.5   0.0030649240621457
130.5   0.00299639933717902
131.5   0.00308984214395176
132.5   0.00300885837808206
133.5   0.00301508789853357
134.5   0.00287803844860023
135.5   0.00277836612137598
136.5   0.00287803844860023
137.5   0.00265377571234566
138.5   0.00267246427370021
139.5   0.0027472185191184
140.5   0.0029465631735669
141.5   0.00247311961925171
142.5   0.00259148050783051
143.5   0.00258525098737899
144.5   0.00259148050783051
145.5   0.0023485292102214
146.5   0.00253541482376687
147.5   0.00226131592390018
148.5   0.00239213585338201
149.5   0.00250426722150929
150.5   0.0026288576305396
151.5   0.00248557866015474
152.5   0.00267869379415173
153.5   0.00247311961925171
154.5   0.00232984064886685
155.5   0.00243574249654262
156.5   0.00242328345563958
157.5   0.00231738160796382
158.5   0.00256656242602444
159.5   0.00221770928073957
160.5   0.00241705393518807
161.5   0.00228000448525473
162.5   0.00236098825112443
163.5   0.00216787311712744
164.5   0.00197475798313046
165.5   0.00203705318764562
166.5   0.00209311887170926
167.5   0.00193115133996985
168.5   0.00177541332868196
169.5   0.00165705244010316
170.5   0.00160098675603952
171.5   0.00154492107197588
172.5   0.0011150841608213
173.5   0.00115869080398191
174.5   0.00107770703811221
175.5   0.000946887108630378
176.5   0.000853444301857643
177.5   0.000822296699600065
178.5   0.00072885389282733
179.5   0.000753771974633393
180.5   0.000766231015536424
181.5   0.000566886361087923
share|improve this question
1  
Just looking at it, it looks like a skewed normal distribution. Are you sure the outliers are necessary for your analysis? –  Brandon Bertelsen Aug 5 '10 at 19:43
1  
My analysis will consist solely of drawing a pretty graph over-top the bars :) But it would be cheating to pretend there was no top tail... –  Ian Boyd Aug 5 '10 at 19:53
    
@Ian: Since your histogram looks like you have 33 bins I would guess a 33rd order polynomial would fit nicely ;) –  Benjamin Bannier Aug 5 '10 at 20:12
    
@honk Please no! Even imaging it hurts. –  mbq Aug 5 '10 at 22:04
    
@Brandon Please no! Even imag(in)ing it hurts. –  walkytalky Aug 5 '10 at 22:07
show 2 more comments

9 Answers

Let me ask a more basic question: what do you want to do with this distributional information?

The reason I ask is because it may well make more sense to approximate the distribution with some sort of kernel density estimator, rather than insist that it fit into one of the (possibly shifted) exponential family distributions. You can answer almost all of the same sorts of questions that a standard distribution will let you answer, and you don't have to worry (as much) about whether you've selected the correct model.

But if there's a fixed minimum time, and you must have some sort of compactly parameterized distribution to go with it, then just eyeballing it I'd subtract off the minimum and fit a gamma, like others have suggested.

share|improve this answer
    
i want to draw the theoretical distribution curve over it, find the mean, median, standard deviation. Mostly i'd like the median, but really i want to draw the theoretical curve. –  Ian Boyd Oct 7 '11 at 14:35
1  
I think Rich meant something like, "What kind of question are you trying to answer?" Determining the distribution of the data is a question, yes, but surely there's some other question you expect to be able to answer once you know the distribution. What is it? –  Matt Parker Oct 7 '11 at 15:26
    
@MattParker i want to know what the "peak" time is. –  Ian Boyd Apr 17 '12 at 23:55
2  
@IanBoyd I don't think you really need to know the theoretical distribution to get that - would the median work? And for shading in the adjacent 'standard deviations', you could just use quantiles. If your objective is to communicate how long most pings take, I don't see anything wrong with reporting the median with the 25% and 75% quantiles. Or, if you want to emulate the 68% that falls within 1 SD of the mean in the normal distribution, the 16% and 84% quantiles. –  Matt Parker Apr 18 '12 at 15:47
add comment

Weibull is sometimes used for modelling ping time. try a weibull distribution. To fit one in R:

x<-rweibull(n=1000,shape=2,scale=100)
#generate a weibull (this should be your data).
hist(x)
#this is an histogram of your data.
library(survival)
a1<-survreg(Surv(x,rep(1,1000))~1,dist='weibull')
exp(a1$coef) #this is the ML estimate of the scale parameter
1/a1$scale     #this is the ML estimate of the shape parameter

If you're wondering for the goofy names (i.e. $scale to get the inverse of the shape) it's because "survreg" uses another parametrization (i.e. it is parametrized in terms of the "inverse weibull" which is more comon in actuarial sciences).

share|improve this answer
add comment

There is no reason to expect that any real world data set will fit a known distributional form...especially from such a known messy data source.

What you want to do with the answers will largely indicate an approach. For example, if you want to know when the ping times have changed significantly, then trending the empirical distribution may be a way to go. If you want to identify outliers, other techniques may be more appropriate.

share|improve this answer
    
Really i want to draw the mathematical curve that follows the distribution. Granted it might not be a known distribution; but i can't imagine that this hasn't been investigated before. –  Ian Boyd Aug 5 '10 at 19:57
2  
Look up 'density estimation'. –  PeterR Aug 6 '10 at 11:21
add comment

A simpler approach might be to transform the data. After transforming, it might be close to Gaussian.

One common way to do so is by taking the logarithm of all values.

My guess is that in this case the distribution of the reciprocal of the round-trip times will be more symmetrical and perhaps close to Gaussian. By taking the reciprocal, you are essentially tabulating velocities instead of times, so it still is easy to interpret the results (unlike logarithms or many transforms).

share|improve this answer
add comment

Try the gamma distribution which is parametrized as $x \sim Gamma(k,\theta)$. If you see these pdf plots of the gamma from the wiki you will see that there are some plots that look similar to what you have.

Update- Estimation Process

The estimation via maximum likelihood is tricky but possible. I imagine you can start with the approximate solution given by the wiki for $\hat{\theta}$ and $\hat{k}$ and if the plots look ok and if needed you can estimate $\hat{k}$ more accurately using the details in the wiki.

share|improve this answer
    
i was actually going to efforts to avoid mentioning the Gamma distribution. i saw it on Wikipedia, i cannot actually find the formula for the distribution, or the formulas to estimate the parameters in that formula. And then i got really nervous when i saw "There is no closed-form solution for k." And i tried it anyway with some formulas - but when you get a packet that comes back in 0ms, the ln(0) blows up. –  Ian Boyd Aug 5 '10 at 19:40
    
Because while i have good understanding of the normal distribution, from my university days, i am over my head when we get to things like "Kullback–Leibler divergence". –  Ian Boyd Aug 5 '10 at 19:42
    
If you have a packet that comes back in 0ms it is not 'really' zero, right? Perhaps, you can set it to a small value and in any case you need not worry about ln(0) as the parameter $k$ does not refer to the data you have. I will update my answer with some details about the estimation process. –  user28 Aug 5 '10 at 19:45
    
Yes, technically it should be referred to as <1ms. And this plot doesn't include zero, because it's going over a higher latency link (modem). But i can run the program just as well over a faster link (i.e. ping another machine on the LAN), and routinely get <1ms and 1ms, with much less occurrences of 2ms. Unfortunately Windows only provides resolution of 1ms. i could manually time it using a high-performance counter, getting µs; but i was still hoping to be able to put them into buckets (to save memory). Perhaps i should add 1ms to everything... 1ms ==> (0..1] –  Ian Boyd Aug 5 '10 at 19:55
    
simply fitting gammas with R: docs.google.com/… –  apeescape Aug 6 '10 at 20:15
add comment

Another approach, that is more justified by network considerations, is to try to fit a sum of independent exponentials with different parameters. A reasonable assumption would be that each node in the path of the ping the delay would be an independent exponential, with different parameters. A reference to the distributional form of the sum of independent exponentials with differing parameters is http://www.math.bme.hu/~balazs/sumexp.pdf.

You should probably also look at the ping times vs the number of hops.

share|improve this answer
add comment

Looking at it I would say a skew-normal or possibly a binormal distribution may fit it well.

In R you could use the sn library to deal with skew-normal distribution and use nls or mle to do a non-linear least square or a maximum likelihood extimation fit of your data.

===

EDIT: rereading your question/comments I would add something more

If what you're interested into is just drawing a pretty graph over the bars forget about distributions, who cares in the end if you're not doing anything with it. Just draw a B-spline over your data point and you're good.

Also, with this approach you avoid having to implement a MLE fit algorithm (or similar), and you're covered in the case of a distribution that is not skew-normal (or whatever you choose to draw)

share|improve this answer
    
+1 i thought binomial as well, when i first saw the histogram. (Not sure why this got downvoted). –  doug Aug 6 '10 at 6:35
    
Well its not just pretty lines. i'd also like to be able to point to the true "peak" in the curve, as well as shade in one standard deviation unit on either side. –  Ian Boyd Oct 7 '11 at 14:41
    
@IanBoyd: A B-spline will suffice for those things, especially as the histogram is relatively low-res. Then, it always depends on what you mean by "true" peak. –  nico Oct 7 '11 at 15:37
add comment

Based on your comment "Really i want to draw the mathematical curve that follows the distribution. Granted it might not be a known distribution; but i can't imagine that this hasn't been investigated before." I am providing a function that sort of fits.

Take a look at ExtremeValueDistribution

I added an amplitude and made the two betas different. I figure your function's center is closer to 9.5 then 10.

New function: a E^(-E^(((-x + alpha)/b1)) + (-x + alpha)/b2)/((b1 + b2)/2)

{alpha->9.5, b2 -> 0.899093, a -> 5822.2, b1 -> 0.381825}

Wolfram alpha: plot 11193.8 E^(-E^(1.66667 (10 - x)) + 1.66667 (10 - x)) ,x 0..16, y from 0 to 4500

Some points around 10ms:
{{9, 390.254}, {10, 3979.59}, {11, 1680.73}, {12, 562.838}}

Tail does not fit perfectly though. The tail can be fit better if b2 is lower and the peak is chosen to be closer to 9.

share|improve this answer
add comment

The distribution looks log-normal to me.

You can fit your data using two parameters: scale and location. These can be fitted in much the same way as a normal distribution using expectation maximisation.

http://en.wikipedia.org/wiki/Log-normal_distribution

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.