1
$\begingroup$

I am using Matlab to try and find a good fit for this curve: cumulative distribution None of the built-in formulas seem to work well.

Any suggestions?

$\endgroup$
  • $\begingroup$ This appears really to be a question about fitting a distribution, not a regression question: see stats.stackexchange.com/questions/10594/… . $\endgroup$ – whuber May 10 '11 at 20:23
  • $\begingroup$ Well, I need to be able to get F(x) given x through a direct formula (without summation or anything fancy) so isn't that regression? $\endgroup$ – Mike Furlender May 10 '11 at 20:38
  • $\begingroup$ Yes, but this is an extremely specialized application: the curve has to go from 0 to 1 monotonically; the original data are extremely highly correlated; and the meaning of "good" depends on how you are going to use the fit. (E.g., you might be better off fitting the derivative of this curve rather than the fit itself, or you might need to get a much better fit near $F(x)= 1$ than elsewhere: have you read Taleb's Black Swan tirade?) If you don't disclose all those things, even the best experts could unknowingly give you formulas that perform poorly. $\endgroup$ – whuber May 10 '11 at 21:10
  • $\begingroup$ @whuber: I did tell Mike @ m.SE that he needed to have some idea about how things are distributed in his application, and that assuming things are "normal" doesn't always work... oh well. $\endgroup$ – J. M. is not a statistician May 12 '11 at 11:16
2
$\begingroup$

Looking at the charts in your first question, this looks slightly like the absolute value of a standard normal distribution, what Wikipedia calls a half-normal distribution, and your curve here looks like the top half of the cdf of a normal distribution.

One way to check is to look at various quantiles of your distribution, compared with those of the absolute value of a standard normal. In R, using your data, this might look like

> mydat <- read.csv("http://furlender.com/data.csv", header=FALSE)
> mean(mydat$V1)   # compare with sqrt(2/pi) = 0.7978846 for half normal
[1] 1.014772
> median(mydat$V1) #              compare with 0.6744898 for half-normal
[1] 0.9162514
> percents <- c(0.01, (1:19)/20, 0.99, 0.999, 0.9999, 0.99999, 0.999999)
> quantile(mydat$V1, percents) / qnorm(percents/2 + 0.5)
      1%       5%      10%      15%      20%      25%      30%      35% 
1.479777 1.455286 1.443074 1.435248 1.429568 1.421616 1.409148 1.400204 
     40%      45%      50%      55%      60%      65%      70%      75% 
1.386696 1.372990 1.358436 1.341848 1.323763 1.306209 1.287204 1.266550 
     80%      85%      90%      95%      99%    99.9%   99.99%  99.999% 
1.245071 1.223053 1.198247 1.172505 1.148309 1.141927 1.220474 1.412236 
99.9999% 
1.371697 

and this suggests that your distribution is slightly more dispersed than the absolute value of a standard normal, perhaps somewhere between 1.15 and 1.4 times. Even the tails fit this description. Drawing the ecdf confirms this compared with scaled normals.

> plot.ecdf(mydat$V1, lwd=3)
> curve(2*pnorm(x/1.4 )-1, 0, 7, col="red"  , add=TRUE)
> curve(2*pnorm(x/1.15)-1, 0, 7, col="green", add=TRUE)

ecdf

If you wish, you could stop there, and choose some intermediate value (perhaps 1.2718 to give a similar mean). Or you could look for some normal-like symmetric distribution which has even moments about 0 similar to your data, and then take the absolute value. These moments are easily calculated: for example the second and fourth are:

> mean(mydat$V1^2) 
[1] 1.516602
> mean(mydat$V1^4) 
[1] 5.972706

suggesting that (before taking the absolute value) the symmetric distribution would be slightly platykurtic even though the extreme tail is normal-like.

$\endgroup$
  • $\begingroup$ +1 for the really nice idea of checking against quantiles. A good way to fit tails is to fit the 1/4, 1/8, 1/16, ... quantiles. Note that this is not really half-normal: the mode is slightly positive. Also, it's short tailed compared to the normal. You can correct that with a Box-Cox transformation or something similar and get a much closer fit overall. $\endgroup$ – whuber May 11 '11 at 0:27
1
$\begingroup$

Check out Eureqa for a neat evolutionary approach to finding the mathematical form of an otherwise ambiguous function. It's native to Windows but works fine on Linux & Mac via Wine (in which case I'd suggest you use winebottler).

$\endgroup$
  • 1
    $\begingroup$ This is one case where Eureqa will likely fail unless some clever pre-processing is done. The problem is that CDFs like this typically have some error function-like tails and Eureqa has no function sufficiently like Erf. If you first smooth this "curve" (it's really the EDF of 400K+ points) and ask Eureqa to fit its derivative in an appropriate manner it might do a good job. (I suggest a Box-Cox transformed normal pdf for starters.) That presumes the OP needs a good fit to the PDF itself, of course. $\endgroup$ – whuber May 10 '11 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.