8
$\begingroup$

Suppose I do an experiment $N$ times and get a vector $X$ of results. Let $C_X(y)$ be the empirical cumulative distribution function of $X$. Suppose $X$ is sorted so that $x_1 \leq x_2 \cdots \leq x_N$. Approximately, $$C_X(y)=0\textrm{ if }y \leq x \textrm{ for all }x \in X$$ $$C_X(y)=1\textrm{ if }y > x \textrm{ for all }x \in X$$ $$C_X(y)=\frac{i+\frac{y-x_i}{x_{i+1}-x_i}}{N} \textrm{ if } x_i \leq y \leq x_{i+1} $$

Question: What is the most efficient way to compute the corresponding empirical PDF of $X$? Just interpolate through the histogram?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ What is preventing you from taking the (piecewise) derivative of $C_X(y)$, which immediately gives you the PDF? $\endgroup$
    – Alex R.
    Commented Oct 19, 2017 at 20:15
  • 2
    $\begingroup$ What do you mean by "efficient"? Are you talking about continuous or discrete data? Histogram itself is a density estimation method, kernel density estimation is other commonly used method. $\endgroup$
    – Tim
    Commented Oct 19, 2017 at 20:40
  • $\begingroup$ I am estimating a continuous process. "Efficient" loosely meaning pretty fast way to get pretty good approximation. The above stated ECDF method (sorting the values and finding the index) sounds pretty good and pretty fast, for ECDF. Maybe histogram is pretty good and pretty fast for EPDF, but it requires you to choose bins and hence lose accuracy. Piecewise derivatives sounds simple, accurate and fast. Kernel density estimation sounds slower and trickier. $\endgroup$
    – Lars Ericson
    Commented Oct 19, 2017 at 20:56
  • $\begingroup$ I am looking at a small number of points, for example change in FX price from one minute to the next for 40 minutes. I get a reasonable looking but lumpy ECDF. If I interpolate that and then do numerical derivative, I get a sawtooth wave function. So now I'm leaning towards kernel density estimate, because I want something smoother (if more inaccurate). I.e. I am doing:X=X+[i/100000000. for i in range(X)] cdf=numpy.array(sorted([(X[i],i/40.) for i in range(len(X))])) (x,y)=(cdf[:,0],cdf[:,1]) f = InterpolatedUnivariateSpline(x,y, k=1) dfdx = f.derivative() dydx = dfdx(x) $\endgroup$
    – Lars Ericson
    Commented Oct 19, 2017 at 22:34

2 Answers 2

Reset to default
3
$\begingroup$

One of two things:

1) make fixed histogram bucket sizes and then count the number of points you get that occur in each bucket. In other words, break up the range of $x$ into n equal intervals, and then the count for each interval is the number of times your CDF has a 'step' up in that interval, for each interval. Caveat: you will need to normalize, when done, so that all buckets add to 100% probability.

2) Just take the differences between each pair of CDF points (thus the change in height between them), divide by $\delta x_i$ to get the slope of the CDF at that point along the $x$ axis, and use lines of those slopes to connect the points of a PDF plot. Essentially, you are taking and using the numerical approximation to the derivative to the CDF, which is the PDF. Warning: you will need to think through very carefully if how you do this does not, accidentally, shift the distribution up or down by something like $\delta x_i/2$ at each point. In other words, centering each segment will be important to get right.

If you have a good number of points, method 1 will be a lot less error-prone - e.g., with 1000 points you can probably get a good discrete histogram representation to something like a normal distribution with 20-50 buckets which you can do numerical statistics on easily (mean, moments). Since that is usually what you want, it does the job.

I sense your desire to do something that looks more like a continuous function, which method 2 would get, but I would warn you away from that, unless you have a small number of data points. You will find that: (1) it is going to be hard to represent somehow (i.e., on a spreadsheet or as a data structure); (2) it will be hard to work with even a good representation, and (3) it will take a lot of thought to get right.

I do a lot of numerical methods with unknown distributions and method one is surprisingly accurate most of the time (again, with enough points).

Improve this answer
$\endgroup$
3
  • $\begingroup$ (2) is interesting but begs the question of how to deal with the endpoints. Someone following the instructions you give will not produce a density that integrates to unity. (This is obvious when $N=1$, where your construction gives the zero function, but the problem persists for all $N$.) $\endgroup$
    – whuber ♦
    Commented Oct 19, 2017 at 21:59
  • 1
    $\begingroup$ Good catch. I accidentally deleted a line that said "you need to still make sure the Area Under the Curve is 1.0 with method 2". But you are dead on. I have built some pretty sophisticated numerical methods for working with distributions lately, and the devil is in all those details. $\endgroup$
    – eSurfsnake
    Commented Oct 19, 2017 at 22:18
  • $\begingroup$ How would you do this for a multivariate distribution? $\endgroup$
    – Sia Rezaei
    Commented May 5, 2021 at 23:50
6
$\begingroup$

The empirical PDF of a random sample is a discrete probability distribution which assigns probability mass $1/N$ to each observation if there are no ties, 2 if there are 2 tied observations, 3 and so on.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.