# Converting arbitrary distribution to uniform one

I have 500,000 values for a variable derived from financial markets. This variable has a arbitrary distribution. I need a formula that will allow me to select a range around any value of this variable such that an equal (or close to it) amount of values fall within that range. From what I understand, this means that I need to convert it from arbitrary distribution to uniform distribution. I have read (but barely understood) that what I am looking for is called "probability integral transform."

Can anyone assist me with some code (Matlab preferred, but it doesn't really matter) to help me accomplish this?

Edit: I uploaded my dataset as a .csv file and compressed .rar file

I used the empirical distribution function within MatLab and got the following plot: Does this look about right? Here is a histogram of the raw data for reference:

• Do you have any idea what probability distribution your variable follows? The financial sector has this unfortunate tendency to assume a normal distribution even when it's not appropriate... Commented May 10, 2011 at 9:48
• Cross-posted: (simultaneously) math.stackexchange.com/questions/38221/… Commented May 10, 2011 at 11:55
• @J.M If I use the distribution fitting tool in Matlab I get the best fit with normal distribution. It is less peaked at the center than a normal distribution though. Commented May 10, 2011 at 12:25
• A QQ plot will give you a quick and intuitive sense of how close the underlying data is to being normally distributed. Commented May 10, 2011 at 12:30

If $X$ has the (cumulative) distribution function $F(x)=P(X<x)$, then $F(X)$ has a uniform distribution on $[0,1]$. You don't know what $F$ is, but with N = 500,000 data points you could simply use the empirical distribution function:

$$\hat{F}(x) = \frac{1}{N} \sum_{i=1}^N 1[x_i\leq x]$$

where $1[A]$ is the indicator function, $1[A]=1$ is $A$ is true and $1[A]=0$ if $A$ is false. The function $F$ is often also called the quantile function.

In coding terms, once you've written your function F you now have two objects, x containing your data and q containing the transformed data, so you could write a function Finv which takes a number in [0,1] and returns the value of your sample distribution at that quantile (using linear interpolation or some other appropriate method for filling in the gaps).

Now if you want to take e.g. 5% of the data either side of the value x0, your range will be Finv(F(x0) - 0.05) to Finv(F(x0) + 0.05).

• @Chris Can you please explain the difference between the larger X and the smaller x in F(x)=P(X<x) ? I have a shameful lack of knowledge of mathematical notation. Commented May 10, 2011 at 12:35
• @Mike The capital $X$ is the random variable, then lowercase $x$ is a fixed real number, so $P(X<x)$ means "the probability that the random variable $X$ takes a value less than the real number $x$" Commented May 10, 2011 at 13:46
• Could you please take a look at the pictures I added to my question? Am I on the right track? Commented May 10, 2011 at 15:02
• The pictures look identical to ones I generated in R using your data, so I'd say you're on the right track. Commented May 10, 2011 at 15:25

Suppose you have a cumulative distribution function $F$ of the variable in question. Suppose the value given is $x$, and the range is $[r_1,r_2]$ with $x\in[r_1,r_2]$. Then if you select the amount of values falling into that range $N$, the following should hold:

$$F(r_1)-F(r_2)=\frac{N}{500 000}$$

This is an equation with 2 unknown variables, so we need some restrictions to solve it. The popular one would be setting $r_1=x-\varepsilon/2$ and $r_2=x+\varepsilon/2$. This would give us the equation of one variable, which could be solved pretty easily, using any optimisation algorithm.

The only thing you need is the function $F$. You can either model it, or use some non-parametric estimate. In the latter case probably optimisation algorithm may be unnecessary, as it should be possible to work out the solution.

Note: This is only one possible approach, depending on data it might not work.

• Excellent, thanks for your answer! Am I correct in assuming that I could use Matlab's regression toolbox in estimating F? Commented May 10, 2011 at 12:23
• @Mike, I am not that familiar with Matlab, so I cannot say. Judging from the toolbox name (regression) probably not. Do not hurry to accept this answer as I do not feel that it is a complete one. Commented May 10, 2011 at 12:27
• @Mike, and please do not cross-post. Though in this case you got lucky, from the answer here and at math.SE, you got something. Commented May 10, 2011 at 12:31