4
$\begingroup$

I have 500,000 values for a variable derived from financial markets. This variable has a arbitrary distribution. I need a formula that will allow me to select a range around any value of this variable such that an equal (or close to it) amount of values fall within that range. From what I understand, this means that I need to convert it from arbitrary distribution to uniform distribution. I have read (but barely understood) that what I am looking for is called "probability integral transform."

Can anyone assist me with some code (Matlab preferred, but it doesn't really matter) to help me accomplish this?

Edit: I uploaded my dataset as a .csv file and compressed .rar file

I used the empirical distribution function within MatLab and got the following plot:empirical cumulative distribution Does this look about right? Here is a histogram of the raw data for reference: histogram of the raw data

$\endgroup$
  • $\begingroup$ Do you have any idea what probability distribution your variable follows? The financial sector has this unfortunate tendency to assume a normal distribution even when it's not appropriate... $\endgroup$ – J. M. is not a statistician May 10 '11 at 9:48
  • $\begingroup$ Cross-posted: (simultaneously) math.stackexchange.com/questions/38221/… $\endgroup$ – cardinal May 10 '11 at 11:55
  • $\begingroup$ @J.M If I use the distribution fitting tool in Matlab I get the best fit with normal distribution. It is less peaked at the center than a normal distribution though. $\endgroup$ – Mike Furlender May 10 '11 at 12:25
  • $\begingroup$ A QQ plot will give you a quick and intuitive sense of how close the underlying data is to being normally distributed. $\endgroup$ – Chris Taylor May 10 '11 at 12:30
  • 1
    $\begingroup$ Mods: If you would like to migrate this to stats.SE we would merge it with the version posted there. Alternatively, we would be happy to migrate our version here, but the question looks more statistic-y than math-y. $\endgroup$ – whuber May 10 '11 at 13:31
7
$\begingroup$

If $X$ has the (cumulative) distribution function $F(x)=P(X<x)$, then $F(X)$ has a uniform distribution on $[0,1]$. You don't know what $F$ is, but with N = 500,000 data points you could simply use the empirical distribution function:

$$\hat{F}(x) = \frac{1}{N} \sum_{i=1}^N 1[x_i\leq x]$$

where $1[A]$ is the indicator function, $1[A]=1$ is $A$ is true and $1[A]=0$ if $A$ is false. The function $F$ is often also called the quantile function.


In coding terms, once you've written your function F you now have two objects, x containing your data and q containing the transformed data, so you could write a function Finv which takes a number in [0,1] and returns the value of your sample distribution at that quantile (using linear interpolation or some other appropriate method for filling in the gaps).

Now if you want to take e.g. 5% of the data either side of the value x0, your range will be Finv(F(x0) - 0.05) to Finv(F(x0) + 0.05).

$\endgroup$
  • $\begingroup$ @Chris Thanks so much! That is extremely helpful. I am going to try to program it in Matlab in a few hours. $\endgroup$ – Mike Furlender May 10 '11 at 12:13
  • $\begingroup$ @Chris Can you please explain the difference between the larger X and the smaller x in F(x)=P(X<x) ? I have a shameful lack of knowledge of mathematical notation. $\endgroup$ – Mike Furlender May 10 '11 at 12:35
  • $\begingroup$ @Mike The capital $X$ is the random variable, then lowercase $x$ is a fixed real number, so $P(X<x)$ means "the probability that the random variable $X$ takes a value less than the real number $x$" $\endgroup$ – Chris Taylor May 10 '11 at 13:46
  • $\begingroup$ Could you please take a look at the pictures I added to my question? Am I on the right track? $\endgroup$ – Mike Furlender May 10 '11 at 15:02
  • $\begingroup$ The pictures look identical to ones I generated in R using your data, so I'd say you're on the right track. $\endgroup$ – Chris Taylor May 10 '11 at 15:25
2
$\begingroup$

Suppose you have a cumulative distribution function $F$ of the variable in question. Suppose the value given is $x$, and the range is $[r_1,r_2]$ with $x\in[r_1,r_2]$. Then if you select the amount of values falling into that range $N$, the following should hold:

$$F(r_1)-F(r_2)=\frac{N}{500 000}$$

This is an equation with 2 unknown variables, so we need some restrictions to solve it. The popular one would be setting $r_1=x-\varepsilon/2$ and $r_2=x+\varepsilon/2$. This would give us the equation of one variable, which could be solved pretty easily, using any optimisation algorithm.

The only thing you need is the function $F$. You can either model it, or use some non-parametric estimate. In the latter case probably optimisation algorithm may be unnecessary, as it should be possible to work out the solution.

Note: This is only one possible approach, depending on data it might not work.

$\endgroup$
  • $\begingroup$ Excellent, thanks for your answer! Am I correct in assuming that I could use Matlab's regression toolbox in estimating F? $\endgroup$ – Mike Furlender May 10 '11 at 12:23
  • $\begingroup$ @Mike, I am not that familiar with Matlab, so I cannot say. Judging from the toolbox name (regression) probably not. Do not hurry to accept this answer as I do not feel that it is a complete one. $\endgroup$ – mpiktas May 10 '11 at 12:27
  • $\begingroup$ @Mike, and please do not cross-post. Though in this case you got lucky, from the answer here and at math.SE, you got something. $\endgroup$ – mpiktas May 10 '11 at 12:31
  • $\begingroup$ Ah, ok I did not realize that was not allowed... probably should have checked. $\endgroup$ – Mike Furlender May 10 '11 at 12:41
  • 2
    $\begingroup$ @Mike, not exactly disallowed, frown upon is the better word. $\endgroup$ – mpiktas May 10 '11 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.