3
$\begingroup$

I have generated a dataset (available here) for which I try to find out the best fitting probability distribution.

I first generated uniformly distributed random directions and then calculated the angle between two succeeding directions. The dataset are these angles. You can interpret the random directions as random points sampled on the surface of the unit sphere.

The dataset of random directions is available here: random_dirs.txt

I generated the random directions through polar coordinates $\theta = 2*\pi*u$, $\phi=acos(2v-1)$ (where $u,v$ are uniformly distributed from (0,1)); this method is also listed on this site.

I then generate normalized direction vectors $\overrightarrow{n_i}=\left(\begin{array}{c} \sin(\phi)cos(\theta)\\ \sin(\phi)\\ \cos(\phi)) \end{array} \right)$

The angle between two suceeding directions is then calculated using

random_dirs = load('random_dirs.txt'); 
rand_dirs_1 = random_dirs(1:end-1,:);
rand_dirs_2= random_dirs(2:end,:);
angles = acos(dot(rand_dirs_1, rand_dirs_2,2));

I tried the normal distribution:

Density (PDF) pdf

Cumulative Probability (CDF)

enter image description here

The CDF matches pretty good, but the PDF could be fitting "better".

Is the data distributed according the normal distribution? How do I verify that? or is there a better matching distribution?

The reason I need to identify a distribution is that I want to compare with other datasets using mean and standard deviation, so I guess first finding out a matching distribution which calculates these measures is necessary.

e.g. this is the same original data, but I applied a spatial sorting algorithm to the random directions and then calculated the angles between neighbouring directions. The angles between the sorted directions are available here. The sorting uses a space-filling curve (e.g. Hilbert curve) to map the 3D data into 1D space and sort the directions accordingly.

enter image description here

enter image description here

Now I want to compare how much "improvement" in terms of mean and sigma the sorting introduced.

Or are there other measures I could use to express the "improvement"?


How many bins should I use for fitting the data? I used 61 according to "Scott's method" (Scott, David W. (1979). "On optimal and data-based histograms".).

According to "Sturges' method" (Sturges, H. A. (1926). "The choice of a class interval") I should use 18 bins which changes the appearance of the data quite a bit, it looks more like a normal distribution:

normal fit with 18 bins

$\endgroup$
  • 1
    $\begingroup$ Why do you need to identify a distribution? $\endgroup$ – Glen_b Aug 17 '15 at 10:25
  • $\begingroup$ How did you generate it? $\endgroup$ – mark999 Aug 17 '15 at 10:27
  • $\begingroup$ @mark999 I updated my question with a description of the generation $\endgroup$ – m.s. Aug 17 '15 at 10:30
  • $\begingroup$ @Glen_b I want to compare other datasets using mean and standard deviation, so I guess first finding out a matching distribution which calculates these measures is necessary $\endgroup$ – m.s. Aug 17 '15 at 10:32
  • 2
    $\begingroup$ If you're interested in improvement in mean and standard deviation (as you say in the question) ... you could just directly measure the change in mean and in standard deviation. $\endgroup$ – Glen_b Aug 17 '15 at 17:07
4
$\begingroup$

Edit: It now looks like these are random directions on the sphere, in which case the pdf of the distance between random vectors should be $\frac12\sin(\theta);\quad 0<\theta<\pi$. The remainder of this answer is not relevant in that case; I'll probably delete some of it eventually.

I'll need to come back to look at the ordered case to see if I can get anywhere with that.


Looking at your data, I'm a little concerned by some lack-of-fit in your data to the theoretical density:

![enter image description here

Given your large sample size, this suggests you may not actually be getting quite a random distribution for some reason. There's slightly too many angles around $\pi/2$ and slightly too few in the region around $\pi/3$ and $2\pi/3$

While the deviation is small, it's large enough to lead to rejection by a Kolmogorov-Smirnov test ($p$ about $1.35\times 10^{-5}$). I'm not sure why that's happening but your simulated s.d. will be slightly too small as well.


(my old answer)

Is the data distributed according the normal distribution?

Your data are not drawn from a normal distribution.

How do I verify that?

You can't. You can sometimes rule it out, as here, but you can't conclude that the data were drawn from a normal distribution on the basis of the appearance of the data.

or is there a better matching distribution?

Oh, an infinite number of them. If you would seek a simple model for data like that, a scaled (i.e. to be on $(0,\pi)$) beta distribution might be adequate, but if you know something about what the values consist of it's likely you can come up with better options.

I try to find out the best fitting probability distribution.

"Best" depends on your criteria. What are we to try to do best at? (e.g. what are you using this to do?)

$\endgroup$
  • $\begingroup$ I think it more like a beta distribution $\endgroup$ – Deep North Aug 17 '15 at 12:39
  • $\begingroup$ how do you know the PDF of the random directions is $\frac12\sin(\theta);\quad 0<\theta<\pi$? can you point me to any source please? I did not think of a sine/cosine distribution, but now that you mention it: should I go with the Raised cosine distribution? $\endgroup$ – m.s. Aug 17 '15 at 13:20
  • 2
    $\begingroup$ It's pretty straightforward. It follows immediately from the distribution of polar angles here (you can just call one of the vectors the pole); there's also a derivation here for example. $\endgroup$ – Glen_b Aug 17 '15 at 13:25
  • $\begingroup$ thanks for the links, they really cleared things up. which additional explanation do you need? I will gladly update my question with that necessary information. The sorting uses a space-filling curve (e.g. Hilbert curve) to map the 3D data into 1D space and sort the directions accordingly. $\endgroup$ – m.s. Aug 17 '15 at 13:54
  • $\begingroup$ I added this detail also to my question; what do you mean by "simulate the other one"? $\endgroup$ – m.s. Aug 17 '15 at 17:20
0
$\begingroup$

For a one-dimensional distribution of angles, you might look into the von Mises distribution. One thing the von Mises distribution does is account for the equivalence between the endpoints of your domain, e.g. an angle of 0 is the same as an angle of $2\pi$.

There are other 1-dimensional distributions that have this quality - for example you could "wrap" the density of pretty much any continuous distribution onto the range 0 to $2\pi$ - see "wrapped Normal distribution", or wrapped t, or wrapped Cauchy.

$\endgroup$
  • 2
    $\begingroup$ Being angles between 3D vectors, these angles are on the half-circle. $\endgroup$ – Glen_b Aug 17 '15 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.