How can I determine the best fit normal distribution from this information?

Question

I'm trying to take a normal distribution of points, and force them to become a uniform distribution. I've had little success on S.O., so I thought I'd ask here.

Basically, I have a hash function which takes an X, Y, and seed value and generates numbers in a normal distribution with a mean of 0. These numbers vary only slightly from the numbers on all sides of them. When mapped out with the X and Y and the hashed value being Z, it creates a terrain map. The map has some large peaks far past 1 and -1, but the middle 50% of the values lie between (-0.4,0.4). I'm trying to smooth this map out so that it retains its shape, but has a more-or-less uniform (rather than normal) distribution.

As I said, the middle 50% of the values lie between (-0.4,0.4). The theoretical bounds for the hash function is (-2.25,2.25), though after generating a billion samples, the range of numbers found were about (-1.75,1.75).

I think I need to take the above information to determine the best fit normal distribution, then use it to transform each value. As I found out on S.O., it's a really tough problem for me to explain. Hopefully someone here can at least point me in the right direction explaining it, or understands what I'm trying to do.

---

The probability densities of my distribution, in blue, and Normal(0,.72), in red: [-3.8,3.8]

---

Their cumulative probabilities: [-3.8,3.8]

---

The probability density after inverse probability transform: [0,1]

---

And the cumulative probability compared to Uniform: [0,1]

Answer 1

Interpreting your question as asking for the "inverse probability transform," as ars has indicated and you have confirmed in comments to the question, gives a simple solution: perform a sort to rank all the $z$ values in ascending order from $1$ to $N$ (the amount of data), then convert each $z$ into $2 Rank(z)/(N+1) - 1$. If you want to be really careful, scan over the ranked $z$'s to look for ties and assign each value within a group of tied values the average of their ranks.

Answer 2

The inverse probability transform seems to be what you want.

To get a better uniform fit, the CDF $F(x)$ in the inverse probabiliy transform needs to be the actual distribution generating the data. If this isn't actually normal, then that could account for the relatively poor fit. You might try a Q-Q plot or normality test first. Or your data may very well be normal, just with some noise, in which case you can obtain a best fit. I would suggest trying in order:

1. Standardize your data to N(0, 1) and see if the transformation improves.
2. After a normality test, fit a normal distribution to your data, then apply the transformation.

Answer 3

OK, your hash function is generating numbers according to some distribution. You're thinking of it as normal, but it's really not.

There's a simple way to convert between any univariate distribution and a uniform distribution. That is to use the cumulative distribution function CDF, which is a simple monotonic function, S-shaped, going from y = 0 to y = 1.

So to convert a number x from your hash distribution into a uniform number y, simply take y = CDF(x). To convert from uniform y to hash number x, just invert the CDF function.

To get the CDF function, just make a table lookup. Generate a large number N of your hash function numbers, put them in a big array, and sort them in ascending order. That is your table. Then to calculate CDF(x), just lookup x in the table, by binary search. Then its index i in the table tells what y is, by y = i/N. (Actually, I'm cheating a bit. You will be more accurate if you do interpolation between two adjacent entries.)

If you want to invert the CDF function, just take your uniform number y, and get i = N*y. That gives you an index in the table, where you find x. Of course, you should interpolate, but if N is big enough, and you're not really fussy about accuracy, you don't really need to.

P.S. I'm glossing over some details, like what to do at the very ends of the table, or what if your table contains duplicate values, but this should get you going.

Answer 4

Why does your transformation need to be a smooth deformation? Just take the binary representation of each number in 64-bit two's complement fixed point, or 64-bit IEEE floating point, and throw that into SHA-1. Boom, instant hash with uniform distribution of the outputs (assuming no duplicates).

Answer 5

To recap, you have a function $f(x,y)$ which is continuous, and pseudo-random uniform at integer $(x,y)$, but nearly normal over the plane. You wish to make $f$ pseudo-random uniform over the whole plane. Suppose you could somehow make $f$ parametrized by some parameter $\lambda$, say by using $\theta$ to twiddle the pseudo-random uniform number generated at the lattice points. Then if you took $g(x,y) = \sum_{i} f(x,y,\lambda_i)$ for a whole bunch of $\lambda_i$ values, $g(x,y)$ would look much more normal, via the Central Limit Theorem. You could then invert the normal to get a uniform field. (I was inspired by Mike Dunlavey's comment on adding 12 uniforms to get a normal).