I know that there is a way to "redistribute the frequencies of a variable" as stated here:

Slide number 14 and 15 about redistribution

and here:

Dorian Pyle book chapter 7 section 2 paragraph 3 (7.2.3):

The easiest way to adjust distribution density is simply to displace the high-density points into the low-density areas until all points are at the mean density for the variable. Such a process ends up with a rectangular distribution.

Illustrating this passage are figures like these:

Figures 7.7 and 7.8

In both the references the problem is clear and solved but does not provide sufficient information for writing an algorithm and in particular does not say how to calculate the displacement (the way to move the frequencies on the left or right of the center of the distribution).

Please can you help me to understand it ??

  • 2
    $\begingroup$ The problem is not with you: I'm afraid those slides don't say anything at all. They are too vague to give anyone a definite idea about what they are describing. Try to contact their author for clarification. Better yet, tell us about the actual task you need to perform and where you're stuck. $\endgroup$ – whuber Sep 10 '12 at 15:46
  • $\begingroup$ the probelm is stated in the book: "Data preparation For data mining" Dorian Pyle page 261 but i can't understand how write the algorithm... the question was how do i calculate the displacement??.. if you read the book you can understand that it is not just my immagination... $\endgroup$ – T-student Sep 12 '12 at 14:14
  • $\begingroup$ temida.si/~bojan/MPS/materials/… chapter 7 ... exactly it is in 7(chatpter).2.3 paragraph $\endgroup$ – T-student Sep 12 '12 at 14:16

This is an empirical version of the "probability integral transform."

Suppose you have $n$ data values. Sorting them in ascending order, index them as

$$x_1 \le x_2 \le \cdots \le x_n.$$

Replacing each $x_i$ by $i$ gives as uniform a distribution as possible. That's all there is to it.

If you would like the replacement values to lie within a given range, such as the range $[0,1]$ in the illustration, then rescale the $i$. In general, if the range is to be $[a,b]$, then rescale $i$ to $a + (b-a)(i-1/2)/n$. To see that this works, note that the rescaling is linear in $i$ and by sending $i=1$ to $a + (b-a)/(2n)$ and $i=n$ to $b - (b-a)/(2n)$ it places the extreme values of $x$ at small, equal distances just inside the interval. Alternatively, rescale $i$ to $a + (b-a)(i-1)/(n-1)$ to place the extrema exactly at $a$ and $b$.

Example (in R)

Generate and plot some sample data.

n <- 100
x <- 10*rbeta(n, 1, 3)
hist(x, col="Gray")

Histogram of X

Compute the "displacement" (as the difference between $x$ and the rescaled index $i$, given here as $y$) and plot its graph:

x <- sort(x)
x.min <- min(x)
x.max <- max(x)
y <- x.min + (x.max - x.min) * ((1:n)- 0.5) / n
displacement <- y - x
plot(x, displacement, type="l", lwd=2, main="'Displacement Graph'")

Displacement graph

Finally, confirm the effect by plotting a histogram of the "displaced" values of $x$:

n.breaks <- 10
breaks <- (0:n.breaks) * (x.max - x.min)/n.breaks + x.min
hist(x + displacement, breaks=breaks, col="Gray")

Uniform histogram

| cite | improve this answer | |
  • $\begingroup$ if i could give you 10 votes i gave ... $\endgroup$ – T-student Sep 14 '12 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.