32
$\begingroup$

I want to represent a variable as a number between 0 and 1. The variable is a non-negative integer with no inherent bound. I map 0 to 0 but what can I map to 1 or numbers between 0 and 1?

I could use the history of that variable to provide the limits. This would mean I have to restate old statistics if the maximum increases. Do I have to do this or are there other tricks I should know about?

$\endgroup$
9
  • 7
    $\begingroup$ Because any nondecreasing function from $[0, \infty) \to [0, 1]$ will do the trick, you have a lot of flexibility. But some methods will be better than others, depending on the application. What is your purpose in seeking such a re-expression? $\endgroup$ – whuber Mar 14 '11 at 21:12
  • 1
    $\begingroup$ I am measuring content across many different dimensions and I want to be able to make comparisons in terms of how relevant a given piece of content is. Additionally, I want to display values across these dimensions that is explicable and easily understood. $\endgroup$ – Spencer Mar 14 '11 at 21:14
  • 1
    $\begingroup$ @Spencer Exactly how are you measuring content and "relevance"? E.g., on arbitrary scales, as counts, proportions, frequencies of views, correlations with other content, etc. etc. Different types of measurements benefit from different kinds of re-expressions. $\endgroup$ – whuber Mar 14 '11 at 21:29
  • 1
    $\begingroup$ I am measuring them on arbitrary scales. How old the content is. How many "points" a piece of content is received. Self-reported "interest" in the domain of the content. $\endgroup$ – Spencer Mar 14 '11 at 21:38
  • 2
    $\begingroup$ One of the simplest transformations you could use is to convert your data into quantile scores. $\endgroup$ – charles.y.zheng Mar 15 '11 at 1:28
37
$\begingroup$

A very common trick to do so (e.g., in connectionist modeling) is to use the hyperbolic tangent tanh as the 'squashing function". It automatically fits all numbers into the interval between -1 and 1. Which in your case restricts the range from 0 to 1. In r and matlab you get it via tanh().

Another squashing function is the logistic function (thanks to Simon for the name), provided by $ f(x) = 1 / (1 + e ^{-x} ) $, which restricts the range from 0 to 1 (with 0 mapped to .5). So you would have to multiply the result by 2 and subtract 1 to fit your data into the interval between 0 and 1.

Here is some simple R code which plots both functions (tanh in red, logistic in blue) so you can see how both squash:

x <- seq(0,20,0.001)
plot(x,tanh(x),pch=".", col="red", ylab="y")
points(x,(1 / (1 + exp(-x)))*2-1, pch=".",col="blue")
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. That solves the problem of bounding. For my data it goes to 1 very quickly for my data so I guess the next thing I need to do is to scale this information to concentrate on the interesting range which I could do based on the history of it without fear of leaving the bound, just hitting the limit. $\endgroup$ – Russell Gallop Aug 2 '10 at 15:19
28
$\begingroup$

As often, my first question was going to be "why do you want to do this", then I saw you've already answered this in the comments to the question: "I am measuring content across many different dimensions and I want to be able to make comparisons in terms of how relevant a given piece of content is. Additionally, I want to display values across these dimensions that is explicable and easily understood."

There is no reason to normalize the data so that the max is 1 and the min is zero in order to achieve this, and my opinion is that this would be a bad idea in general. The max or min values could very easily be outliers that are unrepresentative of the population distribution. @osknows parting remark about using $z$-scores is a much better idea. $z$-scores (aka standard scores) normalize each variable using its standard deviation rather than its range. The standard deviation is less influenced by outliers. In order to use $z$-scores, it's preferable that each variable has a roughly normal distribution, or at least has a roughly symmetric distribution (i.e. isn't severely skew) but if necessary you can apply some appropriate data transformation first in order to achieve this; which transformation to use could be determined by finding the best fitting Box–Cox transformation.

$\endgroup$
2
  • $\begingroup$ so there is nothing to do with forcing the data to the $[0, 1]$, and in general I do agree with standardization if the author really looks for it :) I answered before the discussion appeared, so probably delete my answer if this one is correct :) $\endgroup$ – Dmitrij Celov Mar 15 '11 at 10:27
  • 1
    $\begingroup$ If one is worried about outliers, one might consider using the median absolute deviation (from the median) instead of the standard deviation. In R, use the mad() function. And if one is worried about skewness, one can use the ranks of the data instead of the original observations. In R, this would be rank(), but if one’s going to use this on new data, ecdf() would be a better alternative (ecdf(x) returns a new function that basically gives the value $p$ to the $p$-quantile of $x$, e.g., 0 (really $1/n$) to the lowest value of $x$, $1$ to the highest value, 0.5 to the median etc.) $\endgroup$ – Karl Ove Hufthammer Apr 1 '15 at 9:32
11
$\begingroup$

Any sigmoid function will work:

$\endgroup$
3
  • $\begingroup$ erf is not a very handy function, provided you don't want to rather use it for its derivative. $\endgroup$ – user88 Aug 2 '10 at 15:26
  • $\begingroup$ I ended up using a simple logistic function with some small tweaks: (1 / (1 + java.lang.Math.exp(-1 * (factor * i))) - 0.5) * 2. I picked a factor of 0.05 which seems to work well for i between 0 and a few hundred. $\endgroup$ – Jilles van Gurp Sep 22 '15 at 12:08
  • $\begingroup$ 1.0/(1.0+exp(-1.69897*(x-mean(x))/sd(x))) is a close approximation to pnorm $\endgroup$ – Chris Aug 24 '16 at 9:51
4
$\begingroup$

In addition to the good suggestions by Henrik and Simon Byrne, you could use f(x) = x/(x+1). By way of comparison, the logistic function will exaggerate differences as x grows larger. That is, the difference between f(x) and f(x+1) will be larger with the logistic function than with f(x) = x/(x+1). You may or may not want that effect.

$\endgroup$
2
$\begingroup$

Another customisable approach that you can explore is to simply divide all values by the maximum value and take it to the power of a positive shape value ($\gamma$) that best satisfies your desired tranformation objectives. See example below in R in which the dashed line is the simple case of dividing x by max(x):

scaled_power_transform <- function(x, gamma=0.25)
{
    ## x must be nonnegative
    stopifnot(all(x >= 0))
    ## scale to [0, 1]
    x_scaled <- x / max(x)
    ## customise the shape
    x_scaled <- x_scaled^gamma
    return(invisible(x_scaled))
}



x <- seq(0, 1000)

plot(x = x,  y = scaled_power_transform(x, gamma = 0.1), col = 'blue',  
     type = 'l', lwd = 2, ylab = 'x transformed')
lines(x, x/max(x), lty = 2)
lines(x = x, y = scaled_power_transform(x, gamma = 0.5), col = 'green', 
      type = 'l', lwd = 2)
lines(x = x, y = scaled_power_transform(x, gamma = 2), col = 'red',   
      type = 'l', lwd = 2)

legend(x = 0.6*max(x), y=0.3, 
       legend = c(expression(paste(gamma,'= 0.1')), 
                  expression(paste(gamma,'= 0.5')), 
                  expression(paste(gamma,'= 2.0'))), 
       pch = rep('*', 3), col = c('blue', 'green', 'red'))

Created on 2020-10-10 by the reprex package (v0.3.0)

$\endgroup$
1
$\begingroup$

My earlier post has a method to rank between 0 and 1. Advice on classifier input correlation

However, the ranking I have used, Tmin/Tmax uses the sample min/max but you may find the population min/max more appropriate. Also look up z scores

$\endgroup$
1
$\begingroup$

To add to the other answers suggesting pnorm...

For a potentially optimal method for selecting parameters I suggest this approximation for pnorm.

1.0/(1.0+exp(-1.69897*(x-mean(x))/sd(x)))

pnormish

This is essentially Softmax Normalization.

Reference Pnorm in a pinch

$\endgroup$
1
$\begingroup$

There are two ways to implement this that I use commonly. I am always working with realtime data, so this assumes continuous input. Here's some pseudo-code:

Using a trainable minmax:

define function peak:
    // keeps the highest value it has received

define function trough:
    // keeps the lowest value it has received

define function calibrate:
    // toggles whether peak() and trough() are receiving values or not

define function scale:
    // maps input range [trough.value() to peak.value()] to [0.0 to 1.0]

This function requires that you either perform an initial training phase (by using calibrate()) or that you re-train either at certain intervals or according to certain conditions. For instance, imagine a function like this:

define function outBounds (val, thresh):
    if val > (thresh*peak.value()) || val < (trough.value() / thresh):
        calibrate()

peak and trough are normally not receiving values, but if outBounds() receives a value that is more than 1.5 times the current peak or less than the current trough divided by 1.5, then calibrate() is called which allows the function to re-calibrate automatically.

Using an historical minmax:

var arrayLength = 1000
var histArray[arrayLength]

define historyArray(f):
    histArray.pushFront(f) //adds f to the beginning of the array

define max(array):
    // finds maximum element in histArray[]
    return max

define min(array):
    // finds minimum element in histArray[]
    return min

define function scale:
    // maps input range [min(histArray) to max(histArray)] to [0.0 to 1.0]

main()
historyArray(histArray)
scale(min(histArray), max(histArray), histArray[0])
// histArray[0] is the current element
$\endgroup$
1
  • $\begingroup$ This can all be implemented in Max/MSP/Jitter with the [peak] and [trough] objects for the first example and with [jit.3m] for the second example. $\endgroup$ – terrace Aug 3 '10 at 15:40
0
$\begingroup$

A very simple option is dividing each number in your data by the largest number in your data. If you have many small numbers and a few very large ones, this might not convey the information well. But it's relatively easy; if you think meaningful information is lost when you graph the data like this, you could try one of the more sophisticated techniques that others have suggested.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.