9
$\begingroup$

(This relates to my programming question on Stack Overflow: Bell Curve Gaussian Algorithm (Python and/or C#).)

On Answers.com, I found this simple example:

  1. Find the arithmetic mean (average) => Sum of all values in the set, divided by the numbers of elements in the set
  2. Find the sum of the squares of all values in the set
  3. Divide output of (2) over the numbers of elements in the set
  4. Subtract the square of mean (1) from the output of (3)
  5. Take the square root of the outcome of (4)

Example: Set A={1,3,4,5,7}

  1. (1+3+4+5+7)/5 = 4
  2. (1*1+3*3+4*4+5*5+7*7)=1+9+16+25+49 = 100
  3. 100 / 5 = 20
  4. 20 - 4*4=20-16 = 4
  5. SQRT(4) = 2

(This comes from a post on wiki.answers.com.)

Now given all that, how can I fit the above data to a bell curve (such as a credit score) ranging from 200 to 800. Obviously the number 5 in the above set would be 500. But then what is the formula for determining what 3 should be on the same scale. Even though the original set Set A={1,3,4,5,7} is not a bell-curve, I want to force it into a bell-curve.

Imagine these are scores of 5 people. Next month the scores might change as follows: Set A2={1,2,4,5,9} (one guys loses a point, and the top guy gains two more points - the rich get richer and the poor get poorer). Then perhaps a new guy comes into the set: Set A3={1,2,4,5,8,9}.

$\endgroup$
  • 2
    $\begingroup$ If your set changes time from time it is impossible to fit it to the same bell-curve. Suppose you have the set $A=\{1,2,3,4\}$, so 4 should be 800, but if a new observation 5 arrives, then it should become 800. $\endgroup$ – mpiktas Dec 31 '10 at 7:43
  • 1
    $\begingroup$ Welcome to CrossValidated, NealWalters. You may find that a common response to asking a statistician "How do i do this" is "Why do you want to do that?" $\endgroup$ – onestop Dec 31 '10 at 11:37
  • $\begingroup$ @onestop - see my link to previouspost. In school, I remember of hearing of teachers who graded "on the curve". So no matter what score you got on a test, only a certain percentage of people would get each grade. I have no idea how credit scores work, but that's very similar to what I'm simulating. I give people points for example when they pay off a loan. But some people will pay off dozens of loans. I want to condense my wild points into a set of normalized points. $\endgroup$ – NealWalters Dec 31 '10 at 14:59
  • $\begingroup$ reading your last comment, I wonder what approach is best: give, e.g. 1 point for each loan paid off, other points for other things, then try to tame the enormous score at the end, or perhaps do something more meaningful with each component that makes up the final score? For example, if you give points for loans paid off you could calculate your Loans Paid Off (LPO) points as log (1 + NumberOfLoansPaidOff). So someone with zero loans paid off gets an LPO of 0, someone with 3 loans paid off gets an LPO of 1.1, while someone with 100 loans paid off gets an LPO of 4.6. Cap LPO at 5. $\endgroup$ – Wayne Jan 10 '11 at 20:38
13
$\begingroup$

A scaled range, like 200 to 800 (for SATs, e.g.), is just a change of units of measurement. (It works exactly like changing temperatures in Fahrenheit to those in Celsius.)

The middle value of 500 is intended to correspond to the average of the data. The range is intended to correspond to about 99.7% of the data when the data do follow a Normal distribution ("Bell curve"). It is guaranteed to include 8/9 of the data (Chebyshev's Inequality).

In this case, the formula 1-5 computes the standard deviation of the data. This is simply a new unit of measurement for the original data. It needs to correspond to 100 units in the new scale. Therefore, to convert an original value to the scaled value,

  • Subtract the average.

  • Divide by the standard deviation.

  • Multiply by 100.

  • Add 500.

If the result lies beyond the range $[200, 800]$ you can either use it as-is or "clamp" it to the range by rounding up to 200, down to 800.

In the example, using data $\{1,3,4,5,7\}$, the average is $4$ and the SD is $2$. Therefore, upon rescaling, $1$ becomes $(1 - 4)/2 * 100 + 500 = 350$. The entire rescaled dataset, computed similarly, is $\{350, 450, 500, 550, 650\}$.

When the original data are distributed in a distinctly non-normal way, you need another approach. You no longer compute an average or SD. Instead, put all the scores in order, from 1st (smallest) up to $n$th (largest). These are their ranks. Convert any rank $i$ into its percentage $(i-1/2)/n$. (In the example, $n=5$ and data are already in rank order $i=1,2,3,4,5$. Therefore their percentages are $1/10, 3/10, 5/10, 7/10, 9/10$, often written equivalently as $10\%, 30\%$, etc.) Corresponding to any percentage (between $0$ and $1$, necessarily) is a normal quantile. It is computed with the normal quantile function, which is closely related to the error function. (Simple numerical approximations are straightforward to code.) Its values, which typically will be between -3 and 3, have to be rescaled (just as before) to the range $[200, 800]$. Do this by first multiplying the normal quantile by 100 and then adding 500.

The normal quantile function is available in many computing platforms, including spreadsheets (Excel's normsinv, for instance). For example, the normal quantiles (or "normal scores") for the data $\{1,3,4,5,7\}$ are $\{372, 448, 500, 552, 628\}$.

This "normal scoring" approach will always give scores between 200 and 800 when you have 370 or fewer values. When you have 1111 or fewer values, all but the highest and lowest will have scores between 200 and 800.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1, I've deleted my answer, since your was way better and made mine obsolete :) $\endgroup$ – mpiktas Dec 31 '10 at 18:42
  • $\begingroup$ Where are you getting the 5 in the 1-5? 4 is the average. Should it be 1-4? I'm testing today with Python, thanks again. $\endgroup$ – NealWalters Jan 9 '11 at 15:56
  • $\begingroup$ @NealWalters Sorry, that was a slip of the finger. I have fixed it now so it reads (1-4)/2*100+500. $\endgroup$ – whuber Jan 9 '11 at 16:55
4
$\begingroup$

You could try this approach - normalise your data set to range between the values -1 and +1 thus: $$ \left(\frac{\text{individual_value} - \text{min_of_all_values}}{\text{max_of_all_values} - \text{min_of-all_values}}-0.5\right)*2. $$ This will convert every value in your data set to a value between -1 and +1, with the actual maximum and minimum values being set to +1 and -1 respectively, and then reset these +1 and -1 values to be +0.9999 and -0.9999 (necessary for following calculations.)

Then apply the Fisher Transformation to each of the above normalised values to "force it" to approximately conform to a normal distribution, and then "un-normalise" each of these Fisher Transform values to range in value between 200 and 800 thus: $$ \frac{\text{Fish_value} - \text{min_all_Fish_values}}{\text{max_all_Fish_values} - \text{min_all_Fish_values}}*600 + 200 $$ The maximum Fisher Transform value will be set to exactly 800, the minimum Fisher Transform value will be set to exactly 200, and all the other values will lie between these two extremes, according to an approximate normal distribution.

Referencing your original question on SO and the issue of scalability, the advantage of this approach is that provided any new data point is not itself a new maximum or minimum for the data set as a whole you can apply the above calculations to the new data point to get its score between 200 and 800 without affecting any of the existing scores of the original data set. If a new data point is a new maximum or minimum you will have to recalculate the scores for the whole data set with this new "normalising" maximum or minimum value.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.