1
$\begingroup$

I have an idea to use ECDF conversion of data to their uniform distribution of equal sample sizes. So, for say 1000 pieces of data, each value should more or less correlate with a .1% representative value, if a value is repeated, that repeated value is translated into another correlated 1/1000%.

So if the number 2 appears 2 times out of 1000 entries.

The overall % of that value is .2%

However, whatever value that was before it, say 1 , that appeared once would be .1%.

so 1 - Maps to .1% and 2 - Maps to .3% (i.e. 2 counts of 2 out of 1000 = .2%, add prior values for cumulative distribution function. aka .1% and .2% = .3%)

I also have a formula that flags for skewed distributions by testing the ECDF converted mean of a distribution for a max error of .275 from .5 mean. If it is, we do a frequency check on the dataset for the median value, and if the frequency is >50% of the values, then we treat the distribution differently. We don't want the 0% to be valued at a high value.

Instead we remove all 0's from the list, convert to ECDF rating. Then re add back 0's. That way we normalize the tail end of the skew.

So would this work to say compare varying sets of data to each other on an additive scale by normalizing them in this method?

$\endgroup$
1
$\begingroup$

I believe what you are doing, or intending to do, is an example of the probability integral transform. In a nutshell, if you know the underlying distribution of data, when converted to the appropriate CDF, those CDF points are uniformly distributed. Here, you are working under the assumption that the empirical is "correct".

$\endgroup$
  • $\begingroup$ U know what I think ur right $\endgroup$ – thistleknot Jun 18 '14 at 15:50
0
$\begingroup$

I believe I was looking for a Rank Transform, or a Rank Normalization.

http://wires.wiley.com/WileyCDA/WiresArticle/wisId-WICS1216.html

http://en.wikipedia.org/wiki/Data_transformation_%28statistics%29

ECDF ~Rank Transform. Although one of the articles discusses using means to separate equal ranks

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.