Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a distribution that is skewed to the left (i.e. a lot of low values, aka positive skew - verified)

Basically, I have either a lot of similar low [~minimum] values, or 0 values, or something like 1 value that is the only value below the median value.

The data has already been normalized to 0 to 100% based on an ECDF run [transformation]. AKA an Empirical Cumulative Distribution Function, very similar to a rank transformation, it's very simple. Is value equal to or less than the rest of the values, if not, then count the number of values less than or equal to current value, and count that towards that values overall Cumulative Distribution Function.

So... is there a simple way to mean adjust the ecdf distribution to a .5 mean, and slope the data from 0 to 100%

My first guess was to track the difference from a .5 mean.

My second guess was to derive the # of unique #'s in my [skewed] distribution to establish categories I needed to divide by from my difference from mean.

Sample set of data: 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2

After ecdf conversion 0.05 0.55 0.55 0.55 0.55 0.55 0.55 0.55 0.55 0.55 0.55 1 1 1 1 1 1 1 1 1

mean: 0.7275

I had one idea to factor DOWN anything equal to or below median by .5

our desire to achieve .5 mean is because we are combining the distribution with other distributions that after ecdf conversion already have a ~.5 mean. The problem with our skewed data set is it's promoting low skewed values to high %'s and we don't want to do that. So, we would prefer to achieve a .5 with the skewed data.

share|improve this question
1  
On your first paragraph: Your description seems at odds with your terminology - it sounds right skew (positively skew) - can you show your distribution? The remainder of your question is very unclear. I have no idea what you mean by "adjust the ecdf distribution to a .5 mean" or "slope the data" or "track the difference from a .5 mean" or "establish categories needed to divide by" or "from difference from my difference from mean". As you may note, that's almost the whole question ... and I'm not all that certain about the rest of it. Your terminology is confusing. Please try to clarify. –  Glen_b Jun 25 at 1:46
    
thanks for chiming in, I edited it up. Sorry for my poor grammar. –  thistleknot Jun 25 at 2:47
1  
I understand what an ECDF is, but it's not exactly clear what you're doing. Can you show some pictures or tables perhaps of what you have done and what exactly you want to have mean 1/2 - and preferably, make it clear why you want to do that. It seems you want the values taken by the ecdf itself to have mean 1/2 ... but I can't seem to make sense of that with highly discrete data. What would that even mean? –  Glen_b Jun 25 at 2:51
    
I updated the info to show a sample data set I have. I have other data sets that I'm trying to compare it with. If you would like, I could supply those, but generally speaking, I get a flat distribution curve after running the other data through an ecdf conversion. –  thistleknot Jun 25 at 17:01
1  
Data from a continuous distribution should be a discrete uniform after transformation (what I presume you mean by 'flat'). You might note that this accords with what one would expect since for a continuous distribution, the result of the probability integral transform $F_X(X)$ is uniform. Discrete data won't have that property, and the more data you have in the mode the less uniform it will look. –  Glen_b Jun 25 at 21:32
add comment

1 Answer 1

Sorry if my question was poorly worded.

I found that if I combined a simple ( (rank function / count) + (ecdf %) ) / 2 gave me a skew corrected curve.

Example of it in action

https://docs.google.com/spreadsheets/d/1gitnUzUyaROi-QroCXvXbY2raFBJTHZk7YCMWTlOHjw/edit?usp=sharing

It's friggin' ingenious. The way ecdf %'s are calculated is it counts the last rank position of similar elements, and rank (aka excel rank, or Google docs rank) returns the first position of similar elements.

I noticed the behavior when I compared a rank% vs an ecdf% and noticed my mean for rank% was like the opposite of the mean for ecdf%.

So now we found a better way to standardize all our data by finding the center points between the two methods to get a more direct % of a number.

We also found that 50% means "neutral" now, and not necessarily bad. If an entire distribution is of equal value (to include 0), the data will fall into a 50% range. This means, that a person [in our case] is neither good or bad for a particular job.

Hope you guys like it, it'll be implemented in the next version of Dwarf Therapist ;)~

It even did a mixture model for me! http://m.imgur.com/KKljFOg

That scale runs from 40% to 100% only had 128 unique values, the rest were 0s

Here's how the conversions worked

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.