2
$\begingroup$

I have a dataset that contains music genres, songs and a "speechiness" rate. An example looks like this:

Song, Genre,  Speechiness
Dance with wolves, Trance, 0.05
My heart will go on, Pop, 0.09

I have used this data to create charts of the with diffent parameters per genre (including speechiness). Thing is that the data is highly skewed (the danceability ranges from 0-1 and 80% of the data is <0.1.

If you plot the data this is what I get:

https://www.flickr.com/photos/112983354@N05/23025503853/in/dateposted-public/

If I plot the average "speechiness" per genre the visables do not really cut it. If I plot it in a radar chart, this is what I get (the S represents the speechiness):

https://www.flickr.com/photos/112983354@N05/23356885200/in/dateposted-public/

All are between 0.05 and 0.1. I now would like to change the values so the graphs better show the difference between the genres. Of course I could limit the axis but something tells me there is a better option.

Any thoughts on something like a scaling method so I get a better and more informative graph?

$\endgroup$
  • $\begingroup$ Could you please provide a graph? I'm having a hard time understanding what you are exactly asking, so all your distributions are overlapping and you cannot discern their difference? $\endgroup$ – plumSemPy Dec 10 '15 at 11:38
  • $\begingroup$ Also providing sample data would help people to produce meaningful examples. $\endgroup$ – Tim Dec 10 '15 at 11:52
  • $\begingroup$ What are "visables" here please? $\endgroup$ – Nick Cox Dec 10 '15 at 13:20
  • $\begingroup$ @Tim: I've updated my post $\endgroup$ – Frank Gerritsen Dec 10 '15 at 13:22
2
$\begingroup$

For heavily skewed data I would normally get the log() of the value. The only issue here is that your scale is so small it would become hard to interpret. You could scale up your data and then the get the log e.g. log(x*1000) or alternatively you could normalise it, center the scale at zero and adjust the standard deviation to one.

set.seed(9467)

d <- abs(rnorm(10000, 0.075, 0.005))    #generate most freq values
d.larg <-rnorm(10, 0.5, 0.3)            #generate some high skewing values

ds <- rbind(d, d.larg)                  #combine data
skewness(ds,na.rm=TRUE)                 #highly skewed 


# Using the log to scale back larger values
x11()
par(mfrow = c(3,1))
hist(d)
hist(ds)
hist(log(ds + 0.000001))                #adding the 0.000001 is so you don't get NA's for zero values


#centering the dataset by normailising it 
normalised.ds <- (ds - mean(ds))/sd(ds)
x11()
par(mfrow = c(3,1))
hist(d)
hist(ds)
hist(normalised.ds) 
hist(normalised.ds, breaks = 20)        #could add more breaks to see smaller distributions
$\endgroup$
  • 2
    $\begingroup$ The major issue is likely to be with any reported zeros. Adding a very small number such as 0.000001 (as commented in your code) is not at all a conservative solution. Consider log base 10 for convenience; then zeros are mapped to -6; 0.01 to just above -2 and 1 to just above 0. So the fudge will create a massive outlier spike. The issue arises with any other base, naturally. A better although still imperfect solution is to add a number such as half the smallest reportable positive value: thus if results are reported to 2 d.p., adding 0.005 will reduce (but not remove) this problem. $\endgroup$ – Nick Cox Dec 10 '15 at 13:09
1
$\begingroup$

The problems are definitely

  • Extreme skewness

and very likely

  • Reported zeros

  • Granularity of results, e.g. reporting to $2$ decimal places, i.e., $0, 0.01,$ etc.

Absent data to play with, I invented some as, given a sample $x$ of size $1000$ from a uniform on $(0,1)$, "scores" $\lfloor 100 x^5 \rfloor/100$. Here is what they look like fairly conventionally in a particular sample which may understate the problem as $662$ are less than $0.01$.

enter image description here

The first idea is clearly a transformation of the scale. Logits are ruled out by the occurrence of exact zeros. Rather than fudge or generalise the definition of logit, I suggest considering folded powers as discussed by J.W. Tukey, most accessibly in 1977. Exploratory data analysis. Reading, MA: Addison-Wesley. By analogy with $\text{logit}\ x = \ln[x/(1-x)] = \ln x - \ln (1 - x)$ consider any positive power $p$ and $x^p - (1 - x)^{1-p}$. Square roots $(p = 1/2)$ and cube roots $(p = 1/3)$ spring to mind. Even the cube root leaves that distribution skewed and the granularity is a little disconcerting.

enter image description here

The problem is naturally that we have not transformed frequencies either. Here a root scale is also a possibility (which Tukey elsewhere called a rootogram).

enter image description here

The trade-off here is delicate. To see any but the grossest skewness of the data, we need to think on what are likely to seem unconventional or unfamiliar scales.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.