# How to analyse a right skewed dataset

I am trying to analyse several heavily right skewed datasets in order to determine their mode.

The story behind the sets is that they are calculations of performance data from marketing posts. The datasets are the calculations and I am trying to use to determine a baseline for scoring future posts. My theory is that the centre of the mode (after binning the data and creating a histogram) would be the target baseline and new posts will be scored against this baseline to determine a score from 0-100. The reason for the heavily right skewed datasets are largely due to badly performing posts. The tail of these sets can be huge. I am aware of major outliers and have ways to manage that.

The reality is though... I am not a mathematician and this is essentially a side project I am doing my best to see it through. My theory above is as close as I have got to a working model for the scoring system but I'm open to suggestion on better/more accurate statistical frameworks for doing this... I would say though that any suggestions should be explained as if I'm 5.

• Hi, welcome to the site! Can you expand on what you mean by "mode"? Because I think it's something different than what statisticians have in mind :) Jul 25, 2022 at 21:06
• Hi John, thank you for responding and apologies if I've got my terminology mixed up. I am binning the data then checking for the bin with the highest number of data points in that bin. This is what I am interpreting as the 'mode'. None of the numbers are the same in any of the datasets, so finding the actual 'mode' was not possible. Hence why i went down the binning/histogram route. TIA Jul 25, 2022 at 21:16
• Ah no OK we have the same idea for "mode"! I'm going to try to rephrase your question, and you let me know what I got wrong. You're saying "I have these results I want to use to help calibrate/quantify future performance data by assigning them a score from 0-100." And you were essentially viewing the mode of the data you've collected so far as a means of doing that, is that right? Jul 25, 2022 at 21:20
• That sounds about right yes Jul 25, 2022 at 21:30
• Yeah word, I'm going to ditch the idea of using the mode in my answer, let me know if that was essential for some reason. Jul 25, 2022 at 21:52

The approach I would take is to score new performance data by computing what its percentile would be on your existing data. You can either do this directly or by modeling your data with a probability distribution.

Or, well, because a percentile tells us what percent of our previous data is less than a particular datapoint, we probably want to look at 100-the percentile instead, that way 100 is as good as possible instead of as bad as possible.

I'm going to demonstrate this procedure using python. I'm going to be using these libraries:

import numpy as np
import scipy.stats as stats


The first thing I'm going to do is create some example data (you can skip this step, since you already have some):

# Create some fake data (you already have some so can skip this :)).
old_data = stats.gamma.rvs(5, scale=22, size=1000)
new_data = np.array([50, 80, 100, 200])


Here's the first approach, of manually computing percentiles. We just calculate what percent of the existing performances we exceed:

# Approach 1: Empirical Percentiles.
for i in range(len(new_data)):
print(100-np.mean(new_data[i] > old_data) * 100)


The result of this is

91.5
68.1
49.4
5.5


If you fancy it, you can use a parametric distribution instead. I'm trying the Gamma distribution out:

# Approach 2: Model-based percentiles.
# Step 1: Find a Gamma distribution that fits your data
alpha, _, beta = stats.gamma.fit(data, floc = 0.)

# Step 2: Use that distribution's CDF to get percentiles.
scores = 100-100*stats.gamma.cdf(new_data, a = alpha, scale=beta)
print(scores)


The result of this is:

[92.38913684 70.38297238 52.46867026  4.86655548]


It would have to be a rather precocious 5 year old who understands this, so feel free to fire off questions.

• Wow! Thank you John! I understood everything up to the word parametric and also tried the python code myself. I'm building this in ruby due to the projects code base but I can convert the calculations. It's a really neat and tidy model for creating the scores and I believe it will produce more accurate results too. If you could expand on the parametric side of things, I would be grateful. I assume this is smoothing the data in some way? Jul 26, 2022 at 20:46
• @bubbaspaarx happy to help. The parametric approach smooths the data, but also imposes additional assumptions. Frankly, it's probably an inferior approach unless you have very few historical data, but it does allow for nonconstant scoring outside of the range of our previous data, which our first approach does not. Jul 26, 2022 at 20:55
• @bubbaspaarx can you clarify: it sounds like rather than just doing this once, you've got several datasets for which you would like to develop individual scoring rules? In this case, it's a little more complex, but it may be possible to reliably go below "100 data" by borrowing strength between datasets. I have "100 data" in quotes cause it's not clear to me off the top of my head what a reasonable cutoff is. Jul 26, 2022 at 21:05
• @bubbaspaarx gotcha, the good news is that it's not hard to implement ourselves using Method of Moments estimation. If you look at the Wiki page for the Gamma distribution, the "info panel" on the right at the very bottom you'll see method of moments. E[X] means your sample mean, and V[X] means your sample variance, and its how we go from data to parameters for the Gamma distribution. One "gotcha" is that there are two common parameterizations of the Gamma (hence the two columns on Wiki), and you've got to figure out which one your Ruby Gamma cdf function is using. Jul 29, 2022 at 16:38
• @bubbaspaarx doesn't matter whether you use N-1 or N for variance calculation (lord knows why society has spent so much time thinking about that) i.e. whether you use Bessel's correction or not. The inverse of the CDF is either called just the "inverse cdf" or the "probability point function" (ppf), I don't see it in the docs you linked unfortunately. Aug 1, 2022 at 14:02