2
$\begingroup$

I apologize if this question has been asked before. I am looking for the best way to describe my data set. A little background - I am a algorithmic programmer with a physics/mathematics background. However, I never really dived way too deep into statistics, anyways long story short; statistics is definitely a different beast than what I have been exposed too.

At my company I have been given a task to essentially create a 'EBITDA' calculator for a very specific and refined area of our operations. It is essentially a Revenue - Cost = EBITDA sort of deal. The challenge came in where our Costs vary considerably due to a wide range of factors. I spent the last couple of weeks doing internal research to see the different distributions of our independent variable costs.

I used a Monte Carlo approach to create millions of trials and I have a data set that is essentially the result of these trials.

However, I am trying to slowly move our operational data analysis team from using simple averages and standard deviations (good lord they love them) to describe and present data of all sorts.

I know there are a multitude of different ways to look at a dataset and uncountable statistical figures to use. I started down the path of educating myself a little more into statistics to figure out what figures can we use to best describe and present this data ... Alas, the more I dived into statistics the more overwhelming it got.

Long story short - my final Monte Carlo simulation distribution results looks like a combination of a lognormal/weibull/exponential distribution set.

The consumers of this program really wants two things to be spat out ... an average with a confidence interval. Something like this:

$2,500 +/- $20

From my research I have been advised to use geometric averaging, but what is completely eluding me is what to use for this 'Confidence Range' or 'Margin of Confidence' (I apologize if I am butchering or not properly using well known statistical nomenclature for its designed purpose).

Also correct me if I am incorrect, but if a dataset closely resembles a normal distribution then an arithmetic mean and the standard deviation, in my opinion, would be enough to describe the data.

However, my data is NOT normally distributed and skewed very left. I can't justify using standard deviation as my 'confidence interval' as I feel it incorrectly portrays the data and would give false information to the user.

This is really dragging on too long, but at the end of the day I need a number to represent the average (I think i got this) and I need another number to represent: "If you stay between a and b, statistical within x%, you will be okay"

$\endgroup$
  • $\begingroup$ Why not present 95% confidence intervals? And if the distributions are skewed, you could show the median instead of the mean. $\endgroup$ – mkt - Reinstate Monica May 3 '18 at 17:02
  • $\begingroup$ I think we need more information. When you say "If you stay between a and b, you'll be ok", are you wanting people to stay very close to the average (median would be best here rather than mean, due to the skewed nature of the data), or do you just want people to avoid being outliers on one side or the other? Also, if we're talking cost, are you sure it's left skewed and not right skewed? Knowing absolutely nothing about your business or the concerns, it seems like it's the upper outliers that you'd be concerned about. $\endgroup$ – dankernler May 4 '18 at 4:53
  • 2
    $\begingroup$ It will help to clean up some terminology. "Confidence interval" means a specific thing. You may or may not be interested in that. It sounds like you are looking for a measure of location and a measure of dispersion. There are many options. Measures of location include mean, median, trimmed means, Huber M estimators, and others. Measures of dispersion include standard deviation, interquartile range, standard error of the mean. As you suggest, if your data is skewed, mean and standard deviation might not be the best for this data. (Continued...) $\endgroup$ – Sal Mangiafico May 4 '18 at 12:45
  • $\begingroup$ (... continued) How you would determine what statistics would satisfy, "If you stay between a and b, statistical within x%, you will be okay". I don't know. It might help to look back at this kind of data empirically: Did mean or median work better? Did standard deviation or interquartile range work better? $\endgroup$ – Sal Mangiafico May 4 '18 at 12:46