1
$\begingroup$

I have a data which has heavily positively skewed variables. My manager has told me to fit it some distributions (univariate). The problem is that for most of the data the mean is far closer to the lower limit than the upper (large values are outliers or maybe the distribution of the data is like that itself)

Can I fit Weibull distribution to this? Its graph looked the most similar.

Here is a sample histogram :

Histogram of Total Paid Amount. Univariate Number 5000 observations

$\endgroup$
  • 2
    $\begingroup$ Your example underlines that a histogram is not really helpful here as the fine structure of all but the first bin is hidden completely. More useful plots typically show the data on a logarithmic scale and some function of the cumulative probability also on a logarithmic scale. If there are zeros in the data, they need different treatment. $\endgroup$ – Nick Cox Nov 5 '15 at 12:24
2
$\begingroup$

So, Tot_Paid_Amt is zero-heavy and/or concentrated around small values less than ~500. Otherwise, it is sparsely populated with a few values ranging out to ~500k, correct? To me, Tot_Paid_Amt sounds like and is behaving like information from insurance claims data where there are lots of small claims and a few huge ones. In instances like these, the mean would be a very biased and unreliable measure of location. You would be much better off using a more robust measure of location such as the median.

In terms of distributional assumptions and assignments -- given that the histogram appears to be heavy tailed or extreme valued -- estimating a tail index for your data would be a more rigorous method for classification than just "guessing" Weibull. In this PDF, Cosima Shalizi presents some rigorous approaches to evaluating a tail index ...

http://www.stat.cmu.edu/~cshalizi/2010-10-18-Meetup.pdf

as well as here ...

https://www.cs.purdue.edu/homes/agebreme/Networks/papers/clauset-powerLaw-siamRev05.pdf

However, there are less rigorous, "quick and dirty" proxies for doing this based on OLS regression that Shalizi disapproves of but that may be "good enough" for your purposes, whatever they are ...

http://www.eco.uc3m.es/temp/jbes.2009.06157.pdf

Having obtained a tail exponent, there is a nice discussion of what they mean as well as assignment rules in the Examples section on this Wiki page ...

https://en.wikipedia.org/wiki/Tweedie_distribution

How would this be helpful in an applied context such as yours? As noted, any effort at predicting average claims amounts, stoploss thresholds -- whatever -- rooted in classic, textbook approaches such as OLS regression will be hugely biased and unreliable. While standard actuarial workarounds would prescribe deleting outliers (based on a data-driven cutoff) to obtain a more "representative" distribution enabling mean-based estimators, this has the serious drawback of needlessly throwing away useful information -- as the tail exponent indicates. Just one of the challenges with using this approach is that, having thrown away the outliers, how do you add them back in to predicted values based on that trimmed distribution to obtain the "right" predicted totals (or summed claims amounts)?

In my opinion, robust predictive modeling methods such as quantile regression should provide a "good" (or less biased), tractable solution to these challenges. The default is typically to estimate the median quantile but a grid search across a range or set of possible quantiles would pin down a better value in terms of a bias-variance tradeoff. Given the positive skew, you can safely assume quantiles in a range less than the median will be most likely to minimize some loss function.

If none of these suggestions work for you, please elaborate on how you would like to use Tot_Paid_Amt.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Independently of these good points, there is a quite different defence of the mean. If we guess that the variable here is total paid amount, its total in turn (and to that extent its mean) is a natural and well-defined quantity for a company or other institution, regardless of the distribution. However, that has no bearing on the question of how to work with a long-tailed distribution. The point is general for any additive or extensive variable. $\endgroup$ – Nick Cox Nov 5 '15 at 12:21
  • 1
    $\begingroup$ When dealing with time series such as the history of paid amounts we often encounter long-tailed distributions. After identifying one-time anomalies (pulses) the resulting histogram reflecting outlier-adjusted values is often less long-tailed. In a similar vein if the observed series has a level shift i.e. bi-modal where the mean shifts from A to B this generates an original histogram that suggests a two-mean approach. Detecting level/step shifts then renders the level-shift adjusted series to be more homogenous. $\endgroup$ – IrishStat Nov 5 '15 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.