3
$\begingroup$

I have 490 data points, which are very unlikely to be I.I.D. Below is a summary in Million dollars. My goal is to fit a distribution so that its 99.9th quantile captures the 70.22M maximum.

Min. 1st Qu. Median Mean 3rd Qu. Max.

0.00854 0.01135 0.01588 0.18370 0.02997 70.22000

Lognormal, Loggamma, Generalized Pareto, 2 parameter g- and h- distribtion have all failed so far. Generalized Pareto has outperformed the other two with a 99.9th quantile of ~ 0.65M.

I would like to report my current approach to this problem: I declare the max of 70.22 to be an outliner and the rest fits reasonably to a GPD, passing the KS and AD tests so far.

$\endgroup$
12
  • 1
    $\begingroup$ Because this is a data summary, all six statistics are uncertain indicators of the underlying distribution. You really ought to account for that uncertainty when fitting a distribution. (Trying to recreate them exactly would be overfitting--and would likely require unnecessary mathematical contortions.) To account for the uncertainty, we need to know (a) whether your data can be considered independent and (b) how many values you have. $\endgroup$
    – whuber
    Commented Feb 8, 2016 at 19:44
  • $\begingroup$ Thanks Whuber. I have 490 data points, and I am almost sure that they are not I.I.D. $\endgroup$
    – Ye Tian
    Commented Feb 8, 2016 at 20:17
  • $\begingroup$ @YeTian what do you mean by "not I.I.D"? Are the values dependent from each other somehow? $\endgroup$
    – Tim
    Commented Feb 8, 2016 at 20:19
  • $\begingroup$ Maybe they're not identically distributed either $\endgroup$
    – Aksakal
    Commented Feb 8, 2016 at 20:22
  • $\begingroup$ Thanks. In fact, they can be dependent or nonidentically distributed. To make my life simple, I start by assuming that they are iid. $\endgroup$
    – Ye Tian
    Commented Feb 8, 2016 at 20:26

1 Answer 1

5
$\begingroup$

This does not necessarily answer your question but hopefully it will help you develop your question and the goals of your analysis.

Many individuals, when challenged with estimating a distribution for their data, think that the best/only way to do so is by going "shopping" among a finite number of parametric distribution and seeing which one fits best (i.e. log-normal, gamma, Pareto, etc).

While distribution shopping sometimes has it's benefits, it's often not the best way to go. This is especially true when you have a large sample size, cannot assume your data is iid, or are not particularly interested in estimating population parameters via maximum likelihood.

Below are two alternative, non-parametric, techniques which estimate the distribution of your data. They are very easy to implement. These techniques are often applied to non-iid data*.

I also comment more on non-iid assumptions below, and how you may be able to improve your question and get an even better answer by providing more context.

*(This last sentence is sort of a rabbit hole because the non-parametric estimators below are derived with iid assumptions. However, even non-identical dependent data can still converge on a stationary distribution...help from another user here would be appreciated)

If All You Care About Are Quantiles...

Use an empirical distribution function:

Given your data $x_1,x_2,...,x_n$ you can estimate the quantile of any value $t$, denoted $\hat F_n(t)$, with the following equation: $$ \hat F_n(t) = \frac{1}{n+1} \sum_{i=1}^n \mathbf{1}_{x_i \le t} $$ where $\mathbf{1}_{x_i \le t}$ is an indicator which equals 1 if $x_i \le t$ is true and zero otherwise. In your case $n=490$, and the quantile of your max value, $70.22$, is approximately $99.8$%.

You can also use the empirical quantiles to estimate the probability that $X$ will be in any interval i.e. $$ Pr(t_1< X \leq t_2) = \hat F(t_2)-\hat F(t_1) $$

To show this works I generate a sample of size 490 from a Gamma distribution with parameters $\alpha=2$ and $\beta=0.1$ and emperically estimated the quantiles. I then plot my estimates against the true quantiles (which are calculated using the Gamma distribution). Empirical Distribution Function

As you can see they are very close, not perfect, but close. The estimates get better with increased sample size.

If you Want a Density Estimate; $\hat f(X)$...

You can use Kernel Density Estimation (KDE)

This technique is related to mixture distributions (it can be interpreted as a mixture). The density estimate $\hat f(t)$ can be calculated as: $$ \hat{f}_h(t) = \frac{1}{nh} \sum_{i=1}^n K\Big(\frac{t-x_i}{h}\Big) $$ KDE requires you to choose a kernel function (a common choice is the normal pdf) and a bandwidth $h$. Generally the choice of $h$ is the one that effects the KDE more drastically. Discussion about choosing $h$ and $K$ is provided in my link above and the references cited therein.

Below I also provide a plot of the KDE against the true density using the same sample as above.

KDE

Important Note on non-iid

I understand that you want to use something robust to the iid assumption because after all, how do you know for sure that the data are actually iid? However, it is important that you present some reasoning for why your data may be (1) not independently distributed and (2) not identically distributed. For example;

"my observations are household income from multiple geographic regions. I know observations from similar regions tend to be correlated (not-independent) and the variance of household income within each region can vary (not-identical). Unfortunately I do not know which geographical region each observation comes from (the data are unlabeled)"

Providing this context would give someone enough information to formulate another model (like a mixture) that could potentially outperform my suggestions above. Articulating the context of your problem also helps you develop a better understanding of your data and the goals of your analysis i.e. (continuing with the above example)

"Do I really just want a distribution of all the data mixed together? or perhaps, would it be more informative to also group the data into clusters and summarize them seperately?"

$\endgroup$
4
  • $\begingroup$ Thank you very much for your effort. I don't think either empirical quantiles or kernels have the ability to make a given quantile (in my case 99.9th) arbitrarily large, unfortunately. $\endgroup$
    – Ye Tian
    Commented Feb 10, 2016 at 20:18
  • $\begingroup$ @YeTian when estimating a distribution, the 99.9% quantile will only be as large as the data (and model) suggests it is. How would making the 99.9% quantile "arbitrarily large" be conducive of estimating a probability distribution that is consistent with the data? If you're interested in estimating $Pr(X>70.22)$, then perhaps you should look into extreme value theory. Technically, you could also replace $\frac{1}{n+1}$ with $\frac{1}{n+k}$ for $1<k<n$ which would decrease the quantile of 70.22. It would, however, be unusual. This is also different from what you asked for in your question. $\endgroup$ Commented Feb 10, 2016 at 20:52
  • $\begingroup$ I don't disagree with you. However, my largest 14 data points are essentially asking me to fit something that has +Inf as the 99.9th quantile. Notice that my 3rd quantile is only 0.03M and my max is 70M. Moving to a nonparametric distribution will put the 100th quantile to 70M (I haven't tried but it is my intuition), and this in turn defeats the purpose. Btw, the 99.9th quantile is industry standard and I didn't come up with the number myself. $\endgroup$
    – Ye Tian
    Commented Feb 10, 2016 at 21:18
  • $\begingroup$ It may be easier to fit the natural log of your data, if that is a possibility. Other than that I wouldn't know how else to help you. $\endgroup$ Commented Feb 10, 2016 at 21:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.