0
$\begingroup$

I have a bunch of samples, about 35, drawn from a fat-tailed distribution. I think it is reasonable to assume that the samples are all drawn from distributions from the same distributional family, though the parameters will vary from sample to sample.

The top end of the sample is censored. The sample itself is random. (Well, really it's probability weighted. But let's ignore that for purposes of this question).

I have a couple hundred observations for short distance below the censorship threshold, enough to estimate the average value and the slope at the threshold point. I do not think they are spread out enough to estimate, e.g. curvature.

For the interval between the censorship threshold and a second, higher value equal to about five times the threshold value, I have the number of observations and the average value.

I have the number of observations above the higher value just described.

So, that is five pieces of information for each sample.

I have several candidate three- and four-parameter distributional families. I would like to use the data I have to estimate the parameters and select between the distributions.

The best thing I have been able to think of is to use a basic hill-climbing optimization algorithm on the parameters of each distribution to minimize, say, the product of the squared differences between the observed and estimated value of the five parameters (the product because the observations are in three different units, so they can not be added). But this procedure is admittedly very ad-hoc, and I would prefer something more principled. Also, I'm really not sure I should be treating the slope the same way as the other values. Finally, I'd just be comparing fit, like selecting the model with the highest R squared. That does not seem ideal

Is there a better way to do this? Is there any principled way to do it? I have seen maximum likelihood approaches for binned data, but I do not know how to implement them with such diverse summary information, especially when I am comparing different functional forms.

$\endgroup$
1
$\begingroup$

Let's call the bounds $u_1$ and $u_2$, let's further assumed your distributions have pdf $f(x)$ and cdf $F(x)$. Except for one thing, the easiest thing would seem to be to use maximum likelihood with contribution $f(x_i)$ for observations with $x_i \leq u_1$, $F(u_2)-F(u_1)$ for observations between the two bounds and $1-F(u_2)$ for observations beyond the upper bound. The one thing that does not fit perfectly into this framework is that you know the mean of the values falling between $u_1$ and $u_2$. It sort of feels wrong to throw away the available information in the average, so one alternative possibility is to use approximate Bayesian computation.

What is the process for having that situation (with knowing the average)? The only type of data where this tends to happen is, if the data is really available (someone must have calculated the average), but somehow you cannot get it all (e.g. anonymized tax data). Perhaps looking into the literature of how other people deal with this type of data would be helpful.

$\endgroup$
  • $\begingroup$ Dear Bjorn -- yes, it's anonymized survey data. Good call. The average over the threshold is supplied by the agency. Except it is really the average between the two thresholds, with observations above the upper threshold included in the average as if they were at that threshold. This is not even mentioned in the standard documentation, and so far as as I can tell only one other person has used the information. And I don't think they did it correctly. Seems like most people either believe the average (it's the Census, after all!) or disbelieve it and use something Pareto-like. $\endgroup$ – andrewH May 31 '16 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.