I have a dataset with the number of businesses, e.g. bakeries or doctors, in some towns and cities.

I want to calculate the relative distribution of certain professions, but as the population size varies greatly between cities, the result can become quite volatile.

For example, if I have by chance one lawyer registered in a town of 20 people, the resulting 'lawyer density' of 5% by far surpasses the results from all other cities (especially the larger ones, where the value is more stable).

How to best / correctly deal with this effect? I calculated the Clopper-Pearson intervals, but the results are only able to remedy my problem to a certain extent.

  • 3
    $\begingroup$ Just as big a problem is that you may have many zeros too, as 0 lawyers out of 20 seems just about as likely. Hence one simple answer that log ratios often behave better than ratios can't be applied just like that. More general answers: (a) if you're pooling the ratios, use denominators as weights; (b) don't use these ratios as such, use the denominator as an offset variable; (c) some kind of shrinkage estimation, Bayesian or otherwise, as in @Hugh Perkins' helpful answer. $\endgroup$
    – Nick Cox
    Oct 12 '17 at 10:38

Initial idea

How about: model the number of lawyers in each town as the prediction of a Bayesian model? The actual number of lawyers is used as the input.

You can draw stochastically from the resulting model, which means some towns with no lawyers will occasionally be drawn with a lawyer, and some with three lawyers will occasionally be drawn as having no lawyers.

Alternatively, you could work with the resulting output as an estimate of the number of lawyers. So a town with no lawyers might show with an estimate of 0.2 lawyers, and a town with one lawyer might show with an estimate of maybe 0.22 lawyers.

Actual usable algorithm

In practice, because you have so many towns, what you can do is a slight nuance on this: create a single global model, per profession, which predicts a probability distribution over the number of lawyers in a town, given the number of people in the town, and a set of parameters:

$$ p(N_{\text{lawyer}} \mid N_{\text{town}}, \theta) = f(\theta, N_{\text{town}}) $$

Using this, and given your data $\mathcal{D} = \{ \mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_n \}$, we can obtain the probability of the data given the parameters:

$$ p(\mathcal{D} \mid \theta) = \prod_{i=1}^n p(\mathcal{D}_i \mid \theta) $$

Then use Bayesian inference on this to find the posterior distribution over $\theta$:

$$ p(\theta \mid \mathcal{D}) = \frac{p(\mathcal{D} \mid \theta)\,p(\theta)} {\int_{-\infty}^\infty p(\mathcal{D}, \theta)\, d\theta} $$

... which can be estimated using eg VAE, https://arxiv.org/abs/1312.6114

or you could use a MAP estimation, which avoids handling the intractable marginalization for the evidence:

$$ \theta^* = \text{argmax}_{\theta} p(\mathcal{D} \mid \theta)\,p(\theta) $$

Once you have either $\theta$ or a probability distribution over $\theta$, well $\theta$ is pretty much the solution to your problem directly, since it is the parameters of a probability distribution showing how the number of people having a certain profession varies with the size of the town. But you can then do things like:

  • find the total number of lawyers, by integrating over all towns (eg numerically); or
  • estimate the number of lawyers, holding the size of the town fixed

If you have these data on a lot of small towns and some big ones, then one solution would be to pool the small towns within a specific region (e.g. state in the USA) so you would have something like:

New York City, Albany, Buffalo, Rochester, Yonkers, Syracuse, small town NY

for New York State.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.