Modelling the distribution of rare events. How to validate? I have been asked to create a model for the distribution of the number of items any previous customer might buy from a shop. Each customer will get a distribution of their own based of their past behavior.
Let's say I have a model for this. On any given month, a fraction of customers will visit the shop once. For each of these, how do I tell if my predicted distribution is any good?
At the moment I count how many items fall within each decile of their predicted distribution. When summed over all predictions, roughly 10% of the samples fall in each bin.
Is this a good approach? Is there a better approach? Is there any way to test if my model systematically miscalculates certain models?
 A: I would venture to guess that you wish to fit a Poisson or a negative binomial model with potential overdispersion.  You could do this with a GLMM (generalized linear mixed-effects model).
Considering the constraints that you have placed, you will have problems getting enough data if you want to actually pioneer a new distributional form.
Hence my suggestion of using the standard count data distribution: the Poisson.  It is a simple, one-parameter distribution.  However, your suggestion that you want this customized to the customer implies you believe the Poisson parameters are "over-dispersed" among customers.  I agree that this seems likely but you can test it in the model. The random-effect term will cause each customer's estimated random-effect to undergo "shrinkage" toward the mean but I think that is HIGHLY desireable in your case as I think Stein-effects are good in your case.
You could fit a Poisson model to EACH customer's data that has come in more than once. That is essentially the FIXED effects model but this will not allow you to really "fit" sparse customers...the MIXED effects model will use all the information you have...even for customers who have only come in once.
I think it likely that the customer's Poisson visit parameters are probably going to be over-dispersed, also.  If so, you will probably find it easier to MODEL the customer-to-customer over-dispersion in the random-effects part of the GLMM while treating the within customer over-dispersion as a nuisance parameter by using the negative binomial as the distribution.  The over-dispersion within customer may also be predictable by seasonal effects but I would not attempt modelling that, at first.
Is this too much?
