I'm trying to estimate a binary function of $Y$~$Bernouli(p_{i})$ where $p_{i}=f(age)$ where f is an unknown continuous function. There are obviously a lot of methods to try to reconstruct f (splines, Gaussian processes, etc), but what if my age observations are collapsed into buckets (18-29, 30-44,etc)?

Obviously, it's impossible to reconstruct f even with infinite data. And reconstruction of f is going to depend fairly strongly on assumptions about the nature of f. But this seems like it should be a common problem and I'm curious what approaches people have used/would use.

edit 7/14/2014:I should clarify that in this case, I have a pretty good distribution of the overall distribution of age in the population.


This is tricky because the model on the categorical variable will estimate bucket-means, but the bucket means depends on the distribution of ages within each bucket.

Any model that attempts to get some kind of smooth estimates and back out the individual age-means will rely - perhaps heavily in some cases - on the assumptions about the distribution of age. If the average age within a bucket is much younger than the midrange of each bucket, the estimates should be very different than if the average age within a bucket is much older than the midrange. Consider the simple case that it's really linear - then there's a hidden bias in the intercept that you can't find in the binned data.

Or consider where the average age in the younger bins is near the high end, while the average age in the older bins is down the low end (imagine the underlying age distribution has high kurtosis). Again consider the simple case where it's really linear - now you have a (hidden) bias in the slope as well.

So this is not a simple "let's just smooth it in some clever way" kind of problem. The cleverest smoother in the universe doesn't know the critical information about where the ages are within the bins unless you can supply it.

This effect may be far greater than the relatively more simple issue of the choices involved in doing the estimation once you have assumed an age distribution in each bucket. There are still some issues related to bias, for example (again, consider the linear case - a binned estimate where each age is centered at the right bin-mean will underestimate the slope), but the effect of this bias can be estimated if you have the distribution within the bin. Simulation may be helpful in examining the effect of different sets of assumptions.

  • $\begingroup$ In this particular case, I do know the exact age distribution of the underlying distribution. Or at least something close to it. But this is an interesting point in general. $\endgroup$ – DavidShor Jul 15 '14 at 1:56
  • $\begingroup$ That's important information; I'll try to discuss this further soon. $\endgroup$ – Glen_b Jul 15 '14 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.