This is tricky because the model on the categorical variable will estimate bucket-means, but the bucket means depends on the distribution of ages within each bucket.
Any model that attempts to get some kind of smooth estimates and back out the individual age-means will rely - perhaps heavily in some cases - on the assumptions about the distribution of age. If the average age within a bucket is much younger than the midrange of each bucket, the estimates should be very different than if the average age within a bucket is much older than the midrange. Consider the simple case that it's really linear - then there's a hidden bias in the intercept that you can't find in the binned data.
Or consider where the average age in the younger bins is near the high end, while the average age in the older bins is down the low end (imagine the underlying age distribution has high kurtosis). Again consider the simple case where it's really linear - now you have a (hidden) bias in the slope as well.
So this is not a simple "let's just smooth it in some clever way" kind of problem. The cleverest smoother in the universe doesn't know the critical information about where the ages are within the bins unless you can supply it.
This effect may be far greater than the relatively more simple issue of the choices involved in doing the estimation once you have assumed an age distribution in each bucket. There are still some issues related to bias, for example (again, consider the linear case - a binned estimate where each age is centered at the right bin-mean will underestimate the slope), but the effect of this bias can be estimated if you have the distribution within the bin. Simulation may be helpful in examining the effect of different sets of assumptions.