My exact problem is this.
I have a number of sources of traffic with different conversion rates. I have good evidence that conversion rates vary based on the source. For each traffic source I have how many leads we got, and how many converted. I want an estimate of each source's conversion rate. I do not need individual sources to be accurately estimated (this would be impossible given that many traffic sources only have 0-5 measured conversions). Instead I would like to be able to say that, across all traffic sources estimated at 1%-1.5% average conversion rates, the average future conversion rate is likely to be somewhere in that general ballpark.
The obvious approach of using past performance to predict future performance for each individual source fails badly because of the well-known phenomena of regression to the mean. For instance, among small traffic sources with no conversions there is a reasonable expectation of future performance being better than that. And likewise small traffic sources with lots of conversions are unlikely to maintain their record.
My naive idea is to take my data, and use it to produce a reasonable Bayesian prior for the true conversion rate of a random traffic source. Then for each source I can start with that prior and produce a posterior distribution for the true conversion rate of that source. And then my estimate of the average conversion rate for that source will be the average of the posterior.
My initial idea for how to fit data to a reasonable prior works like this. My prior will be the sum of piecewise linear functions, which will individually be 0 to a starting point, rise to a midpoint, then fall to the ending point after which it is 0 again. For a given division of the likely range of conversion rates into these intervals and midpoints, the prior I'd produce would be the one to maximize the sum of the logs of the likelihoods that each traffic source would have the conversions that it did.
My problem is that the more pieces I divide my interval into, the more closely I can fit my existing data set. But at some point I'm clearly over-fitting. Are there any guidelines that I can use to get a sense when I'm over-fitting my prior to my data?
I'm thinking there might be something like a statistic that I can compute to test how closely my measured data fits the prior - if the measured data has a better fit to the prior than a "random" data set should, then I've probably gone too far.
I would be grateful if anyone can suggest a statistic, an alternate rule of thumb to avoid over fitting, or a different approach to the original problem. Since I do not have access to a university library, please only suggest specific books or papers behind paywalls if they are guaranteed to be very relevant to my problem.