Building an odd predictive model So I've perused this site and others, and have a question that I can't seem to find the answer to (possibly because I'm unsure how to even word it).
Here goes:
So I'm coming from an ecological stats background, and have done modeling, stepwise regression, etc, but have an interesting problem right now.
I am working on a project where there is scoring done, and it is basically binomial (either they have a response or they don't), but the scores are weighted differently, so one could be 0 or 9 and the next could be 0 or 4.  There is one variable that is discrete, but the rest are in that fashion.  
What I want to do is to use the very small sample of responses we have (about 150 out of 40000) to test and validate a model that will tell us what variables are most important.
The hard part is that the weights are going to probably be half stats based, half based on the requirements of the business people in my group.  They have set the scores, and have some flexibility, but it wouldn't work for me to tell them that the model is most significant in a certain way if that didn't line up with their business definitions.
What I've done is created a model that was fully binomial, and started with that.  However, I want to incorporate their scoring, or at least be able to get to a place where we can add in the scoring element while staying in the middle between completely numbers driven and completely business driven.
My long winded question is this: What is the best way to approach this modelwise? Letting them define weights and then just going from there? Or is there a way where I can really just meet them in the middle?  
 A: The solution you are looking for is called Bayesian decision theory.  It constructs the Bayesian posterior density function as you would for any purely statistical question, but then minimizes a cost function over that posterior density to find the lowest cost point solution.  You could treat different models as parameters themselves and solve for the probabilities that the models are good models, and then over that space, which would be discrete, impose a cost so that if you had two models that were close in probability but vastly different in cost you could allow the interaction of costs and probabilities to decide your model.
The model could then be used predictively, again with a cost function, and then scored based on the cost of the prediction being wrong.
Parmigiani produces a good book on introductory Bayesian and Pearson-Neyman decision theory.  Its ISBN-13 is 9780471496571.
This may cause your business office to rethink their costing as well.  When you impose discipline and structure on the decision making, people start talking about their hidden assumptions.  It does happen, from time to time, that when people talk about things out loud with others a better model appears.
