# Ranking Prediction Intervals - Multiple Comparisons?

I fit a model that tries to match the personality of sales reps to customers based on demographics. It's a hierarchical bayesian model that predicts the probability of conversion with sales rep[i] given the customers demographic. I.e. Are some sales reps good at talking to older / younger customers?

$$P(\text{Conversion w/ Sales Rep[i]} | \text{Customer's Demographic}) = \alpha[\text{sales rep}[i]] + \beta[\text{sales rep}[i]] X$$

I then take an incoming customer, and make a prediction for the probability of conversion of that customer with each and every agent and rank the prediction intervals - the best agent to then match this customer with is the one that has the highest probability of conversion credible interval.

1. Is this reasonable to do or does it violate the multiple comparisons problem since I begin comparing 100 different prediction credible intervals? Is it ok because its bayesian?

2. If 1 is reasonable, would the same approach still hold true If I used a frequentist model and used confidence intervals instead?

3). What's a good way to assess the model? It seems like I don't care as much about predictive capability, but more so that the coverage of the credible intervals are plausible - is there a good way to validate that for logistic models such as this?

1. You will almost surely make a mistake by doing this, but that doesn't make the method invalid in my opinion. The problem of multiple comparisons doesn't seem to be an issue since you aren't doing a hypothesis test per se.

2. A frequentist mixed effects model sounds fine to me too, but I'm not sure how you would use the interval themselves to make the decision. Assuming two intervals overlapped, what would you do? It isn't clear from your question.

3. It seems like I don't care as much about predictive capability

Really? It kind of does since you are interested in predicting which rep to deploy. In any case, I think posterior predictive checks are probably the best way to go here. What checks to perform depends on the problem, and only you know best what matters in this context.

Aside from all this, I would probably reframe this as a Bayesian decision problem. Rather than deploy the rep which the largest estimated probability, instead compute the expected loss of deploying rep $$i$$. It might be the case that some effects of some reps are quite large, but fairly uncertain. In such cases, the expected loss would be larger than a rep who has smaller yet more precisely estimated probability of conversion.

• Thank you! I'm ranking with the 95% least plausible value (aka the 5th percentile value) so that i have a conservative comparison that takes into account agents with wider uncertainty intervals. I'm going to take a look at loss functions more closely and see if I can come up with something, thanks again! – J Doe Sep 21 '20 at 12:20
• I wouldn't take the lower end point as an estimate in a worst case scenario. My advice is to turn this into a Bayesian decision problem, compute how much money you would lose if you chose the wrong sales person, and minimize your expected loss. – Demetri Pananos Sep 21 '20 at 14:13
• thanks Demetri! – J Doe Sep 21 '20 at 16:29