Ranking Prediction Intervals - Multiple Comparisons?

Question

I fit a model that tries to match the personality of sales reps to customers based on demographics. It's a hierarchical bayesian model that predicts the probability of conversion with sales rep[i] given the customers demographic. I.e. Are some sales reps good at talking to older / younger customers?

$$ P(\text{Conversion w/ Sales Rep[i]} | \text{Customer's Demographic}) = \alpha[\text{sales rep}[i]] + \beta[\text{sales rep}[i]] X $$

I then take an incoming customer, and make a prediction for the probability of conversion of that customer with each and every agent and rank the prediction intervals - the best agent to then match this customer with is the one that has the highest probability of conversion credible interval.

1. Is this reasonable to do or does it violate the multiple comparisons problem since I begin comparing 100 different prediction credible intervals? Is it ok because its bayesian?

2. If 1 is reasonable, would the same approach still hold true If I used a frequentist model and used confidence intervals instead?

3). What's a good way to assess the model? It seems like I don't care as much about predictive capability, but more so that the coverage of the credible intervals are plausible - is there a good way to validate that for logistic models such as this?

Answer 1

1. You will almost surely make a mistake by doing this, but that doesn't make the method invalid in my opinion. The problem of multiple comparisons doesn't seem to be an issue since you aren't doing a hypothesis test per se.

2. A frequentist mixed effects model sounds fine to me too, but I'm not sure how you would use the interval themselves to make the decision. Assuming two intervals overlapped, what would you do? It isn't clear from your question.

3. It seems like I don't care as much about predictive capability

Really? It kind of does since you are interested in predicting which rep to deploy. In any case, I think posterior predictive checks are probably the best way to go here. What checks to perform depends on the problem, and only you know best what matters in this context.

Aside from all this, I would probably reframe this as a Bayesian decision problem. Rather than deploy the rep which the largest estimated probability, instead compute the expected loss of deploying rep $i$. It might be the case that some effects of some reps are quite large, but fairly uncertain. In such cases, the expected loss would be larger than a rep who has smaller yet more precisely estimated probability of conversion.