I'm trying to come up with the solution for a recurrent problem in many different industries. I have an idea of what can be done, but I'm struggling to model mathematically. Here's an example:
Imagine a plumping services aggregator. Basically, it's an application that lists plumbers in several different locations, and allow the user to book a visit. One of main metrics to measure quality is the number of times the plumber don't go to the client's house. We count every time it happens, so we can have the No-Visit Rate (nvr = no-visits/bookings). The system also have the classical 5-star rating, so the customers may evaluate the service (when it's actually provided). There's a $10 penalty when the service provider have a No-Visit. The problem with that penalty is that some providers operate in many different locations, and the quality varies in different cities. While they perform great in NYC, they might have poor quality in San Francisco. The revenue they get from NYC might compensate the effects the penalties they suffer in other locations.
The goal is to reduce the number of No-Visits in the system and improve the quality of service. We have the rating for each individual provider, number of bookings and No-Visit for each provider. The idea is to create a score based on that data, so we can rank them in the search based on that, and drive traffic for the providers with best quality. It's a way to drive the competitiveness between the providers by quality, not only price.
My current idea is to get the No-Visits Rate average per location, check how 'far' each provider is from that average (distance_nvr = provider_no_visit_rate - location_no_visit_rate). That would give a low (negative) number, if some specific provider is performing much better than the average for that location. Only that value is not enough to create a score. A provider might have a great score, but only have a couple bookings, therefore is not very significative. To overcome this problem, we can calculate the 'weight' of bookings for that specific provider (weight = number_of_bookings/total_bookings_for_that_location).
If a provider have a high distance value and is responsible for the majority of the bookings for that location, they should have a lower score than a provider with the same distance value, but much lower relative number of bookings. How much weight each value should have in the calculation is my main struggle. For instance, imagine a location where a single provider have the majority of the bookings. In this case, it's very likely they will have a 'distance' close to zero. Some other providers might have a lower distance, but as they have a much lower number of reservations, they might have a lower score and won't have a chance to rank higher to compete. The idea is to give them a chance, and check their performance data to recalculate the score for the next month. I'm clueless on how to make it happen mathematically.
What would be the best way to calculate the score based on those two values and overcome that limitation?
In the future, I wish to include price and ratings to the model, but I'd like to keep things simple at first, and handle only the No-Visit Rate, but some ideas on how to include the other two are welcome too :)
What kind of improvements (or problems) do you see in this model? Some feedback on it (and some suggestions) would be very appreciated.