Timeline for My machine learning model has precision of 30%. Can this model be useful?
Current License: CC BY-SA 4.0
6 events
when toggle format | what | by | license | comment | |
---|---|---|---|---|---|
Mar 11, 2023 at 12:30 | comment | added | Stephan Kolassa | That is definitely one possibility. I would also suggest you do not limit your decision to the output of the model alone - if there is an "important" site (important customer, big installation) with a lower predicted break down probability, then it might be worthwhile to inspect that rather than a "less important" site with a lower predicted probability. Similar thoughts could apply once you look at costs of inspections, e.g., in terms of distance - maybe you can inspect five installations in a day if they are close together, rather than three that are far apart. | |
Mar 11, 2023 at 12:14 | vote | accept | wmmwmm | ||
Mar 11, 2023 at 12:04 | comment | added | wmmwmm | Thanks fot the links Stephan. If i understand correctly, probabilistic classifications would label or 'rate' a future inspection. The idea is also to divide all possible visits in 2 groups where 1 group will still be picked randomly, and the other group by the model. So e.g. if 500 inspections per year are possible, simply let the model pick 250 and the other 250 are still randomly chosen. This also provides a nice benchmark group to see model performance, and still one is able to pick up changes in the environment from the random group | |
Mar 8, 2023 at 12:27 | comment | added | Stephan Kolassa | @Dave: I would argue that precision is meaningless. Whether a given probabilistic model adds any value will depend on the alternative. If we have budget and resources to do 500 inspections, we can either do them at random, or as guided by a model (e.g., by picking the 500 highest predicted probabilities), or by some other rule (e.g., inspecting those installations where the last inspection was the longest time ago). If we then have a handle on the cost structure (!), we can say whether the decisions (!) made in either of these ways are better than others. | |
Mar 8, 2023 at 12:03 | comment | added | Dave | I do agree that there would be value gained from looking at the probabilistic output of the model and letting those plus the costs of wrong decisions guide how to proceed. However, this seems just to delay addressing the question: if, after considering the probabilities of event outcomes and the costs associated with misclassification, the optimal classification scheme only gives $30\%$ precision, is that enough? | |
Mar 8, 2023 at 10:38 | history | answered | Stephan Kolassa | CC BY-SA 4.0 |