Timeline for My machine learning model has precision of 30%. Can this model be useful?
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 14, 2023 at 8:41 | comment | added | ttbek | @wmmwmm If the datasets are representative, sure, I don't disagree. I'm curious why you carve out reinforcement learning as the exception when they change. We do have some methods for modeling nonstationary data, see for example: people.stat.sc.edu/hitchcock/stat520ch5slides.pdf Just because the means change with time doesn't mean we strictly cannot model that change. I'm just not particularly seeing utility for reinforcement learning on this, unless perhaps you mean to update the model while it is live. Not impossible, but that has opened systems to manipulation in the past. | |
| Mar 11, 2023 at 12:14 | vote | accept | wmmwmm | ||
| Mar 11, 2023 at 12:14 | |||||
| Mar 11, 2023 at 11:50 | comment | added | wmmwmm | @ttbek, see my argument to Dave, if the datasets are representative one could do that imho. Otherwise modelling would per definition be useless because the dataset would always change ( except for reinforcement learning) | |
| Mar 11, 2023 at 11:48 | comment | added | wmmwmm | @Dave; the argument is basically the precision only can say something about the model, and not the underlying data. The outher group(number 1) says the data is representative for the population, and both training and test dataset are evenly distributed. | |
| Mar 8, 2023 at 10:32 | comment | added | ttbek | To rephrase, the default assumption should be that there is likely a fair amount of variation in the distribution. Yes, it's obnoxious, why can't they have gathered even larger data and included the entirety of what data we might see? Sometimes they even did gather almost everything out there... but the distribution drifts with time, welcome to non-stationary data hell. | |
| Mar 8, 2023 at 10:29 | comment | added | ttbek | Of course you can't expect a probability of 30%... unless you have good reason to believe that your test data will be distributed identically to your training data, rarely true in practice. "Unless there is reason to believe the new data are inherently different" -- Yes Dave, and the more complicated the ML model, the more likely it is to be relying on things that are likely to be different. I've seen tons of models that do great on training data (30% > heuristic), pass validation (say 25% better), and fall apart on the real new data which is gathered independently (2% worse vs. heurestic). | |
| Mar 7, 2023 at 22:10 | comment | added | Dave | @wmmwmm What justification do they give for that? Precision is literally the probability of a case belonging to the predicted category ($P(Y=1\vert \hat Y=1)$). Unless there is reason to believe the new data are inherently different (could be, but that’s a very different discussion), that group appears to reject the standard definition of precision in machine learning. | |
| Mar 7, 2023 at 21:35 | comment | added | wmmwmm | Exactly! I also introduced costs of FN and FP in my presentation. But group 2 stays unconvinced. They keep claiming "you just can't expect a probability of 30% the model will find new cases in new data" | |
| Mar 7, 2023 at 21:03 | history | answered | Thomas Lumley | CC BY-SA 4.0 |