7
$\begingroup$

Ever since Breiman, maximizing predictive accuracy has become a predictive modeling gold standard, of sorts. That it has evolved to this status is understandable: it can be "optimized," is easily calibrated across k-fold samples and, for the most part, is a consistent statistic for internal model validation. However, all too often it is the sole criterion for model value -- a mechanistic and rigidly applied metric -- despite its being prone to p-hacking, gaming, and analyst fraud as well as little or no thought being given to its wider business and/or strategic impact and meaning.

Is it time to re-evaluate this metric's relative importance in the statistical, machine learning and predictive modeling industries?

$\endgroup$
  • 5
    $\begingroup$ As Matthew Drury comments, the Mike Hunter answer above to the Mike Hunter question is "a nice and thoughtful post." But let's give credit where credit is due. Except for most of the second paragraph, this seems to be a word-for-word copy of a post by Thomas Ball on Linked-In a week or so ago, under the title of "Reconsidering predictive accuracy." And other than the first and last wrapper sentences, the 2nd paragraph is a word-for-word copy of part of my comment back to Thomas Ball. $\endgroup$ – user79936 Jun 16 '15 at 20:53
  • 4
    $\begingroup$ Doug, thank you for bringing this up: we respect the work of others on this site and insist on attribution.. I have found the original at linkedin.com/grp/post/…. $\endgroup$ – whuber Jun 16 '15 at 22:19
  • 6
    $\begingroup$ It doesn't work like that, Mike. If you want to be anonymous here--which is perfectly within your rights and many of us do it--then to maintain your credibility you have to behave like someone who is honest. It would have been simple and appropriate to provide a link to the LinkedIn site and summarize the thoughts posted there, especially because you had been warned about the site policy on attribution several days earlier. The issue now before us is since we cannot verify that you are indeed Thomas Ball, we still have to act as if you are not. Maybe you're pretending to be him? $\endgroup$ – whuber Jun 16 '15 at 22:34
  • 3
    $\begingroup$ We only ask that you play by the rules, Mike. If you do not, you will not be able to interact here. $\endgroup$ – whuber Jun 16 '15 at 22:49
  • 6
    $\begingroup$ Unacceptable (and flagged) comments deleted. Please take the opportunity to follow the help link @whuber has indicated on behavior and ponder it carefully. I note you've previously been warned about plagiarism; the standard response to your actions here (which doesn't only involve your own work even assuming your claim is correct) would have been to immediately suspend your account, rather than try to investigate whether there was any grounds for additional benefit of the doubt. I'd suggest you try options (2) and/or (3) above as likely to most quickly have a good eventual outcome for our site $\endgroup$ – Glen_b Jun 17 '15 at 4:19
7
$\begingroup$

This is a nice and thoughtful post, and in my working life I have observed the things you outline to be correct - the successful statisticians and scientists at my workplace are those that can step back from raw predictive accuracy and deliver a model or analysis that is holistically appropriate for the problem at hand. Sometimes this is raw predictive power, but often it is not.

I explicitly look for this when interviewing, my favorite initial answer to a modeling question is

Well, it depends...

I'll add some examples to your list.

Implementation Costs

Many businesses run core systems on outdated technology, cobol or fortran codebases running on ancient mainframe architectures. They are often reluctant to replace them because of the high fixed costs for doing so (even though the variable costs of maintaining them are high). This can have drastic consequences for model implementation. It may be possible to get a predictive boost from a random forest or gradient booster, but implementing a model of that complexity in a production environment can be completely infeasible.

Shelf Life

Related to implementation costs, a model, once implemented, may have a very long shelf life, and be expected to deliver reasonable predictions for a long time. A model with maximum supportable complexity fit very hard to the data is less robust to distribution shifts in the population and predictive relativity changes between segments.

Tinkering

Business people have a tendency to tinker with production models, and we as modelers sometimes have to assist with hot fixes in production systems. Complex models are more sensitive to this, it is harder to accurately asses how they will react to a production adjustment (talk to a mechanic about whether its easier to get under the hood of a car manufactured in 1980 vs. 2010).

Robustness to New Information

A categorical predictor may obtain new categories in the future, and is often desirable to have principled way to deal with these without refitting a model and pushing it to production.

Model Componentization

A model may be part of a larger system optimization, which imposes environmental constraints on its form and properties. One common source of this is when a model is a component of a larger mathematical optimization scheme, with some causal predictor in the model being manipulated as a lever to enhance business results. This can impose smoothness or differentiability constraints on the predictors that are very important to consider.

Locality Constraints

Some models have better locality properties than others. For example, if I wish to estimate the price elasticity of a customer for small adjustments, then a highly local model (i.e. a density smoother with small bandwidth, a regularized spline with small parameter, or a gradient tree booster with lots of cuts) will invariably use less of the data to support inferences even on a local scale. This can be undesirable when these inferences are used to make important decisions, and should be supported by as much data as possible.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.