2
$\begingroup$

In all the models I've built, my prediction accuracy has always been around 75%. Around 80% is my max. In examples online, I have never seen anything above 85%. My understanding is that only more complex machine learning algorithms such as neural networks or random forests can help you predict with a very high level of accuracy.

Is it possible to get in the 90% range, and in what cases/situations does this occur?

$\endgroup$
  • 2
    $\begingroup$ It depends on your data. If your data is cleanly split by a level in one predictor, linear regression can detect it and get 100% accuracy. If there is a zone with mixed classes, no model can get 100% accuracy. $\endgroup$ – Pere Jul 19 '17 at 19:19
  • 1
    $\begingroup$ Logistic regression's job is not to provide accurate class assignments, it is to provide accurate class probabilities. If you believe the world, or at least our knowledge of the world, is probabilistic, then you should be wary of the quest for 90% accuracy. $\endgroup$ – Matthew Drury Jul 19 '17 at 19:31
  • $\begingroup$ @Pere, it also depends a lot on the dataset, correct? For example if a dataset is 95% yes and 0% no, you could simply predict only yes, and your classifier would be, surprise!, about 95% correct. $\endgroup$ – tonychen Jul 19 '17 at 19:46
2
$\begingroup$

Unless you describe your application in much more detail (what is the objective, what are the inputs, what are the output classes, how the study was designed, etc.)., it's hard to give you an answer.

A possible issue may be that the modeling hypotheses of logistic regression are not satisfied: maybe the log-odds of class A, say, are not a linear function of predictors (features). Said in another way, logistic regression classifiers are linear classifiers (the classification boundary is an hyperplane). Instead, SVMs or neural network classifiers are non-linear: the classification boundary can be any manifold, it doesn't have to be an hyperplane. To have an idea of whether the classification boundary is linear or not, you may proceed this way:

  • assuming you have features $X_1,\dots,X_p$, make scatterplots of $X_i$ vs $X_j$, colored by class (so, red dots are class A, and blue dots are class B, for example)
  • 2D projections of an hyperplane are lines. Thus, in the scatterplots you should see that class A and B are approximately separated by a linear boundary. If this isn't the case, then you could consider using nonlinear classifiers.
| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

There are many possible responses to your question. First, predictive accuracy is an easily fudged and manipulated metric. This is especially true when it is the only objective, goal or outcome that is under measure. Stepping out of purely statistical considerations, it is well known that otherwise objective metrics can and will become biased and unreliable in politically charged environments where monetary outcomes are dependent on their results. So, if an analyst is being evaluated and/or compensated based on the accuracy of his models, unscrupulous practitioners may elect to employ practices that boost results. These unscrupulous practices are virtually undetectable, even by his manager(s), unless they sit down side-by-side with him and, in a kind of forensic audit, rigorously groom the code driving the model(s).

Next and expanding @Pere's comment, accuracy is a function of the information used. Borrowing old rules of thumb taken from applied modeling concerned with expected R-square values for predictive (or descriptive) models, in survey-based research, it used to be that R-squares ranging between 10% to 20% (or so) were expected. With financial, business and/or corporate-related information and models, somewhat higher R-squares in the 40%-60% range were the norm. At the low end of the range, such as response models developed in direct marketing applications, R-squares of 5% or less, in some cases even approaching zero, were expected. On the high end (with one exception that I'm aware of), R-squares much more than 70%-80% could be expected to be the result of overfitting or a violation of model assumption(s). In other words, extremely high R-squares were highly likely to be erroneous or, worse, fraudulent. The one exception to this generality were time series models developed for evaluating or forecasting retail products where the full set of so-called causal information is available. Here, I'm thinking of marketing mix models built, e.g., from Nielsen market-level scanner data for FMCGs (fast-moving consumer goods or products, like ketchup, potato chips or soda). With a full set of marketing instruments (price, promotion, distribution, advertising, etc.), R-squares that are 90% or more, even up to 99%, are not uncommon. However, the more disaggregate the unit of analysis in a marketing mix model becomes -- e.g., going from market-level, to store-level, to household-level information -- the lower one's expectations have to become regarding expected R-square.

Of course, I'm suggesting that these hoary rules of thumb wrt R-squares are applicable by analogy to expectations concerning predictive accuracy.

Note that direct marketing was one of the first applied industries where logistic regression-based response modeling saw wide use. With typically quite small response rates (frequently 2%-3% or less), finding signals in such noisy data was based on deciling the predictions from the model and sending out mailings, subscriptions, offers, whatever, to the top 2 or 3 deciles. This methodology usually ensured that those most likely to respond were targeted, resulting in profitable marketing activities. The irony in the wide use of logistic regression in DM is that, as is well known, the logistic function does not fit the tails of its distribution at all well, particularly in the presence of low incidence and/or rare events. See Gary King's papers on rare event modeling for a fuller discussion of this issue, e.g., here ... https://gking.harvard.edu/relogit.

Another issue with predictive accuracy concerns the widespread use of dichotomizing or discretizing otherwise continuously scaled features and information. Many are the statisticians who have written about the fallacies and problems to be associated with these widely employed heuristics. The bottom line is the potentially massive loss of valuable information as a function of these practices that have the unsurprising and ironic effect of blunting or dulling predictive accuracy. For instance, Andrew Gelman recently discussed the truly fundamental and, for the most part, ignored issues concerned with feature measurement, reliability and validity (see here ... http://andrewgelman.com/2017/07/17/continue-not-trust-turk-another-reminder-importance-measurement/). One of the comments on his blog references a biostatistics paper which excoriates this practice of discretizing continuously distributed data (see here ... Senn, Stephen and Steven Julious, 2009, Measurement in clinical trials: A neglected issue for statisticians, Statistics in Medicine 28:3189-3209). This paper references a book by David Hand, Measurement Theory and Practice: The World Through Quantification, which should be required reading for all statisticians and data scientists. Echoing Gelman, Senn, et al, and Hand is Frank Harrell who has written extremely cogent critiques of the errors and fallacies inherent in blindly discretizing data, e.g., see Dichotomania here ... http://www.fharrell.com/#catg.

To sum this all up, expectations regarding predictive accuracy are a function of many factors including its importance as a unit of measure, the tactical objectives of the model, the information under analysis, feature measurement, the use of discretizing heuristics, and more.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.