Success of a logistic regression model

Question

Say I have a model $y=f_n(x_1,x_2,x_3)$.

Here say $y$ is categorical and binomial response. i.e. $y$ can be only 0 or 1. Data shows 87% 1 and 13% 0 values.

I fit a multinomial logit on a test dataset to try to predict $y$. Validation against a validation dataset showed 70% success. (Success meaning Model prediction matches $y$ in the validation set).

Normally I'd be happy with the 70% success rate. But I'm a bit confused because: Say, I had a blackbox model that always answered "$y=1$" no matter what $x_1$, $x_2$ and $x_3$ are. Wouldn't this model anyways achieve 87% success?

What's the point behind my multinomial logit unless I can beat 87%?

What gives? Am I making a blunder?

Answer 1

You are right that performance of trivial models (always predicting 1, or always 0 or uniformly random guessing) are important benchmarks. However, there may be slightly different trivial benchmark models that reflect better the particular way how the model was built.

You should follow Wayne's advise to break down the successes and failures for the two classes to find out what is going on.

However, yo[u also need to be aware of the uncertainty on such performance estimates. Search here for "proportion confidence interval".

Here are some ideas what else could have happened

• Of course, your model may truly be worse than the trivial model.
• Some models train for equal prior probabilities, i.e. regardless of the relative frequency in the training data, the model adjusts to be optimal for equal relative frequencies for the different classes.
  If that is the case, from the model's point of view, the trivial models would all have 50% correct predicitions, and the model is better.
• Some models discard "duplicates" in the training input.
• You say that "the data" has 87% of cases of one class. Does that apply to the training data or to the test data? (Kind of the opposite situation from the point before)
• Some classification problems are ill-posed (typically a "negative" class has only the definition that it is not the "defined" class. For example, having a particular disease vs. not having that disease, but possibly any other disease). In that case, the worse than trivial benchmark performance may occur if certain kinds of samples are not correctly classified.
• (probably not relevant here) Other kinds of adjustment with the relative frequencies (e.g. one class is known to be difficult to recognize) can lead to all kinds of performance compared to the trivial benchmarks.
• (not relevant here) in multiclass classification, particular classes can be much more difficult to recognize than others, so worse than guessing/trivial prediction can happen.
• ... many more possibilities...

Answer 2

When you say "70% accurate" and "87% accurate", you mean "what percentage of validation inputs were correctly classified". But that's not quite the same as "fit for a purpose". Your naive-classifier proposal (call everything 1) is a great illustration of the need for further distinction.

In your case, there are four categories into which each classification your from regression falls: True 1's, False 1's, True 0's, and False 0's. These are the four quadrants of a table you could make where the rows are the actual classes, and the columns are what your regression predicts. Fill them in then look at your results in light of what you actually care about, and what costs are associated with being wrong.

Your example applies when you don't care more about any of these categories more than others. In that case, being 87% right is better than being 70% right. But perhaps your regression identifies 0's 100% correctly (no False 1's) and mis-identifies 1's 30% of the time (30% False 0's). If you care a lot about 0's -- say they represent a non-employee getting into a secure part of your facility -- and you want to avoid False 1's, your regression is better than the naive "call everything 1" strategy, even though it's "less accurate" overall.

(Of course, if False 1's are very expensive while False 0's are trivial, the naive strategy of calling everything 0 may be appealing, even though it's only 13% "accurate".)

Answer 3

Logistic regression doesn't attempt to maximize classification accuracy.

The problem of trying to find coefficients that maximize classification accuracy is actually NP-hard, so in general it's not possible. Instead of maximizing accuracy, logistic regression finds coefficients that minimize the cross-entropy cost, which is convex and hence easy to minimize, and sort of approximates classification accuracy. In particular, if $t_i$ is the actual class label of instance i (0 or 1) and $y_i$ is the predicted probability of being in class 1, logistic regression will find coefficients that minimize

$-\sum_0^n [t_i * ln(y_i) + (1-t_i)*ln(1-y_i)]$

If you plug in the "always class 1" solution (i.e. $y_i = 1$ for all $i$), you get a $ln(0)$ in the second term, and a cost of infinity, so it will never choose that answer. Instead you'll get a solution that always gives some positive probability to every class that it has seen in the training data.

Answer 4

Just think the opposite that you want to estimate "0"s but your model always gives "1"s and then you are 100% failed. you can also deduce this result from a confusion matrix as described above.