Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Say I have a model $y=f_n(x_1,x_2,x_3)$.

Here say $y$ is categorical and binomial response. i.e. $y$ can be only 0 or 1. Data shows 87% 1 and 13% 0 values.

I fit a multinomial logit on a test dataset to try to predict $y$. Validation against a validation dataset showed 70% success. (Success meaning Model prediction matches $y$ in the validation set).

Normally I'd be happy with the 70% success rate. But I'm a bit confused because: Say, I had a blackbox model that always answered "$y=1$" no matter what $x_1$, $x_2$ and $x_3$ are. Wouldn't this model anyways achieve 87% success?

What's the point behind my multinomial logit unless I can beat 87%?

What gives? Am I making a blunder?

share|improve this question
add comment

4 Answers

up vote 3 down vote accepted

You are right that performance of trivial models (always predicting 1, or always 0 or uniformly random guessing) are important benchmarks. However, there may be slightly different trivial benchmark models that reflect better the particular way how the model was built.

You should follow Wayne's advise to break down the successes and failures for the two classes to find out what is going on.

However, yo[u also need to be aware of the uncertainty on such performance estimates. Search here for "proportion confidence interval".

Here are some ideas what else could have happened

  • Of course, your model may truly be worse than the trivial model.
  • Some models train for equal prior probabilities, i.e. regardless of the relative frequency in the training data, the model adjusts to be optimal for equal relative frequencies for the different classes.
    If that is the case, from the model's point of view, the trivial models would all have 50% correct predicitions, and the model is better.
  • Some models discard "duplicates" in the training input.
  • You say that "the data" has 87% of cases of one class. Does that apply to the training data or to the test data? (Kind of the opposite situation from the point before)
  • Some classification problems are ill-posed (typically a "negative" class has only the definition that it is not the "defined" class. For example, having a particular disease vs. not having that disease, but possibly any other disease). In that case, the worse than trivial benchmark performance may occur if certain kinds of samples are not correctly classified.
  • (probably not relevant here) Other kinds of adjustment with the relative frequencies (e.g. one class is known to be difficult to recognize) can lead to all kinds of performance compared to the trivial benchmarks.
  • (not relevant here) in multiclass classification, particular classes can be much more difficult to recognize than others, so worse than guessing/trivial prediction can happen.
  • ... many more possibilities...
share|improve this answer
    
Thanks @cbeleites. I used R's glm with default parameters. I will check to see what it does about duplicates. Although not all predictors were categorical variables. With continuous numeric variables in there it will be hard to have a strict duplicate. –  curious_cat Feb 20 '13 at 21:33
1  
I just happened to read a post by @gung on another thread that said "classification accuracy is maximized if the threshold is equal to the proportion of 1's in the population". Fortuitously, it so happens I was already using a threshold of mean(predictors) which happens to come remarkably close to the proportion (i.e. 87%). Maybe this is always true. –  curious_cat Feb 20 '13 at 21:36
add comment

When you say "70% accurate" and "87% accurate", you mean "what percentage of validation inputs were correctly classified". But that's not quite the same as "fit for a purpose". Your naive-classifier proposal (call everything 1) is a great illustration of the need for further distinction.

In your case, there are four categories into which each classification your from regression falls: True 1's, False 1's, True 0's, and False 0's. These are the four quadrants of a table you could make where the rows are the actual classes, and the columns are what your regression predicts. Fill them in then look at your results in light of what you actually care about, and what costs are associated with being wrong.

Your example applies when you don't care more about any of these categories more than others. In that case, being 87% right is better than being 70% right. But perhaps your regression identifies 0's 100% correctly (no False 1's) and mis-identifies 1's 30% of the time (30% False 0's). If you care a lot about 0's -- say they represent a non-employee getting into a secure part of your facility -- and you want to avoid False 1's, your regression is better than the naive "call everything 1" strategy, even though it's "less accurate" overall.

(Of course, if False 1's are very expensive while False 0's are trivial, the naive strategy of calling everything 0 may be appealing, even though it's only 13% "accurate".)

share|improve this answer
    
Thanks. So, what you are saying is separating the penalty for a False Positive versus a False Negative? I will think about that. In general though, what are typical ways to rate the goodness of fit of a Binomial logistic regression model? –  curious_cat Feb 20 '13 at 21:10
    
I'm not an expert on logistic regression, so I don't know typical goodness-of-fit methods. Just saying that when you say "70% accuracy" and "83% accuracy", there may be more to the story than that. But of course, it depends on the purpose of your regression. –  Wayne Feb 20 '13 at 21:15
    
Interestingly if I use a Naiive Bayes classifier it gets a 85% overall success rate. –  curious_cat Feb 21 '13 at 5:59
    
In the case of naive Bayes, I'd want to see the true/false positive/negative rates. Various classification methods are influenced by the ratio of classes in the training set, and I think naive Bayes may be one of them. –  Wayne Feb 21 '13 at 14:16
    
Right @Wayne! Bayes improved success but gets too many false negatives. –  curious_cat Feb 21 '13 at 20:11
show 1 more comment

Logistic regression doesn't attempt to maximize classification accuracy.

The problem of trying to find coefficients that maximize classification accuracy is actually NP-hard, so in general it's not possible. Instead of maximizing accuracy, logistic regression finds coefficients that minimize the cross-entropy cost, which is convex and hence easy to minimize, and sort of approximates classification accuracy. In particular, if $t_i$ is the actual class label of instance i (0 or 1) and $y_i$ is the predicted probability of being in class 1, logistic regression will find coefficients that minimize

$-\sum_0^n [t_i * ln(y_i) + (1-t_i)*ln(1-y_i)]$

If you plug in the "always class 1" solution (i.e. $y_i = 1$ for all $i$), you get a $ln(0)$ in the second term, and a cost of infinity, so it will never choose that answer. Instead you'll get a solution that always gives some positive probability to every class that it has seen in the training data.

share|improve this answer
    
Thanks @Marc. I did not realize this. –  curious_cat Feb 21 '13 at 4:27
    
Thanks @Marc. I did not realize this. I guess my question then is in domains where cross entropy cost is not a good surrogate for classification accuracy what wins? i.e. What's the criterion to choose the better model when the two metrics indicate a different winner? Any real world examples where one or the other is the better choice of decision metric? –  curious_cat Feb 21 '13 at 4:33
    
Maximizing classification accuracy is NP-hard, so there are no general algorithms that can find a solution that maximizes classification accuracy, except in special cases or when the data set is small. So for most problems, it's just not an option. –  Marc Shivers Feb 22 '13 at 13:34
add comment

Just think the opposite that you want to estimate "0"s but your model always gives "1"s and then you are 100% failed. you can also deduce this result from a confusion matrix as described above.

share|improve this answer
2  
Perhaps you could define what you mean by confusion matrix here? If the OP were to calculate the class sensitivities of the model, i.e. aim to have balance scores for class 0 and class 1 sensitivities –  BGreene Mar 4 '13 at 16:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.