2
$\begingroup$

I've been having an argument with a friend of mine, and it's very possible I'm wrong here.

We are performing binary logistic regression on a dataset with 10000 observations, classifying action as "good" or bad". There are two independent variables (x1, x2), and class variable (y, with values "good" or "bad"). In this dataset, we have 7,500 observations classified as "bad" and 2,500 classified as "good". This is because there are several different ways for a user to perform a "bad" action, but only one way for them to perform a "good" action.

We are doing our analysis in R using the glm() function.

We create training data by randomly sampling 7,500 observations from the dataset, and create test data from the other 2,500 observations. we then build a model using binary logistic regression on the training data, then test it on the test data. The accuracy of our model is 75%.

Can we say our model is better than guessing?

He says that this model is no better than guessing. Even though the error rate is better than 50%, because the original data had a prevalence of "bad" classifiers, we would need our model to predict better than 75% in order to say it performs better than random guessing.

I disagree...but I can't defend my point with anything other than "that doesn't seem right". Can someone shed some light on the correct interpretation, and the reason for it?

$\endgroup$
  • 1
    $\begingroup$ You must clarify what do you mean by "random guessing": Do you mean $A$) "I will flip a fair (50%-50%) coin in order to decide whether I will predict that an observation is "good" or "bad"" or $B$) "I will take the sample proportions are adequate estimates of the underlying probabilities, and so I will construct a coin where the "bad" side will have a 75% probability and the "good" side will have 25% probability of landing, and then start flipping this coin"? $\endgroup$ – Alecos Papadopoulos Apr 28 '14 at 1:31
  • $\begingroup$ option B). Will edit in question. $\endgroup$ – Mike Apr 28 '14 at 1:35
  • $\begingroup$ And are you certain that your friend has also option $B$) in mind,and not $A$)? Because in many cases, when people say "random" they mean "equiprobable". $\endgroup$ – Alecos Papadopoulos Apr 28 '14 at 1:37
  • $\begingroup$ yes (sort of). He's asserting that since we have a 75% chance of drawing a "bad" observation out of the bag of 10,000, our classifier must do better than this in order to be considered more accurate than random selection. $\endgroup$ – Mike Apr 28 '14 at 1:39
1
$\begingroup$

You are right. I suspect that the conceptual mistake of your friend is that he visualizes the $75$%-$25$% "random guessing" as an "agnostic draw from the bag" and compares this with the "model accuracy". But this is not what "model accuracy" measures. "Model accuracy" quantifies a two-step procedure where first you predict and then you draw to see whether your prediction was correct. Without the model, you would have only the unconditional probabilities to go on (accepting that sample proportions are close estimates of them). So, as you clarified in the comments, you would flip a $75$%-$25$% coin to make your prediction. So the "accuracy" you should expect on average in this situation would be

$$\Pr [\{\text{"coin lands bad"}\},\{\text{"draw gives bad"}\}] + \Pr [\{\text{"coin lands good"}\},\{\text{"draw gives good"}\}]$$

These are independent events so the above joint probabilities split: $$\Pr [\{\text{"coin lands bad"}\}]\cdot\Pr [\{\text{"draw gives bad"}\}] \\+ \Pr [\{\text{"coin lands good"}\}]\cdot\Pr [\{\text{"draw gives good"}\}]$$

$$ = (0.75)^2 + (0.25)^2 = 0.5625 + 0.0625 = 0.625$$

You are telling us that, using the model, you have

$$\Pr [\{\text{"model says bad"}\},\{\text{"observation is bad"}\}] + \Pr [\{\text{"model says good"}\},\{\text{"observation is good"}\}] = 0.75$$

So your model improved your chances of predicting correctly. Note that the model does not "draw" from the bag of $Y$'s but from the bag of $X$'s. The model does not use the fact that you have a $75$% probability of drawing $X'$s that are related to a "bad" $Y$, -it uses the numerical values of the $X$'s.

P.S.
Greg Snow's answer correctly points out that your model does not perform better compared to a prediction strategy "always guess bad". But this strategy cannot be considered "random". Still, this remark would be a valid criticism against the practical usefulness of your model in general (and not only compared to "random guessing").

$\endgroup$
  • $\begingroup$ So, conceptually, I am right (and thank you for the thorough explanation). From a practical standpoint, however, our model still needs significant improvement in order to provide any kind of informed insight that couldn't be achieved by a much more simplistic strategy. $\endgroup$ – Mike Apr 28 '14 at 3:53
  • $\begingroup$ That's an exact description of your situation. $\endgroup$ – Alecos Papadopoulos Apr 28 '14 at 9:18
1
$\begingroup$

Well you are both right and you are both wrong. A big part depends on what you mean by "guessing".

If by guessing you mean flip a coin and therefor predict good/bad each with 50% probability independent of the actual value then you would predict correctly 50% of the time. Since you predict the correct value 75% of the time that means that yes your are doing better than guessing.

But, by guessing you could also mean predict bad every time (guess the most likely outcome ignoring the x variables) then you would be correct 75% of the time, since your model does not improve on this, it is not better than guessing (for this definition of guessing).

What this really shows is that percent predicted correctly is not really a very good metric for logistic regression. There are other metrics that are better, if you use the lrm function from the rms package (by the way, there is no glm function in MASS) it will compute some better measures (the C statistic for example).

Do you find value in the actual predicted probabilities? This is sometimes the advantage of a logistic regression model over just choosing a classification. If you only care about the prediction class and you use 0.5 as the cutoff then that means that you will treat a prediction of 0.01 and 0.49 exactly the same as each other, but you will treat a prediction of 0.51 differently than 0.49, but the same as 0.99. Does knowing that one subject has a prediction of 0.1 and another 0.45 (even though they would both be the same predicted class) gives us additional information.

$\endgroup$
  • $\begingroup$ augh, you're right about MASS. will edit. $\endgroup$ – Mike Apr 28 '14 at 1:40
  • $\begingroup$ in this example, we only care about the .49/.51 cutoff. we are not interested in the actual predicted probabilities. because we will always have more "bad" than "good" observations in our data, we will always have a higher probability of bad than good observations. $\endgroup$ – Mike Apr 28 '14 at 1:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.