I ran a MC simulation of $10^5$ GLM regressions (logistic, logit link) in R. To do so, I assumed:

  • The outcomes ($y$) were repeatedly sampled from a Bernoulli distribution ($N=1000$)
  • The one explanatory variable ($x$) was sampled independently from y from a half-normal distribution ($x≥0$)
  • I then calculated the accuracy of predictions (with cutoff 0.5) as true positive + true negatives over all N

Naïvely, I was perhaps expecting an mean/median accuracy of 0.5, but that wasn't true. The average accuracy was around 51.5%. Is there a good intuition or theoretical result for this?

  • 4
    $\begingroup$ Does the explanatory variable have nothing to do with the outcome (at least when you simulate it)? And more importantly, are you evaluating on the same data that you trained on (or do you have an extra 1000 samples for each simulated dataset on which you evaluate the performance)? $\endgroup$ – Björn Aug 27 at 12:38
  • 1
    $\begingroup$ Note that a logistic regression isn't strictly a classification scheme. It returns a probability of class membership, which is generally more important to evaluate than accuracy based on an assumption, often hidden, about the probability cutoff for class assignment. $\endgroup$ – EdM Aug 27 at 15:30
  • $\begingroup$ @Björn yes independent and new samples, have updated. At EdM cutoff was 0.5 because that seems the standard in finance/economics applications. I was trying to approximate a poorly performing model in a paper. $\endgroup$ – Rico Aug 27 at 15:55
  • 1
    $\begingroup$ @Rico A cutoff of $0.5$ might be standard in finance and economics, but please familiarize yourself with why this is misleading. Frank Harrell is a member on Cross Validated and the (former?) chairman of biostatistics at Vanderbilt University. He has two blog posts about this topic, about which is he quite passionate. fharrell.com/post/classification fharrell.com/post/class-damage Shamelessly, I will link to a post of mine on this topic, too. stats.stackexchange.com/questions/464636/… $\endgroup$ – Dave Aug 27 at 15:59
  • 1
    $\begingroup$ @Dave I get your point about proper scoring methods and maybe I should pick something the authors reported like the pseudo-$R^2$ -- but this deviation does not help my question. I'm trying to evaluate model fit in this null case and I have a feeling there's something curious going on with the mean result of > 0.5. $\endgroup$ – Rico Aug 27 at 16:26

You didn't write it, but I will assume that $y \sim \mathcal B(p= 0.5)$. This is not secondary: different class proportions lead to different accuracy scores.

Anyway, the fact is that the model uses the data it has ($x$) to get a fit as good as it can. Accuracy is not the objective function of logit model, and it's not granted to improve with model fit, but it mostly does. However, this is true only if you evaluate the accuracy on the training set: there is no way, whathever the model is, logit, tree or neural net, that predicting a totally independent Bernoully variable with p=0.5 yields an expected accuracy different from 0.5. On the training set instead, predictions are not independent on the observed $y$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.