Circular reasoning in Harrell BBR 18-19? I am looking at chapter 18 (Information Loss) of Harrell's "Biostatistics for Biomedical Reserach": https://hbiostat.org/doc/bbr.pdf.
The example of 18-19 seems like circular reasoning. He argues against classification accuracy by showing that the age + sex model has lower classification accuracy but a higher $\chi^2$ statistic than the age-only model, saying that this shows sex to contain negative information.
Why can't sex contain negative information? I certainly believe that it would be a predictor, but I do not see anything in the statistics that indicates so, unless we take c-index or $\chi^2$ as the performance metric.
So Harrell's argument seems like:

*

*Use c-index or $\chi^2$, not accuracy


*Accuracy makes it look like sex contains negative information, according to c-index and $\chi^2$


*So use c-index or $\chi^2$, not accuracy
That argument seems circular to me; we have to assume that c-index and $\chi^2$ are superior metrics to show the flaw in accuracy as a metric.
Where have I gone astray?
(Note that I am not asking why accuracy is an improper scoring rule. This is about why Harrell’s argument is not circular.)
 A: 
That argument seems circular to me; we have to assume that c-index and $\chi^2$ are superior metrics to show the flaw in accuracy as a metric

In your comments, you mention that you agree and understand why we should prefer c-index and likelihood ratio chi square (LHRCS) to classification accuracy.  In your post, you also mention that you understand that accuracy is not a proper scoring rule.  OK, that's well and good.
The argument as you've presented it is not circular. It is however a bad argument. "Use LHRCS because it is a better metric in this example".
However, this is not a faithful representation of the argument I think Frank is trying to make.  If I were to write out the arguments point by point then the argument may go something like:

*

*Proper scoring rules are preferred to improper scoring rules since by definition proper scoring rules are maximized by the true distribution


*The LHRCS is derived from the log likelihood, which in a probabilistic forecast is the binomial likelihood and hence a proper scoring rule.


*Accuracy is not a proper scoring rule (+ additional arguments against accuracy, like it can be maximized by guessing the most prevalent class all the time, which does not seem like a good property and also falsely overstates confidence by giving a 100% probability to an outcome).


*Given our understanding of proper scoring rules, the example demonstrates that using an improper scoring rule like accuracy will result in wrong decisions about what is important (I assume here that we know a priori that sex is predictive).
This line of reasoning does not require assuming LHRCS is a superior metric because we justify it with knowledge from proper scoring rules.
EDIT:  If your goal is to convince other people the LHRCS is superior, simulation is your friend.  Here, I simulate Frank's example 1000 times.
library(tidyverse)
library(rms)

r = rerun(1000,{
  
  N = 400
  age = round(rnorm(N))
  sex = rbinom(N, 1, 0.5)
  noise = rnorm(N)
  p = plogis(1.6*age + 0.5*sex)
  y = rbinom(N, 1, p)
  
  
  model_1 = lrm(y~age)
  model_2 = lrm(y~sex)
  model_3 = lrm(y~sex + age)
  model_4 = lrm(y~sex + age + noise)
  models = list(model_1, model_2, model_3, model_4)
  
  accs = map_dbl(models, ~{
    preds = as.integer(predict(.x)>0.5)
    Metrics::accuracy(y, preds)
  })
  
  aics = map_dbl(models, AIC)
  
  X1 = anova(model_1)['TOTAL','Chi-Square'] - anova(model_1)['TOTAL','d.f.']
  X2 = anova(model_2)['TOTAL','Chi-Square'] - anova(model_2)['TOTAL','d.f.']
  X3 = anova(model_3)['TOTAL','Chi-Square'] - anova(model_3)['TOTAL','d.f.']
  X4 = anova(model_4)['TOTAL','Chi-Square'] - anova(model_4)['TOTAL','d.f.']
  X = c(X1,X2,X3,X4)
  
  tibble(
    right_accs = which.max(accs)==3,
    right_xs = which.max(X)==3,
    right_aics = which.min(aics)==3
  )
  
}) %>% 
  map_dfr(~.x)


r %>% 
  summarise_all(mean)



I'm fairly confident Frank's chi-square in the example is a partial chisquare, so I've tried to use that in this example.  The true model has largest accuracy less than half the time and largest partial chi-sqaure a little more than half the time.  So clearly better although still not bulletproof.  The results change quite a lot with more data.  Even with 4000 observations (an order of magnitude more), the right model has largest accuracy about half the time but has largest chi square 8 times out of 10!
