One or two-sided test for classifier accuracy? As far as I could reconstruct, caret::confusionMatrix uses a one-sided binomial test to compute the p-value of the accuracy being better than the "no information rate" (NIR). However, for computing the 95% confidence interval, it seems to use a two-sided test. Is this mixing of tests legitimate, and, if so, why? Wouldn't a one-sided confidence interval be more appropriate?
Below is my code to check my assumptions:
library(caret)
library(tidyverse)

# For the example from the reference paper,
# http://www.jstatsoft.org/article/view/v028i05/v28i05.pdf,
# Section 6: Characterizing performance (p. 15)
tb2 = tibble(
  true      = as.factor(c(rep('mutagen', 600), rep('nonmutagen', 483))),
  predicted = as.factor(c(rep('mutagen', 528), rep('nonmutagen', 453), rep('mutagen', 102)))
)
cm = confusionMatrix(tb2$predicted, tb2$true, positive = "mutagen", mode="everything")

print(cm)

# the p-value is internally computed as:
bt1 = binom.test(
  cm$table[1,1] + cm$table[2,2],
  sum(cm$table),
      p = sum(cm$table[,1]) / sum(cm$table),
  alternative = "greater"
)

print(bt1)
print(bt1$p.value == cm$overall[['AccuracyPValue']])

# ...but the confidence interval is computed using a two-sided test:
bt2 = binom.test(
  cm$table[1,1] + cm$table[2,2],
  sum(cm$table),
      p = sum(cm$table[,1]) / sum(cm$table),
  alternative = "two.sided"
)

print(bt2)
print(bt2$conf.int[1] == cm$overall[['AccuracyLower']])
print(bt2$conf.int[2] == cm$overall[['AccuracyUpper']])

Edit:
Consider the following confusion matrix:
Confusion Matrix and Statistics

          Reference
Prediction  0  1
         0  9  4
         1  3 16
                                          
               Accuracy : 0.7812          
                 95% CI : (0.6003, 0.9072)
    No Information Rate : 0.625           
    P-Value [Acc > NIR] : 0.04646         
                                      
[...]

The p-value is < 0.05, but the 95% CI includes the no-information-rate. These two measures send IMO conflicting messages.
Edit:
I could, of course, "manually" compute one-sided CI, using binom.test, as the code above does for the caret example data:
95 percent confidence interval:
 0.6281009 1.0000000

Is there some statistically valid reason why caret isn't doing it for me? In the sense of hypothesis testing, should I be guided by the caret's p-value (one-sided) or the CI (two-sided)? In the former case, I'd reject $H_0$ for my data above, and in the latter I'd fail to reject.
 A: Usually the claim complementary to the one you hope to establish is posed as the null, so you can say that the alternative is consistent with the data when you reject. With the one-sided p-value, you are presumably hoping to establish that your classifier is better. The one-sided null $H_0$ would be that Acc $\le$ NIR versus the one-sided alternative $H_a$ that it is better, Acc $>$ NIR.
The one-sided p-value is smaller than $5\%$, so you reject the null and say the data is consistent with the classifier being better. The interpretation is that if there was no difference between them, the probability of seeing an accuracy of $0.7812$ ($25$ correct in $32$ trials with $p=0.625$) is  1-binomial(32,24,.625) = binomialtail(32,25,0.625) = $4.6\%$, which is fairly unlikely.
I do agree that the one-sided CI would be better for this use case, so let me try to give guidance on how to calculate it to see if it is consistent with the p-value. You can get it by looking at a $90\%$ two-sided CI, because the overlap between two one-sided $95\%$ CIs makes one two-sided $90\%$ CI.
Doing just that gets you:
. cii proportions 32 25, level(90)

                                                         -- Binomial Exact --
    Variable |        Obs  Proportion    Std. Err.       [90% Conf. Interval]
-------------+---------------------------------------------------------------
             |         32      .78125    .0730792        .6281009    .8925531

You can also use the inverse of the right cumulative binomial directly to get the LB like this:
. display invbinomialtail(32, 25, .05)
.62810094

This means that the one-sided interval is $[0.6281, 1]$. This excludes $0.625$, so you would reject the null that the classifier is the same or worse. The one-sided CI and the one-sided p-value are consistent.
So why is there a two-sided CI presented as well? I don't really know why caret defaults to this for certain, but I suspect it is because people often want to know an informative upper bound too, since better than NIR is a not very impressive. People want a sense of how much better and the two-sided interval gives you that, at a price. It tells that true accuracy values between the upper bound and 1, like 0.95, are also inconsistent with the data you have, just like 0.55 was not supported by the data on the left. I expect this is the reason why a CI is reported rather than a p-value: it gives you a better sense of range. Had the true value been outside that interval, the confidence interval construction procedure would have resulted in an interval different than the one observed with probability 95% or greater.
You should use a one-sided superiority test if you want to test the hypothesis that your classifier is better and you don't care by how much, since you would continue to use it as long as it better regardless if it wins by an inch or a mile. But you should really use the test that corresponds to the question you want to answer. Only you know what that is at this point, and you should definitely not pick a test just because it yields significant results. There is no such thing as "in the sense of hypothesis testing". That's like saying "in the sense of a screwdriver", should I use a Phillips head or a flathead? The answer depends on the screw/analysis goal.
Why does caret not report the other one-sided p-value or CI? If you somehow made a classifier that was worse than NIR, you could just do the opposite of what it says, and you would have a better classifier.
To sum up, if you just want to know that your model is better than the most naive thing you can do (or some other baseline), a one-sided test/CI is good enough. If you want to spend your $5\%$ to also acquire more precise knowledge of the extent to which your classifier is better, then you should use the two-sided test. This will cost you some precision at the lower end of the interval. Software gives you both since those are the two most reasonable questions in the context of classifier performance.
