# Statistically evaluating classification accuracy of machine learning model

Let's say I'm trying to evaluate a classification algorithm and suppose there are $$m$$ data points in my test set. Here's my understanding so far: assuming my evaluation metric is the classification error, I can form a Bernoulli random variable $$X$$ with success probability $$p$$. We treat a correct classification as a "success". This means $$p$$, which is the population proportion of successes, can also be interpreted as "true classification accuracy" - i.e., if the ML model were made to run on an arbitrarily large hold-out set, then the proportion of correctly classified data points (successes) would be $$p$$.

I run the model once and can then treat the $$i$$-th data point as an independent realization of that random variable, say $$x^{(i)}_1$$, which equals $$1$$ if correctly classified, $$0$$ if not. The $$1$$ in the subscript denotes that this is the $$i$$-th classification result (out of $$m$$) in the first run of the model. I proceed to run the model $$n$$ times in total and $$x^{(i)}_j$$ denotes the realization of $$X$$ for the $$i$$-th data point in the $$j$$-th run of the model.

Thus for any $$j$$, $$S_j=\{x^{(1)}_j,x^{(2)}_j,\ldots,x^{(m)}_j\}$$ will represent a sample of size $$m$$, and since this is akin to sampling with replacement, $$S_j$$ will be iid for each $$j$$. The classification accuracy for a particular run of the model is just the sample mean for that run. Let's denote it be $$\hat p_j=\frac{1}{m}\sum_{i=1}^mx^{(i)}_j$$. Then the $$\hat p_j$$'s follow the sampling distribution of the sample means for size $$m$$ (the test statistic is the sample mean).

Since for $$X$$, $$\mu=p$$ and $$\sigma^2=p(1-p)$$, the sampling mean is again $$p$$ and the sampling variance is $$\sigma^2/m=p(1-p)/m$$.

First question: is my understanding above correct or are there mistakes?

Now how do I report my results? First I want to clarify the ideal case scenario: I'd run the model a lot of (say $$n$$) times, obtain different sample mean estimates $$\hat p_j$$ where $$j=1,\ldots,n$$, and obtain their mean $$\bar p=\sum_{j=1}^n\hat p_j$$. As $$n\to \infty$$, $$\bar p\to p$$: am I correct in saying this? If $$n$$ is reasonably large, say $$n=30$$, then $$\bar p$$ is a reliable estimate of $$p$$. How do I report the standard deviation of the $$\hat p_j$$'s? I don't know $$p$$ so I can't calculate $$p(1-p)/m$$. Should I use $$\bar p(1-\bar p)/m$$ instead? If so, why?

Let's say I run the model just once and get the mean of all $$x^{(i)}$$'s as $$\hat p$$. Again, should I just report the standard deviation as $$\hat p(1-\hat p)/m$$? If $$m$$ is "sufficiently large", by CLT the sampling distribution will approximate the normal distribution and I can write the confidence interval for $$\hat p$$ as plus or minus $$1.96$$ times the standard deviation, so it's important to know.