Let's say I'm trying to evaluate a classification algorithm and suppose there are $m$ data points in my test set. Here's my understanding so far: assuming my evaluation metric is the classification error, I can form a Bernoulli random variable $X$ with success probability $p$. We treat a correct classification as a "success". This means $p$, which is the population proportion of successes, can also be interpreted as "true classification accuracy" - i.e., if the ML model were made to run on an arbitrarily large hold-out set, then the proportion of correctly classified data points (successes) would be $p$.

I run the model once and can then treat the $i$-th data point as an independent realization of that random variable, say $x^{(i)}_1$, which equals $1$ if correctly classified, $0$ if not. The $1$ in the subscript denotes that this is the $i$-th classification result (out of $m$) in the first run of the model. I proceed to run the model $n$ times in total and $x^{(i)}_j$ denotes the realization of $X$ for the $i$-th data point in the $j$-th run of the model.

Thus for any $j$, $S_j=\{x^{(1)}_j,x^{(2)}_j,\ldots,x^{(m)}_j\}$ will represent a sample of size $m$, and since this is akin to sampling with replacement, $S_j$ will be iid for each $j$. The classification accuracy for a particular run of the model is just the sample mean for that run. Let's denote it be $\hat p_j=\frac{1}{m}\sum_{i=1}^mx^{(i)}_j$. Then the $\hat p_j$'s follow the sampling distribution of the sample means for size $m$ (the test statistic is the sample mean).

Since for $X$, $\mu=p$ and $\sigma^2=p(1-p)$, the sampling mean is again $p$ and the sampling variance is $\sigma^2/m=p(1-p)/m$.

First question: is my understanding above correct or are there mistakes?

Now how do I report my results? First I want to clarify the ideal case scenario: I'd run the model a lot of (say $n$) times, obtain different sample mean estimates $\hat p_j$ where $j=1,\ldots,n$, and obtain their mean $\bar p=\sum_{j=1}^n\hat p_j$. As $n\to \infty$, $\bar p\to p$: am I correct in saying this? If $n$ is reasonably large, say $n=30$, then $\bar p$ is a reliable estimate of $p$. How do I report the standard deviation of the $\hat p_j$'s? I don't know $p$ so I can't calculate $p(1-p)/m$. Should I use $\bar p(1-\bar p)/m$ instead? If so, why?

Let's say I run the model just once and get the mean of all $x^{(i)}$'s as $\hat p$. Again, should I just report the standard deviation as $\hat p(1-\hat p)/m$? If $m$ is "sufficiently large", by CLT the sampling distribution will approximate the normal distribution and I can write the confidence interval for $\hat p$ as plus or minus $1.96$ times the standard deviation, so it's important to know.

Thanks in advance!

  • $\begingroup$ What exactly do you mean by "running the model several times"? Training several models on new training data respectively and predicting the same test data? Or does the test data change as well? $\endgroup$ – cbeleites May 24 at 16:32
  • $\begingroup$ @cbeleites: The model remains the same - e.g. maybe I'm running linear regression each and every time. And yes, the training and test data change every time (for example, as in cross-validation). $\endgroup$ – Shirish Kulhari May 24 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.