Sample size for the evaluation of Deep Learning Models

Question

I'm evaluating the performance and accuracy in detecting objects for my data set using three deep learning algorithms. In total there are 24,085 images. I measure the performance in terms of time taken to detect the objects. To measure the accuracy, I manually count the number of objects in each image and then calculate recall and precision values for three algorithms.

However, since I'm manually counting to get actual object count, I selected only 30 images. Will that sample be enough to make a conclusion that algorithm 1 is better than others in terms performance and accuracy?

Answer 1

For counts presumably you'd want to look at something like Poisson loss or - if you want mean-absolute-error or mean-squared-error of square-root transformed counts or... If it's more about the locations, then you want some kind of definition of what determines whether you have detected something at a certain location (e.g. with bounding boxes you can use accuracy, recall, precision etc. at some IOU level, but you still need to determine whether you want to calculate first a metric per image and then average so that a single image with many objects does not get too much weight).

How many images you then need depends a bit on how many images have how many objects on them and what kind of mistakes you expect your algorithm to make. However, one easy experiment to get a feel for this is to make up some hypothetical performance numbers for a certain number of images (e.g. for 30 images make up the 30 performances per image for each of your algorithms). This might look like this:

Image	Metric for algorithm 1	Metric for algorithm 2	Metric for algorithm 3
1	1.0	0.0	1.0
2	0.5	0.5	1.0
3	0.75	0.25	0.75
...	...	...	...
30	0.9	0.1	0.8

You can then repeatedly (let's say 1000 times) draw 30 rows from this table with replacement. For each of the 1000 repetitions, you calculate the difference between algorithms for your performance metric of interest. Then you have a look at how much that varies (i.e. the bootstrap distribution) and you have an idea of how sure you could be with that many annotated images.

Of course, if you have a bit of a budget, there are online services where you can get people you pay annotate images for you. Or if you have a bunch of friends/colleagues or just more time yourself, then there are good (and free) annotations tools that speed up the process and partially automate it (e.g. you might only have to quickly draw a bounding box with your mouse for each object and hit a key for next image, everything else done automatically).

Answer 2

However, since I'm manually counting to get actual object count, I selected only 30 images.

30 is one of the two conditions of the Central Limit Theorem for a sample mean. If you sample more than 30 cases and there are no extreme outliers, the sampling distribution of the mean is expected to be nearly normal. In the following analysis, we assume that the frequentist methods would be satisfactory.

In your case, I think you can transform the numerical outcome into a categorical binary variable: Correct or Incorrect. If your model detects all the relevant objects in an image correctly you label it as Correct, otherwise Incorrect. In this scenario, you need about 1068 cases with a 95% confidence level and margin of error smaller than 0.03 according to power analysis. That is, the sample proportion is within ±0.03 of the actual proportion in a 95% confidence interval. And here is the derivation(to take the population size into consideration please refer to this article):

$z^{*}\sqrt{\frac{p(1-p)}{n}}<0.03$

For a 95% confidence interval, the z score $z^{*}$ is 1.96, and we obtain:

$1.96 \times \sqrt{\frac{p(1-p)}{n}}<0.03$

We rearrange the inequality to this form:

$n> p(1-p)(\frac{1.96}{0.03})^2$

If we take p as the worse case value 0.5 to make the right side the largest we get this:

$n> .5(1-.5)(\frac{1.96}{0.03})^2=1067.11$

Will that sample be enough to make a conclusion that algorithm 1 is better than others in terms performance and accuracy?

Since you would like to compare two proportions, let's work out how to check the hypothesis that the difference between the two proportions is 0.03. That's is if the model is better by 3% than the baseline. And again we set the confidence interval as 95%, meaning our z score is 1.96.

Suppose that the proportion for your model is $p_1$ and the baseline proportion is $p_2$ and the cases for the two models are not paired.

$\frac{p_1 - p_2 - .03}{\sqrt {\frac{p_1 (1- p_1)}{n}+\frac{p_2 (1- p_2)}{n}}} > 1.96$

After some arrangements we get this:

$n > \frac{1.96^2 \times (p_1 + p_2 -P_1^2 - p_2^2)}{(p_1-p_2-.03)^2}$

We can see from the graph that the maximum of the right side is positive infinity, then let's guess that $p_1$ is .95 and $p_2$ is .9, and we get

$n > \frac{1.96^2 \times (.95 + .9 -.95^2 - .9^2)}{(.95-.9-.03)^2} = 1320.55$

And the minimum sample size is 1321 for each model.

The above is just for illustration and the real answer would be more complicated and you can refer to this book: Sample Sizes for Clinical, Laboratory and Epidemiology Studies or just use this online tool: Sample size calculation: comparison of two proportions. If you apply one test set to the two models you need the paired chi-square test: McNemar’s Test, and this would be also helpful: Statistical Significance Tests for Comparing Machine Learning Algorithms.

Help this would be of any help to you.

Answer 3

When the assumptions about the distributions of the test statistics(for instance normality) don't hold, we can apply the non-parametric tests such as approximate randomization and bootstrap test.

Here is the pseudocode of the latter(paired bootstrap test).

where $\delta(x)$ is the metric of the difference of a certain metric between two algorithms over a test set $x$ of size $b$. Assume we are detecting how often $A$ has an accidental advantage over $B$.

$\delta(x)=M(A,x)-M(B,x)$

and $\delta(x^{(i)})$ is that for a subsample $x^{(i)}$ of size $n$.

$b$ and $n$ should be large enough for a better result. We can set a threshold, for instance 0.05, to check whether the p-value is less than that to check out the null hypothesis: $\delta(x) \le 0$

In addition, due to the underspecification problem, it's hard for us to calculate the sample size because we don't know if the model performs well by skill or just by luck. But I thought we could start off with a sample size of $b_1$ to do a bootstrap test and increase the $b_i$ with some amount each time for several turns of bootstrap tests to see if the p-value plateaus.

Reference:

Speech and Language Processing (3rd ed. draft)