I've been evaluating deep neural networks such as convolutional neural networks and object detection pipelines. Typically the metrics I use are top-k accuracy, precision, recall, and mAP.

I frequently give hand-wavy answers when someone asks me how many image/object examples per class I require in order to trust my evaluation metrics. For example, if I'm evaluating an image classifier that has 10 classes, I'll say I need something like 500 example images of each class. Basically pulling a number out of a hat that I intuitively think is reasonable, without doing any real calculations.

What I'd like to know is, is there a more mathematically sound way of determining how many evaluation examples I should obtain, as well as a way to express how this number of samples reflects the level of confidence in my evaluation scores?

I imagine this might depend on the complexity of the model, the number of possible classes, potentially whether the model is multi-class or multi-label (whether the class probabilities sum to one), what evaluation metric(s) are being used, etc.

If the level of complexity of models like CNNs actually makes this sort of rigorous evaluation mathematically intractable, then that would also be very helpful to know. It also might be the case that I'm drastically overcomplicating something that might be actually quite simple? I'm aware of this equation for confidence intervals for classification error:

$$error \pm const * \sqrt{ \frac{error * (1 - error)}{n}}$$

However, I don't know if/how the number of classes, type of activation function, type of model, etc. might affect this formula. Also I'm not sure if the formula is directly translatable to other evaluation metrics. Any input is greatly appreciated!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.