Do more object classes increase or decrease the accuracy of object detection

Assume you have an object detection dataset (e.g, MS COCO or Pascal VOC) with N images where k object classes have been labeled. You train a neural network (e.g., Faster-RCNN or YOLO) and measure the accuracy (e.g., IOU@0.5).

Now you introduce x additional object classes and add the corresponding labels to your original dataset giving you a dataset with N images where k+x object classes have been labeld.

Will the accuracy of the trained network increase or decrease?

To be more specific, we have a traffic sign dataset with around 20 object classes. Now we are thinking about adding additional traffic sign classes (labeling the new classes, without adding new images or changing our network architecture) and we are wondering if this will increase of decrease performance.

On the one hand I think more object classes will make distinction between classes harder. Additionally, a neural network can only hold a limited amount of information, meaning if the number of classes becomes very large there might just not be enough weights to cope with all classes.

On the other side, more object classes means more labels which may help the neural network. Additionally, transfer learning effects between classes might increase the accuracy of the network.

In my opinion there should be some kind of sweet-spot for each network architecture but I could not find any literature, research or experiments about this topic.

• I don't understand what you are trying to do. To my understanding you have a trained network, which fixes all relevant parameters (input, output, weights, architecture). What are you trying to change? If you change the input data, there is no way of telling what will happen. If you change the shape of either input, architecture or output, the network cannot be used anymore, since it is not longer defined. – cherub May 31 '18 at 14:28
• @cherub sorry if this was unclear, the parameters input, output and weights are changed during the training process of the larger datasets (with additional class labels). But the overall architecture of the network should remain. – SaiBot Jun 4 '18 at 7:02

Specific classification behaviour will depend on the particular model form underlying a classification method. The exact response of a model to additional object classes can be derived mathematically in particular cases, though this may be complicated. Since you have not given details of a particular method, I will assume that you are more interested in the general response of classification models to adding or removing object classes. To answer this, I will provide an intuitive explanation of what you should expect in a rational model of this kind of situation. To the extent that the model departs from this intuitive outcome, under broad conditions, I regard that as a deficiency. Hence, I regard the following responses as a desideratum for an object prediction system.

Prediction in a model with arbitrary object classes: To help facilitate analysis of this problem, suppose you have $$N$$ images of street-signs (or anything else) that are each as single one of $$m$$ types. Without loss of generality, let $$\theta_1, ..., \theta_N \in \mathscr{M} \equiv \{ 1, 2, ..., m \}$$ be the true types of the objects that you are trying to classify, with $$\mathscr{M}$$ being the true object types. Suppose you impose a detection system that classifies each image into types in the finite set $$\mathscr{S} \subset \mathbb{N}$$, where we note that $$\mathscr{S}$$ can include labels that are in $$\mathscr{M}$$, but it can also include values that are not in this set (i.e., it is possible that your detection system may be trying to find object types that aren't there).

A detection system of this kind looks at image data from each of the images, and uses this data to classify each image into an estimated type, based on the allowable types in the model. In general terms, this can be described by the following components:

$$\begin{matrix} \text{Data} & & & & & \text{Model Types} & & & & & \text{Estimates} \\ x_1, ..., x_N & & & & & \mathscr{S} & & & & & \hat{\theta}_1, ..., \hat{\theta}_N \in \mathscr{S} \end{matrix}$$

The probability of correct classification of image $$i$$ for a model with types $$\mathscr{S}$$ is:

$$p_i(\mathscr{S}) \equiv \mathbb{P}(\hat{\theta}_i = \theta_i | \mathbf{x}, \mathscr{S}) = \sum_{s \in \mathscr{M} \ \cap \ \mathscr{S}} \mathbb{P}(\hat{\theta}_i = s | \mathbf{x}, \mathscr{S}) \mathbb{I}(\theta_i = s ).$$

The elements of the latter summation are subject to the probability constraint:

$$\sum_{s \in \mathscr{M} \ \cap \ \mathscr{S}} \mathbb{P}(\hat{\theta}_i = s | \mathbf{x}, \mathscr{S}) = 1.$$

Now, clearly if $$\theta_i \notin\mathscr{S}$$ then we have $$p_i(\mathscr{S}) = 0$$, since the true object type is not included in the model. Hence, if there are elements of $$\mathscr{M}$$ that are not in $$\mathscr{S}$$, this will lead to inability to correctly identify these missing element types. On the other hand, if we exclude an element from the set $$\mathscr{S}$$ then, ceteris paribus, this will increase the probability of prediction of the remaining object types, since the probabilities of predictions must sum to one. Hence, exclusion of an object type will tend to raise the probabilities of prediction for other object types, which raises the probability of correct prediction for true object types that are in $$\mathscr{S}$$.

More detailed analysis would need to posit the connection between the data $$\mathbf{x}$$ and the object predictions. We will not go into detail on that subject here, since the particular model is unspecified. However, we may take it as a general property of prediction models that they will tend to have greater difficulty differentiating object types that look similar and will tend to have less difficultly differentiating object types that look dissimilar. Hence, exclusion of an object type from the set $$\mathscr{S}$$ will tend to increase the probability of prediction of other object types in this set that look similar to this excluded object, in cases where the data is conducive to one of these types.

The above exposition is designed to give some general guidance, stressing the probability constraint in predictions, and the way this impacts on the probability of correct prediction. This leads to the following general principles of a rationally constructed classification model. Ceteris paribus, the following should hold (at least roughly):

• If a true object type is excluded from the classification model, this will reduce the probability of correct prediction of that object type to zero, but it will tend to increase the probability of correct prediction for other object types (particularly object types that look like this excluded type);

• If a true object type is added to the classification model, this will allow the model to have a non-zero probability of correct prediction of that object type, but it will tend to decrease the probability of correct prediction for other object types (particularly object types that look like the added type);

• If a false object type is excluded from the classification model, this will tend to increase the probability of correct prediction for all true object types (particularly object types that look like this excluded type); and

• If a false object type is added to the classification model, this will tend to decrease the probability of correct prediction for all true object types (particularly object types that look like the added type).

These general principles may have some pathological exceptions in particular models, in cases where there is complex multi-collinearity between images. However, they should hold as general rules that will emerge in well-behaved models under broad conditions.

• Thank you for your elaborate answer, it gave me some crucial insights. The answer focuses on my assumption that "more object classes will make distinction between classes harder". Do you think of the four effects I mentioned in my question this is the most significant factor? I still have the feeling that more labels might offset this effect (at least up to some point). In particular the region proposal part of the network will certainly benefit from more data. – SaiBot Jun 4 '18 at 7:21
• It's really hard to say. These kinds of multivariate problems tend to be affected by the principal components (eigenvectors and eigenvalues) in the data matrix you use. Adding new images has a non-trivial effect on this, which is mathematically complicated. I agree that there is offsetting of effects if you add in two opposing effects, but it is very difficult to say the overall effect. – Ben Jun 4 '18 at 23:50
• Thank you @Ben, I think this is a very good but partial answer. I will award the bounty but not accept it for this reason and hope that's ok. – SaiBot Jun 6 '18 at 13:48
• No problem (and thanks for the bounty), but I think you'll probably find that a more detailed answer will require specification of the specific model form you're using, so that people can investigate its mathematical properties. Good luck with your problem. – Ben Jun 7 '18 at 0:01

Here is a detailed theoretical analysis on this topic. https://arxiv.org/pdf/1506.01567.pdf.

I think that depends on the specific problem and model. The mathematical propositions of the answer above can only be said about general statistical models. In an image data, we are looking at very high dimensions, and mathematics at that level (the extreme non-linearity of deep models also add) will be very complicated. What we can think intuitively (using a discriminant function approach), is that the more the classes are (Given inter-class variation is enough) the better model will be able to draw the discriminant function between classes. So, if the discriminant function is more detailed, the generalization capability of model will be greater when predicting an unseen image/example.

Think of it as parting between data clusters in a very high dimension. If you can part away the clusters more precisely, you would more likely to classify an incoming unseen example/image.

BTW, do inform us about the experiment, and did it increase or not. TIA.