Generalizing a Keras model

Question

I have two Keras models that I use to count the objects in an image.

The training set that I use has images with between 1 and 10 objects. The test set that I use has images with between 1 and 20 objects.

The first model is based on MLP and has very good results on the test dataset, but only on images that have between 1 and 10 objects. The second model is based on CNN and has average results on the test dataset, but again, only on images that have between 1 and 10 objects.

Both models have the output defined as Dense(1), but neither of them seem to generalize outside the training data. It is as if they could do a good job for a classification model, but not so much for regression.

What are your suggestions on improvements or approaches that I should take to be able to detect an arbitrary number of objects in images? I'm not ruling out that I am doing something fundamentally wrong either, and if so please point it out.

Edit 1:

MLP model:

model = Sequential()
model.add(Dense(100, input_shape=(128 * 128,)))
model.add(Dense(1))

CNN model:

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), padding="same", activation="relu", input_shape=(128, 128, 1), kernel_initializer="glorot_normal", bias_initializer="glorot_normal"))
model.add(Conv2D(64, kernel_size=(3, 3), activation="relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))
model.add(Flatten())
model.add(Dense(128, activation="relu"))
model.add(Dropout(0.5))
model.add(Dense(1, activation="relu"))

Datasets:

• Training has 10K samples uniformly distributed in images with between 1 and 10 objects
• Test dataset has 100 samples uniformly distributed in images with between 1 and 20 objects

Results:

• MLP: almost 100% accuracy on images with between 1 and 10 objects. 0% accuracy on images with between 11 and 20 objects. Furthermore, prediction will always lie between 1 and 10.
• CNN: around 50% accuracy on images with between 1 and 10 objects. 0% accuracy on images with between 11 and 20 objects. Furthermore, prediction will always lie between 1 and 10.

Answer 1

Unless you have a very simple model, such as linear regression, then it is very hard to construct something that extrapolates to unseen data where one or more features and/or the target variable are outside the bounds of the training data. Most non-linear models will extrapolate poorly in general, unless the underlying "true" model happens to fit nicely to how the function approximation works.

One way to address this would be to construct a pipeline with simpler robust logic for counting objects. Instead of trying to regression the number of objects in scene, train an object detection model working with smaller image patches and scan it across larger images to count. There are likely to be smarter pipelines in the literature, based on generic object detection and classification, or rnn-based attention etc.

Alternatively, just train your object counter on data that is representative of how it will be used.

---

As an aside, you could try removing the first dense layer. I would expect that to reduce overall accuracy but be somewhat better at extrapolation. In addition you could try more regularisation such as dropout or L2 weight loss (maybe try max norm weights limit as the original paper on dropout suggested it was effective used alongside it). These will not fix the problem but might make a minor improvement.

Another unverified approach would be to change your ground truth data to identify the centre of each object (i.e. output would be a feature map, maybe reduced in size from original, where each pixel was probability that it was the centre of a target object). Then a logical process could sum up any centres past a certain threshold. You would need to allow for fuzzy detection of centres.