Overfitting a neural network to a single batch as a sanity check - how small a loss value is small enough and long to run for?

Question

I'm currently developing a neural network for a regression task. Following on from the advice given in places like here, here, and here I'm attempting to overfit my model to a single batch of 5 training examples. I'm trying to predict two sets of parameters. One set between a the range of [-1, 1] (these happen to be Cartesian coordinates) and the other set is between [0,1].

Most of the literature I've found when discussing this uses simple classification as an example where you drive the cost as low as it can go, and aim to get 1.0 for accuracy. The issue seems more complicated and nuanced when dealing with regression.

At present the two main questions I have are:

1. How does one go about deciding whether a model is adequately overfitting on the given small subset of data when dealing with regression? It seems to be this is heavily dependent on the range of values you're attempting to predict. For house prices that are in the hundreds of thousands to millions of a MAE in the range of <10,000 might be acceptable. However, when dealing with a much smaller range of a numbers a MAE of <1 could be considered far too large.

   • I've used MAE for this example just because it directly relates to the units being predicted.

2. How long should you let the network run for trying to overfit the batch? One would assume if it takes a long time to overfit a single batch it's going to take even longer to learn and generalise a larger training set.

   • I would also think this possibly points to an issue with the capacity of the model or potentially the information provided by the input features making it difficult for the network to approximate a mapping.

Currently I've tried to overfit according to standard regression lost functions i.e MSE, RMSE, and MAE.

Answer 1

My experience regarding your questions comes from training an NLP model for predicting similarity between two sentences, which is a value in [0,5]. To output a score between [0,5], the model uses a dense layer with output dimension of 2. The scores are normalized using Softmax. I used regular MSE for loss, where it's computed between probability of label 1 and target labels, which are normalized to the range [0,1]. For overfitting, in my case some 2000 iterations on a single batch of 8 examples were enough (around 3 minutes). To answer your questions:

1. Train the model to predict the probability of a single category, and when computing loss (and in evaluation), normalize the target price range to 0,1.
2. It probably depends on your model. I recommend plotting the loss after some number of iterations to see if it reduces as expected, and then continuing for as long as needed.