I'm currently developing a neural network for a regression task. Following on from the advice given in places like here, here, and here I'm attempting to overfit my model to a single batch of 5 training examples. I'm trying to predict two sets of parameters. One set between a the range of [-1, 1] (these happen to be Cartesian coordinates) and the other set is between [0,1].
Most of the literature I've found when discussing this uses simple classification as an example where you drive the cost as low as it can go, and aim to get 1.0 for accuracy. The issue seems more complicated and nuanced when dealing with regression.
At present the two main questions I have are:
How does one go about deciding whether a model is adequately overfitting on the given small subset of data when dealing with regression? It seems to be this is heavily dependent on the range of values you're attempting to predict. For house prices that are in the hundreds of thousands to millions of a MAE in the range of <10,000 might be acceptable. However, when dealing with a much smaller range of a numbers a MAE of <1 could be considered far too large.
- I've used MAE for this example just because it directly relates to the units being predicted.
How long should you let the network run for trying to overfit the batch? One would assume if it takes a long time to overfit a single batch it's going to take even longer to learn and generalise a larger training set.
- I would also think this possibly points to an issue with the capacity of the model or potentially the information provided by the input features making it difficult for the network to approximate a mapping.
Currently I've tried to overfit according to standard regression lost functions i.e MSE, RMSE, and MAE.