We are training a feed forward neural network as a regressor, with the aim of using the activations of the final layer as a type of embedding vector to represent the input examples.
The 500 feature dimension input examples represent physical objects, and have been centered and scaled before use as training data in the network. The target value is a float representing the lifespan of the product in years e.g. 22.7. The lifespan of the input objects varies considerably from a few days to many years. The network has two hidden layers, layer1=64 neurons, layer2=16 neurons, both with relu activation, then a single linear output neuron for the lifespan.
The plan is to discover if/how these objects differ along each axis of the the embedding space (by visualising images of the objects), in order to find common characteristics that are responsible for the change in the target value. By fixing all but one of the axes, and selecting elements close to a 1-D line in the final axis, we hope to find elements that are common in all aspects except the one expressed by the 'free' axis. For example, a successful result could be the discovery that in general, the larger an object is, the longer its lifespan.
The question is: do we need to optimise for validation loss to obtain valuable insights, or can we simply overfit the data on 100% of the training data to generate the embeddings? We don't have enough data to achieve a good validation loss, but can (of course) achieve a very low training loss.
The opposing views
Person A believes that deliberately overfitting the training data will produce a neuron activation embedding space at the final layer that, even though it cannot be considered generalisable to all physical objects in the world, could still yield useful conclusions about the training set if some clear changes in physical characteristics could be identified along one of the axis. The thinking is that if such characteristics were discovered, they would at least inform us about and open up new avenues of investigation into promising aspects of the physical products e.g. feature reduction / feature engineering on the discovered characteristics could be one avenue, leading to a more generalisable model later.
Person B believes that these conclusions would be useless, as they would only apply to the training set. As a neural network can be trained to fit any function given enough complexity (neurons/layers), the embedding space would not allows us to conclude anything useful, and even if we did see the target (lifespan) correlate with certain physical attributes, these would not be worth exploring further for unseen data.
We'd love to hear your views to help us resolve this disagreement. Are either/both of these positions valid? What are we missing?