I consider myself an intermediate practitioner of neural networks. I've been asked to teach a few of my colleagues some of what I know. Some of my practices may be a bit idiosyncratic, because I study and tinker on my own. I want to teach "best practices," and when I'm in doubt, I'm trying to research exactly what those are.
I'm working with the housing price data set from the book Hands-On Machine Learning with Scikit-Learn and TensorFlow. I'm trying to predict housing prices from the other information in the data set, a pretty classic sort of student problem. The minimum possible housing price is obviously zero, and the mean housing price in this data set is around $250,000. I already have human-readable output from a linear regression model which reports its results in dollars. I would like to preserve that in subsequent models, so we can also do things like compare mean squared errors between architectures.
Knowing my data, I would be inclined to do one or both of the following:
On the output layer, which is a single node with linear activation, set the initial bias to the mean value of 250,000.
Add an exponential activation function on the output node, to make negative prices impossible.
I did a combination of these two things, and designed some demonstration models with very few non-linear ReLU elements which outperform the linear regression after only a few dozen epochs. That's what I wanted to demonstrate.
However, I've never seen any published examples of people doing this kind of hand-tinkering. So I tried leaving off my customization, and repeated the training process. After 2,000 epochs, the network was still working the outputs slowly up from zero towards the mean 250,000 value, and the training errors were still stupidly high. I was using ADAM for gradient descent. I thought that ADAM would have found the mean much more quickly than this because of its momentum feature.
I think I can also improve my training process by training on normalized targets. If the output targets are near zero, and the standard deviation is near 1, the typical weight initialization schemes should find it easy to start training on actual features in the data immediately, rather than wasting epochs simply grinding towards the mean.
If I train to a normalized output, the meaning of the internal mean-squared error in the neural model training process will no longer be comparable to the MSE from the linear regressions. I will also need to perform the inverse of the data normalization process on the output to produce a human-readable result. Both of those tradeoffs are acceptable to me, but I'll have to walk my students through that additional layer of complexity.
What's the "right" way? Thanks for your advice.