Improvement in NN regressor by Negative Log Liklihood loss vs MSE loss

Question

I am trying to write a simple NN based regressor, and I have noticed that if i take two identical NN, one with mean square error loss, ane with sample drawn as gaussian prior over final output, with negative log likelihood loss (nll), the nll loss performs significantly better. My understanding it, nll loss with Gaussian priors are same as MSE so shall the output error be not similar?

######
# MSE loss over NN
######
model = tf.keras.Sequential([
tf.keras.layers.Dense(800,input_shape=(371,),kernel_regularizer='l2'),
tf.keras.layers.LeakyReLU(alpha=0.1),
tf.keras.layers.Dropout(0.1),
tf.keras.layers.Dense(1000,kernel_regularizer='l2'),
tf.keras.layers.LeakyReLU(alpha=0.1),
tf.keras.layers.Dropout(0.1),
tf.keras.layers.Dense(1)
])
model.compile(loss=MeanSquareError(), optimizer=tf.keras.optimizers.Adam(lr=0.0001),metrics=['mae'])


######
#nll loss over Normal drawn of final mean and sigma predictions
######

model = tf.keras.Sequential([
tf.keras.layers.Dense(800,input_shape=(371,),kernel_regularizer='l2'),
tf.keras.layers.LeakyReLU(alpha=0.1),
tf.keras.layers.Dropout(0.1),
tf.keras.layers.Dense(1000,kernel_regularizer='l2'),
tf.keras.layers.LeakyReLU(alpha=0.1),
tf.keras.layers.Dropout(0.1),
tf.keras.layers.Dense(2),
tfp.layers.IndependentNormal(1)
])
model.compile(loss=lambda y_t,y_p: -y_p.log_prob(y_t)
 , optimizer=tf.keras.optimizers.Adam(lr=0.0001),metrics=['mae'])

Is there any reason why nll loss will perform better? Are they not equivalent in my example above?

PS: First model was trained using MSE loss, second model was trained using NLL loss, for comparison between the two, after the training, MAE and RMSE of predictions on a common holdout set was performed.

In sample Loss and MAE:

1. MSE loss: loss: 0.0450 - mae: 0.0292, Out of sample: 0.055
2. NLL loss: loss: -2.8638e+00 - mae: 0.0122, Out of sample: 0.050
3. Kernel ridge regression Out of sample: 0.0575

Answer 1

I finally found the answer in this paper https://ieeexplore.ieee.org/document/374138 , also explained and referenced in a blogpost.

It mentions clearly in the paper that NLL performs better then MSE because the loss function becomes:

$$ NLL = \sum_i \frac{1}{2}\log(\sigma^2(x_i)) + \frac{(\mu(x_i) - y_i)^2}{2\sigma^2(x_i)} $$ Now if i assume $\sigma$ to be constant then my loss function becomes equivalent to MSE * const. which was the basis of my original comment

... nll loss with Gaussian priors are same as MSE ...

However in current network $\sigma$ is a variable, and hence the network gives higher weight to datapoints with lower variance. Resulting in improved learning in current case.

But the paper mentioned that if datasets are not large enough then NLL results in over fitting, which can again be explained on the same basis. I am including the relevant part of the paper below:

---

Nix, D.A. and Weigend, A.S., 1994, June. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 ieee international conference on neural networks (ICNN'94) (Vol. 1, pp. 55-60). IEEE.