How to choose delta parameter in Huber Loss function?

Question

In Huber loss function, there is a hyperparameter (delta) to switch two error function. Currently, I am setting that value manually. But, I cannot decide which values are the best. So, how to choose best parameter for Huber loss function using my custom model (I am using autoencoder model)? If you know, please guide me or send me links.

Answer 1

Huber loss will clip gradients to delta for residual (abs) values larger than delta. You want that when some part of your data points poorly fit the model and you would like to limit their influence. Also, clipping the grads is a common way to make optimization stable (not necessarily with huber).

Set delta to the value of the residual for the data points you trust.

See how the derivative is a const for abs(a)>delta

import numpy as np
import matplotlib.pyplot as plt
def huber(a, delta):
  value = np.where(np.abs(a)<delta, .5*a**2, delta*(np.abs(a) - .5*delta))
  deriv = np.where(np.abs(a)<delta, a, np.sign(a)*delta)
  return value, deriv

h, d = huber(np.arange(-1, 1, .01), delta=0.2)
fig, ax = plt.subplots(1)
ax.plot(h, label='loss value')
ax.plot(d, label='loss derivative')
ax.grid(True)
ax.legend()

Answer 2

As Alex Kreimer said you want to set $\delta$ as a measure of spread of the inliers. Most of the time (for example in R) it is done using the MADN (median absolute deviation about the median renormalized to be efficient at the Gaussian), the other possibility is to choose $\delta=1.35$ because it is what you would choose if you inliers are standard Gaussian, this is not data driven but it is a good start.

To get better results, I advise you to use Cross-Validation or other similar model selection methods to tune $\delta$ optimally.