2
$\begingroup$

I am a beginner reading neural networks. I wonder why there are loss functions, and then there are auxiliary loss functions. (By definition, 'auxiliary' would mean supplementary, help, support).

My question is: Does constructing an auxiliary loss function mean that they come as an 'afterthought'? After the loss function has been created? So is the purpose of having an auxiliary loss function be to improve performance that cannot be done by the loss function alone?

$\endgroup$
1
$\begingroup$

It was common to use auxiliary loss functions at one time (Google Inception Net and PSPNet) because the auxiliary loss hedged against vanishing gradient at layers far away from the loss function. This has fallen out of favor; my suspicion is that residual connections, rectifier activtation variants, and improved initialization schemes made auxiliary losses unnecessary.

$\endgroup$
0
$\begingroup$

Sycorax addresses the use of auxiliary heads / shortcut losses for addressing gradient flow.

But there are many other types of auxiliary losses still in common use today. For example, it is common to add an auxiliary GAN loss or perceptual loss for conditional generative modeling tasks (3D shape reconstruction, depth estimation, etc). In reinforcement learning, people will often add auxiliary rewards for reaching some intermediate states of the environment, or tack some unrelated vision task loss onto an agent's CNN to encourage learning of useful features. Some sort of entropy or diversity loss is used in many models from many different areas (but you could think of that as more regularization than loss).

Usually, we try to optimize the model for something not easily formulated as a differentiable mathematical function -- for example, whether an image looks "natural". The loss function is an initial best attempt at specifying that objective. I like to think of auxiliary losses as additional adjustments to fine-tune and nudge the model towards doing what we want it to do.

So is the purpose of having an auxiliary loss function be to improve performance that cannot be done by the loss function alone?

Yes, exactly -- because the loss function is an imperfect specification of the objective.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.