3
$\begingroup$

I would like to train a neural network on an input signal and have it learn several unrelated decisions simultaneously (performing binary classification, one-class classification, and linear regression) to tell me different things about the input signal which I care about.

Note that this isn't something like multi-class classification, since I'll be using several different objective functions. Would simply summing my objective functions make the network less likely to converge, or take much longer to train? I will probably have to worry about the relative weights of each objective function. Is there any existing work that shows this is a bad idea? Should I just train several different classifiers independently on the same input signal? My motivation for using a single network is to reduce computation during inference.

$\endgroup$
1
  • $\begingroup$ There is Faster RCNN where they predict two things: (corrections of) bounding boxes (regression) and whether or not the box contains an interesting object (classification) in the „same“ network... I was not able to go through the training yet though so I do not know whether or not they simply add the losses or not... but nevertheless it is something that you could look up... $\endgroup$ Feb 6, 2020 at 16:26

1 Answer 1

1
$\begingroup$

All a neural network does is minimize some loss function. A sum (or multiple) of loss functions is a perfectly fine loss function in its own right.

In fact, this is quite a common setup. L(whatever) regularization is an extra term in the loss function, for one example. A VAE doesn't even work properly unless you use a sum of two losses, for another.

If neither loss term dominates the total, this can actually improve your network - e.g. overfitting on one task may increase the loss on another, so the training algorithm has a reason to look for a better solution.

As for the practical effects - depends. You're adding extra parameters, so a single pass is necessarily more expensive just due to the extra operations required, but you only have to train one model instead of three. Convergence depends on a lot of factors, so there's no general answer.

They may converge faster by avoiding bad minima of the individual tasks, or slower/worse if, say, you don't have enough capacity to solve all the tasks at once properly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.