0
$\begingroup$

For a mixture of experts model, is each expert trained individually, or are they trained with respect to the overall loss?

It seems each would be trained individually, since its not a requirement that each expert is a neural network.

$\endgroup$
0
$\begingroup$

This strongly depends on the task and how you combine the models.

Usually, at least at some point, probably only at the combination point, an overall loss is preferred as in the end you want a superior combined model and not a lot of good standalone experts. But it highly depends on the task and it is also common to pretrain experts. Some networks would not really converge even as they are too big. Sometimes the models are even of different nature.

If your task is not too complicated, I would try with a combined one. If you have some well trained specific experts (something like AlexNet and similar), you train those separately and combine them wisely.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.