I'm confused as to what the output would be.
Consider the "blind source separation" problem. Let's say I have a ton of training examples where the input is the final cacophony of sounds as a sound file, and the "answer" is the multiple individual audio files which comprised the final sound file. Maybe some of these are labeled, like "piano" or "man's voice", but others might be unlabeled, like a weird machine or animal. The system should learn what the labeled ones sound like (so it should identify "piano" and "man's voice" in future test data), but it should also be able to separate unknowns. But order doesn't matter on the unknowns -- for example if there is an alien sound and a tree falling sound, it doesn't matter if it puts alien sound as "unknown #1" and tree falling as "unknown #2" or vice versa.
Traditionally, if all source files were labeled, then I could maybe have n*k output neurons where n is the number of frequency bins needed to make a good-enough sound file, and k is the number of all possible sources in the whole world (piano, man's voice). Then when it's being fed the input data, it could be an RNN which fires 0 or more output neurons at each time step (e.g. 1024 per second), to recreate the source sounds.
But with the "unknowns", this turns into more like "sort of classification sort of clustering" problem and I am completely in the dark as to how to use the output neurons. For example if I reserve n*5 output neurons for "unknowns", how would it learn that it's supposed to put unknowns in there, but the order doesn't matter? How would it avoid erroneously "learning" that an "unknown" is supposed to sound like whatever the training data happened to tell it?