I am clustering some pretty fuzzy data with a special k-means like algorithm (a change of algorithm is not an option). Due to random initialization of cluster centers and the fuzziness of the data the results of two identically trained models are different.
My colleague asked if I could combine the results of the models to get a better model. He suggested to keep clusters where both models agree on (to a certain %) and leave out clusters that are (too) different in both models with the reasoning, that data in these clusters is not really clustered correctly. To assign new data you would let it run on both models and if they both agree to assign the new data to a valid, agreed-on cluster, you have the clustered result. If not, then the solution is that the new data can´t be clustered properly.
This idea sounds pretty unintuitive to me but I couldn´t come up with a proper argument against it so far. Of course the new, combined clustering method would "loose" some data that is labelled as "can´t be clustered", but this is acceptable.
Is my intuition right that this will not really work or is this really a valid approach and I was just unsuccessful with my research and haven´t heard of something like that yet?