Let's say I'm working with two models, ResNet-50 and ResNext-50, and I want to compare the results. Due to the stochastic nature of deep learning, would it be advantageous or even encouraged to seed the models? Should the same learning rate schedule and hyperparameters be used? Finally, should the input data be fed in the same order?
Rather than try to initialize the models identically, I suggest running multiple seeds and comparing the results of multiple runs of both algorithms.
Since the architecture of the two models is different, you can't really initialize them identically, so I don't see any benefit of starting with the same seed.
The same goes for the hyperparameters: A different model may be optimal with different values for its hyperparameters (it may not even have the same hyperparameters).
If you can afford it computationally, try cross-validating your results.