Say I have a model that works fine for my task of detecting cats and dogs. I set my hyperparameters and I got a classification accuracy of 97%.
Now I want to see if, by changing somehow the inputs (for instance, converting them to Black/White pictures), and training the network on these new inputs, I get higher or lower accuracy for the task. Let's say I get a 90% accuracy for this task and these inputs.
So, the task remains the same, but the training data is slightly different.
Would it be fair two compare these two results? Would it be fair to say B/W inputs are a worse input to train the network for this particular task. Or I could not really say that without first trying to find the best set of hyperparameters for this new task with these new inputs?