(ANNs) Fair comparisons between different inputs to the model Say I have a model that works fine for my task of detecting cats and dogs. I set my hyperparameters and I got a classification accuracy of 97%. 
Now I want to see if, by changing somehow the inputs (for instance, converting them to Black/White pictures), and training the network on these new inputs, I get higher or lower accuracy for the task. Let's say I get a 90% accuracy for this task and these inputs. 
So, the task remains the same, but the training data is slightly different. 
Would it be fair two compare these two results? Would it be fair to say B/W inputs are a worse input to train the network for this particular task. Or I could not really say that without first trying to find the best set of hyperparameters for this new task with these new inputs?
Thanks
 A: I'd say generally not, but in this case yes. 
There's an underlying assumption here that you are not changing the topology of the dataset and therefore the hyperparameters required should be the same. This is not the case. Changing any values in your data changes the loss landscape and therefore requires different hyperapramters. To find out if your tranformed data is more optimal you should re-search for parameters.
Certain image transformations do in fact (in this case changing colors) conduce better results. This is why we use convolutional layers which extract multiple "color versions" of an image. 
In the case you specified though, changing colored images of cats and dogs to black/white destroys valuable information that helps a network decide between the two.  
Edit: 
(Replying to your comment underneath) I say this because I guess that transforming colored images loses information. For instance, if you were to classify flowers based on their images, preserving color would be important. 
Why I say it wouldn't be fair in general is because the transformations aren't always as straightforward as converting to black and white. Consider the below transformations. You can't really say which are better or worse. To compare, it would be only fair if we optimize the network for each type.  

