2
$\begingroup$

Say I have a model that works fine for my task of detecting cats and dogs. I set my hyperparameters and I got a classification accuracy of 97%.

Now I want to see if, by changing somehow the inputs (for instance, converting them to Black/White pictures), and training the network on these new inputs, I get higher or lower accuracy for the task. Let's say I get a 90% accuracy for this task and these inputs.

So, the task remains the same, but the training data is slightly different.

Would it be fair two compare these two results? Would it be fair to say B/W inputs are a worse input to train the network for this particular task. Or I could not really say that without first trying to find the best set of hyperparameters for this new task with these new inputs?

Thanks

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I'd say generally not, but in this case yes.

There's an underlying assumption here that you are not changing the topology of the dataset and therefore the hyperparameters required should be the same. This is not the case. Changing any values in your data changes the loss landscape and therefore requires different hyperapramters. To find out if your tranformed data is more optimal you should re-search for parameters.

Certain image transformations do in fact (in this case changing colors) conduce better results. This is why we use convolutional layers which extract multiple "color versions" of an image.

In the case you specified though, changing colored images of cats and dogs to black/white destroys valuable information that helps a network decide between the two.

Edit:

(Replying to your comment underneath) I say this because I guess that transforming colored images loses information. For instance, if you were to classify flowers based on their images, preserving color would be important.

Why I say it wouldn't be fair in general is because the transformations aren't always as straightforward as converting to black and white. Consider the below transformations. You can't really say which are better or worse. To compare, it would be only fair if we optimize the network for each type.

Filters

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the comment @Daniel. I might have missed something but I don't see an agreement in your answer. You say: "Changing any values in your data changes the loss landscape and therefore requires different hyperparameters". Then I should look for that set of hyperparameters before making any comparison, right? Why do you say that "in this case yes (you can compare the results)" $\endgroup$
    – sdiabr
    Commented Apr 30, 2018 at 9:59
  • $\begingroup$ @sdiabr I updated the answer. $\endgroup$
    – Daniel
    Commented Apr 30, 2018 at 17:10
  • $\begingroup$ Yes I agree. My case is actually whether to zero-pad or append the same instance at the end of the file, when working with Audio file classification. (I wrote the example of the image since it is a more general one). The bad thing is that it would never be comparable. The only thing you could say would be something like "After 7 weeks looking for hyperparameters, I conclude that this input to the network is a better one". Agree? $\endgroup$
    – sdiabr
    Commented Apr 30, 2018 at 18:15
  • $\begingroup$ @sdiabr It's likely that these input are worse for your model. Although you must be wary if training on the same model. Perhaps this change has increased the complexity of your data and your model capacity isn't large enough to fit the data. $\endgroup$
    – Daniel
    Commented Apr 30, 2018 at 22:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.