0
$\begingroup$

Say I have a regression technique (in my case I'm using ANNs) I am tuning on a data set. Say I am minimising a loss function that is not scale free, such as mean square error. Usually I would normalise the input and the output, perhaps by Feature scaling or standardisation, split my data in training, validation and test, and choose my hyper-parameters based upon performance on the validation set.

However, how can I compare the impact of two different normalisation schema? I am transforming the output as well as the input, so the scale of the output, when normalising by $$Y'=\frac{Y-\mu}{\sigma}$$ or by $$Y'=\frac{Y-Y_{min}}{Y_{max}-Y_{min}}$$ is different. Is it a matter of algebra and rearranging to get two comparable measures of error? I'm also intending to use a scale free measure of loss, such as the coefficient of determination, but I should really be optimising on the measure of loss I use to compare my results, correct (ie. no comparing the MSE of a solution found by minimising MAE etc.)?

$\endgroup$
0
$\begingroup$

I think one solution might: once I have configured and trained each model (using different preprocessing/normalisation schema), I can then apply the model to some data, then before getting results I can apply the inverse of the normalisation function I used. This should give me a measure of error in the original scale of the data, correct?

The only issue I have with this is that the error tracked automatically as the model is trained, such as in the keras.model.history object is in the normalised scale, but I suppose that's only important for seeing convergence?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.