2
$\begingroup$

I am very new to Machine learning. I am trying to implement a feedforward neural network to predict release year of songs based on some audio features.
I have a train.csv file and dev.csv file. I use train.csv to train the model and side by side evaluate my model after each epoch of training on dev.csv data by calculating RMSE. After fully training the model, i need to submit my predictions on "test" data in Kaggle competition where it evaluates and displays RMSE on my predictions on test data. Now the problem i am facing is, say i train my model for 100 epochs and at the end RMSE on "dev" set is 90. Then when i use this model to make predictions on test set, i get RMSE of 30. Now i tune the model further and manage to reduce my RMSE on dev set to 56, then i expected my RMSE on test set would also decrease. But instead it increased to 38. And this has happened quite a few times, as in, i thought one evaluates the model accuracy on dev set because it is a unseen data and model has not been trained on it, so performance on dev set must also in some way directly correlate to performance on test set. But in this case i am not able to see any correlation. Since my end task is to make RMSE on test set as low as possible, how can i go about it. Because even if reduce dev set loss, it is not refelecting on test set. Is it possible that my dev set data is NOT A RELIABLE source? If so, what techniques can i use to get a dataset on which i can get a "reliable" model performance accuracy?

$\endgroup$

1 Answer 1

1
$\begingroup$

If your validation and test sets are

  1. sufficiently large
  2. neither of them have been used to train the model
  3. and are also suspected to have been sampled from the same distribution

then the validation loss will be a good approximation of the test loss.

However, your validation loss is about 3 times greater than the test loss which means at least one of the assumptions is not valid. If I had to guess, I'd guess that your test set is an easier test for your model than your validation set (assumption 3. not valid) as this is a common mistake.

Regarding the RMSE, if the RMSE on the validation set decreases it does not necessarily mean that the RMSE on the test set will also decrease. This is due to the random nature of modelling. Instead, you want the moving average of your validation and test losses to decrease as you train the model from epoch to epoch. In other words, the RMSE of the test going from 30 to 38 is not a sign to be concerned. What's a bit concerning is the difference in magnitude between the test and validation losses.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.