2
$\begingroup$

I've ~16000 labeled data. I split it in ~8000 for training and ~8000 for testing of my RBF neural network, find the best hyperparameters (RMSE from 1.2 to 1.4*) and finally train the model on the whole set of ~16000 labeled data. When I now run these 16000 samples through the final model, I get way worse result (RMSE from 2.0 to 2.3*). How is this possible?

My RBF has 350 neurons in the input layer, 2 in the output one and two hidden layers with 50 and 30 neurons respectively. I use gaussian distribution for weight initialization, sigmoid as an activation functions in the hidden layers and RELU in the last one, Adam to optimize and L2 as loss function.

* depending on the random seed used for train-test split and weight init

EDIT: I use batches of 400 samples and 500 epochs. So I have 10 000 training iteration when I use 8k samples for training and 20 000 when I use the whole dataset.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ What accuracy are you talking about? How do you calculate it? What is your data? $\endgroup$
    – Tim ♦
    Oct 12, 2021 at 16:02
  • $\begingroup$ @Tim, My data is time series with 350 data points (one point per millisecond), scaled into 0-1 interval. The model should find one specific peak and mark the start and end - thus two labels in the training set and output layer size 2. As "accuracy" I denote the ability to predict the start/end of the peak at the same time point as in the training set. The closer the predicted start/end to the respective labels, the more accurate the model is. As mentioned above, I calculate root mean square error (in milliseconds). $\endgroup$
    – dpelisek
    Oct 13, 2021 at 13:03
  • 1
    $\begingroup$ But what accuracy are you referring to? If you split the data to 8k train vs 8k test, you can calculate test accuracy on 8k test samples, if you use whole data, there is no test set, so what you are referring to..? Train accuracy? $\endgroup$
    – Tim ♦
    Oct 13, 2021 at 13:05
  • $\begingroup$ @Tim I also got a bit confused by exactly the same thing. $\endgroup$
    – Diego Queiroz
    Oct 13, 2021 at 21:20
  • $\begingroup$ Yes, when I split the data to 8k train vs 8k test, I calculate test accuracy on 8k test samples. When I use whole data (all 16k) for training, I use these 16k set for testing as well. $\endgroup$
    – dpelisek
    Oct 14, 2021 at 13:42

1 Answer 1

Reset to default
2
$\begingroup$

There are some reasons that could lead to that happening yes, one of such reasons could be due to the increased variance in the data (more outliers or a distributional shift after incorporating the other half of the entire dataset), you might want to employ outlier detection to detect and remove those examples from your dataset.

Also, running the train/test once (a validation scheme usually called holdout) is often not a good measure of the actual performance of any model (it may lead to unstable models both in hyperparameters and accuracy), specially in non-deterministic training models like neural networks that are very sensitive to initialization (which may also be the culprit for your problem too).

Therefore I'd suggest you to try more robust validation schemes, as at least it will give you a good grasp of what is the actual performance of your model:

  1. Holdout with 50% split may be not enough for your model to generalize properly. Usually when validating through holdout, it's recommended to use more data to train than to test, 80/20, 75/25 and even 60/40 splits are way more common.
  2. Instead of holdout, use a proper k-cross-validation scheme, you can start with 10-fold cross-validation (divide data into 10 groups, then run the training from start using 1 of the 10 to test and the others to train, each run which group is the test is changed, then evaluate the average performance obtained of all runs), since i don't know the task i'm not able to really propose something tailored for your situation, you might need to do a nested or stratified cross-validation...

Once you're satisfied with your validation scheme, I'd devise an environment to test values obtained for each hyperparameters to investigate them as well.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks Diego for your comments. I tried various random seeds for weight initialization and train/test split to prevent getting stuck in local minima and overfitting to a specific data set. I'll try the k-fold cross validation. Could you please refer me to a document/book/tutorial that mentions the best practice to choose the holdout size? $\endgroup$
    – dpelisek
    Oct 14, 2021 at 14:24
  • 1
    $\begingroup$ I can refer you to this paper, It takes into consideration many things i've said in my comment and shows examples of why 'rules of thumb' fall short if you follow them blindly. Of note, I can recommend sections 1.3 and 1.6 $\endgroup$
    – Diego Queiroz
    Oct 14, 2021 at 14:47
  • 1
    $\begingroup$ Thanks. With 10-fold CV (instead of early stopping that I used so far) I get 1.6 RMSE on 8k unseen data when trained on another 8k data. If I use all 16k samples to train the model and those same 16k samples to test it, I get 1.3 RMSE. This seems more feasible to me. I probably overfitted the hyperparameters to the validation set when I used 50% holdout. $\endgroup$
    – dpelisek
    Oct 18, 2021 at 8:38
  • $\begingroup$ @dpelisek I'm glad it worked for you, overfit is often caused by these things. You might want to check if there is noise in your data too now that you have a more robust evaluation setup. $\endgroup$
    – Diego Queiroz
    Oct 18, 2021 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.