I've developed a model that predicts a future value of a parameter for the next 72 hours (only 11 hours presented on the chart). I've obtained the hyperparameters for my model with use of RandomizedSearchCV. Than I've retrained my model with those hyperparameters on the development (train) set and tested its performance on the test set (peace of data used only for this purpose). The chart below shows performance of the model on the train and test set. And here is my question: how can I say whether the model has high variance or bias? Is it safe to just assume that since the MAEs of the train and test sets are similar and not spectacularly huge then the RandomizedSearchCV just got me good hyperparameters ant its just fine?

enter image description here

  • $\begingroup$ I don't think you can determine presence of bias or high variance just from this plot / results. $\endgroup$ – user2974951 Sep 26 '18 at 7:17
  • $\begingroup$ @user2974951, but I think that I can say that: since that MAEs on the train set follow the same pattern as the ones form the test set, the findings are not random or due to chance. Moreover, the model's performance on the train set is better as expected. However, the results are of the same order of magnitude, thus if present, then the over-fitting on the training set is rather minimal. What do you think? $\endgroup$ – DexzMen Sep 26 '18 at 11:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.