I created a neural network for time series forecasting. My experiments involve comparing the effects the different regularizers have on the model. I used cross-validation and measured my model's performance using the RMSE and $R^2$ metrics.

The data consists of 187 locations, each containing 3 features.

My objective is to show to the reader which regularizer performed the best.

The problem is that there are too many results to compare. My results are saved in the following structure: for each regularizer experiment there are 8 cross-validation folds and in each fold there are 187 locations. c, d, r represent the error value for each of the 3 features.

             | l1   -> c1, d1, r1
      |  f0 -| ...  -> c, d, r
reg0 -|  ... | l187 -> c187, d187, r187
      |  f7

Putting this in a table seems unfeasible as there would be almost 9000 rows with three columns. I though about selecting a few of the locations to use an an example but this seems like cherry-picking. And using only one of the validation folds seems to defeat the point of using cross-validation.

Is it possible to average the $R^2$ scores over all the locations creating one score for each model? Would this mean anything? I don't believe it to be possible to average the RMSE as the data for each location would be on a different scale.

How can I best compare these results without ignoring some of the locations or validation folds?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.