What is MCRMSE (mean columnwise root mean squared error)?

Question

The MCRMSE evaluation metric was used in the Kaggle Competitions Africa Soil Property Prediction Challenge(6 years ago) and OpenVaccine: COVID-19 mRNA Vaccine Degradation Prediction(On-going) competitions. There was no topic regarding MCRMSE (mean columnwise root mean squared error) on the internet.

AFAIK

Root Mean Squared Error - RMSE is the square root of the mean/average of the square of all of the error.

The use of RMSE is very common and it makes an excellent general purpose error metric for numerical predictions. Compared to the similar Mean Absolute Error, RMSE amplifies and severely punishes large errors. The formula for calculating RMSE is:

$\mathrm{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$

Mean Columnwise Root Mean Squared Error - MCRMSE

$MCRMSE = \frac{1}{m}\sum_{j=1}^{m}\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_ {ij}-\hat{y}_{ij})^2}$

or

$MCRMSE = \frac{1}{m}\sum_{j=1}^{m}RMSE_j$

where:

$m$ - number of predicted variables,

$n$ - number of test samples,

$y_{ij}$ - $i$-th actual value of $j$​-th variable,

$y_{ij}$ - $i$-th predicted value of $j$-th variable

I would like to understand What is MCRMSE? When to use??

When would one use MCRMSE over RMSE?

Answer 1

I believe the reason we are using MCRMSE in these challenges is because there are multiple outputs that we are trying to predict.

Normally, we can calculate RMSE to get a single-number evaluation metric for our prediction, but if we are predicting multiple values at once$-$in the case of the OpenVaccine competition, we need to predict degradation rates under multiple conditions$-$we would get multiple different RMSE values, one for each column.

The MCRMSE is simply an average across all RMSE values for each of our columns, so we can still use a single-number evaluation metric, even in the case of multiple outputs.