2
$\begingroup$

The MCRMSE evaluation metric was used in the Kaggle Competitions Africa Soil Property Prediction Challenge(6 years ago) and OpenVaccine: COVID-19 mRNA Vaccine Degradation Prediction(On-going) competitions. There was no topic regarding MCRMSE (mean columnwise root mean squared error) on the internet.

AFAIK

Root Mean Squared Error - RMSE is the square root of the mean/average of the square of all of the error.

The use of RMSE is very common and it makes an excellent general purpose error metric for numerical predictions. Compared to the similar Mean Absolute Error, RMSE amplifies and severely punishes large errors. The formula for calculating RMSE is:

$\mathrm{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$

Mean Columnwise Root Mean Squared Error - MCRMSE

$MCRMSE = \frac{1}{m}\sum_{j=1}^{m}\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_ {ij}-\hat{y}_{ij})^2}$

or

$MCRMSE = \frac{1}{m}\sum_{j=1}^{m}RMSE_j$

where:

$m$ - number of predicted variables,

$n$ - number of test samples,

$y_{ij}$ - $i$-th actual value of $j$​-th variable,

$y_{ij}$ - $i$-th predicted value of $j$-th variable

I would like to understand What is MCRMSE? When to use??

When would one use MCRMSE over RMSE?

$\endgroup$
1
$\begingroup$

I believe the reason we are using MCRMSE in these challenges is because there are multiple outputs that we are trying to predict.

Normally, we can calculate RMSE to get a single-number evaluation metric for our prediction, but if we are predicting multiple values at once$-$in the case of the OpenVaccine competition, we need to predict degradation rates under multiple conditions$-$we would get multiple different RMSE values, one for each column.

The MCRMSE is simply an average across all RMSE values for each of our columns, so we can still use a single-number evaluation metric, even in the case of multiple outputs.

$\endgroup$
2
  • $\begingroup$ This is the correct answer (+1) for why they use this metric. I see a few reasons not to like it. If one output had much higher variability, then you could win the competition by crushing that one at the expense of the others. (Then again, it might be reason able to prioritize certain predictions, but we must watch out for this.) Second, the more typical way to examine this sort of multivariate regression would be with $L_2$ distance (maybe Mahalanobis) in the output space, which, in some sense, has a bit easier of an interpretation. $\endgroup$ – Dave Sep 13 '20 at 15:10
  • $\begingroup$ Related to that second criticism, this seems like a hybrid of $L_1$ and $L_2$ metrics, harshly penalizing large errors within an model but then mildly punishing a model that, overall, is poor. Perhaps that is desirable, but we should be mindful of that feature of the metric. $\endgroup$ – Dave Sep 13 '20 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.