0
$\begingroup$

In a cross validation scenario "leave one out" I want to measure how the estimated continuous variable fits with the observed variable. I learned from Wikipedia that the Median absolute deviation (MAD) could by used.

My question: How is the MAD to be calculated in this scenario? I have two ideas. The first is inspired from the definition of MAD where the center is the median of all deviations:

  1. Set the deviation $D_i = E_i - O_i$ for each corresponding estimated and observed outcome
  2. Calculate the median $M = \underset{i}{arg\;median}({D_i})$
  3. Set $MAD = \underset{i}{arg\;median}({|M-D_i|})$

The second one seems more appropriated to me in the context of cross validation:

  1. Set the deviation $D_i = E_i - O_i$ for each corresponding estimated and observed outcome
  2. Set $MAD = \underset{i}{arg\;median}{(|D_i|})$.

The last is the median of the absolute deviation between estimated and observed value, literally. However, it seems not to confirm to the definition given by Wikipedia.

What is the best solution in my scenario?

BTW: In the wikipedia article about the Mean absolute error I found an interesting comment about this issue:

The mean absolute error is a common measure of forecast error in time series analysis, where the terms "mean absolute deviation" is sometimes used in confusion with the more standard definition of mean absolute deviation. The same confusion exists more generally.

$\endgroup$
1
$\begingroup$

Wikipedia (emphasis mine):

For a univariate data set X1, X2, ..., Xn, the MAD is defined as the median of the absolute deviations from the data's median

Since you have observed values in your scenario, you are likely looking for something along the lines of Mean Absolute Error or Root Mean Square Error.

EDIT: (additional clarification from comments below) For a univariate data set, MAD is a measure of the variability (i.e. what is the "average" deviation from the "average" value, using the median function to determine averages).

So as you compute it the first way, it tells you the variability among your errors, which isn't necessarily meaningful as a measure of how well your model fits.

In the latter way, you are computing something very close to MAE, but using median instead of the mean. This is a valid way to determine your goodness of fit, but I don't know why you wouldn't use the more common measures of MAE or RMSE.

$\endgroup$
  • $\begingroup$ That is: my univariate data set is the set of ... $D_i$ and the first version in my question is valid? What meaning has MAD in this form for cross validation? $\endgroup$ – Claude May 8 '14 at 8:58
  • $\begingroup$ I checked the article about Mean Absolute Error and found an interesting comment which I have quoted in my original question above. $\endgroup$ – Claude May 8 '14 at 9:07
  • $\begingroup$ For a univariate data set, MAD is a measure of the variability (i.e. what is the "average" deviation from the "average" value, using the median function to determine averages). So as you compute it the first way, it tells you the variability among your errors, which isn't necessarily meaningful as a measure of how well your model fits. In the latter way, you are computing something very close to MAE, but using median instead of the mean. This is a valid way to determine your goodness of fit, but I don't know why you wouldn't use the more common measures of MAE or RMSE. $\endgroup$ – Hao Ye May 8 '14 at 9:50
  • $\begingroup$ I use RMSE already. Thanks for your suggestion to use MAE, I will give it a try. The usage of MAD was inspired by the Wikipedia article. Your last comment was the real answer to my question. Please add the core points to your answer above (besides the definition) and I will accept it. $\endgroup$ – Claude May 8 '14 at 10:17
  • $\begingroup$ I presume OP would want to use mad rather than mae, mse, or rsme for the usual reasons... means are subject to the influence of outliers and medians are not. e.g., stats.stackexchange.com/questions/45875/… $\endgroup$ – Brian D Nov 10 '17 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.