1
$\begingroup$

In a paper by Rousseeuw and Croux from 1993 ("Alternatives to the Median Absolute deviation", page 1274, link to pdf), I came across an indicator I'm considering using. The formula is:
$$ Sn = C\, {\rm median}_i \{{\rm median}_j |x_i-x_j|\} $$ The way I read the formula is that for every $X_i$ I take the differences from each of the other values in the sample. Then, again for every $X_i$, I take the median of the values I get. Finally I take the median of the median values I obtained in the previous step and multiply it by the parameter $C$ which according to the authors is $$ C= 1.1926. $$ I'm not a statistician by any means and I don't understand what the authors mean when they say that the value $C$ is derived through an asymptotic assumption. What I know is that my data is not normally distributed and resists normalization if I try to standardize it. This is why I'm trying to find an indicator where I can rank the data around a central point that does not demand symmetry in the distribution (hence, not using the MAD).

My question is: do I have to maintain the $C$ factor in the equation? What is the underlying assumption? I seem to understand that it involves some kind of Gaussian distribution (what I know as a normal distribution) but since my data is not normally distributed, I am unsure what value for $C$ I should apply and if it's OK to just leave it out.

**My end goal is the following:

  1. I have data not distributed in any meaningful usable way. Yes, there may be ways to transform or fit it in some kind of distribution but that would really complicate things, beyond my ability to explain it to my colleagues.
  2. Each data point is a score, ranging from 0 to several thousand, which is a function of the answers given in a survey.
  3. I need to assess each of the data points in relation to one another, to determine which stand at the bottom of the risk scale (lowest scores) and which at the top (highest scores). To do so I need to define some cutoffs. My plan was to use the MAD or better, the Sn indicator as of above, to have an indication of what the score is for a common risk profile and then use percentiles to associate a lower risk profile to lower scores and higher risk profile to higher scores.

I hope this is not confusing but I'm venturing into really uncharted territory for me here.

$\endgroup$
  • $\begingroup$ The expression "resists normalization if I try to standardize it" is rather curious. Does it mean "I tried, but couldn't manage to transform it to normality"? $\endgroup$ – Glen_b -Reinstate Monica Jan 27 '14 at 23:35
  • $\begingroup$ Yeah, I realize it's a little funny. The thing is that I took Z score using both mean and median and in both ways the distribution fits the data really poorly. I can still say or assume for that matter that the data is normally distributed but it really really is not! $\endgroup$ – Bernardo Jan 28 '14 at 14:26
  • $\begingroup$ How did you assess normality? What are the $x's$ measuring? $\endgroup$ – Glen_b -Reinstate Monica Jan 28 '14 at 21:42
2
$\begingroup$

Your question is a bit confusing. Let me parse it.

This is why I'm trying to find an indicator where I can rank the data around a central point that does not demand symmetry in the distribution (hence, not using the MAD).

The Sn and the MAD are not measures of central tendency, but of scatter.

My end goal is to find a value against which I can compare the other values and perhaps using percentiles, decide what is a common value and what are outliers (above and below such value),

For doing this you need a measure of location (or central tendency). You cannot use either the MAD or the Sn as a yardstick against which to compare the observations to assess if they are unusually large.

EDIT

The Sn is not an estimator of centrality but of dispersion. It makes no sense to classify a datapoint as large or small depending on whether it is larger or smaller than the Sn. To compare individual observations against a yardstick, that yardstick should be a measure of location (for example the median). You can find a summary of what these terms mean here.

To convince yourself that the rule you use produces nonsensical results, just add +10,000 to all your observations and see how your rule fares. Now replace the Sn by the median, retain the rest of your rejection rule and see how that fares.

$\endgroup$
  • $\begingroup$ Thanks for your prompt answer. Let me try to provide context. I have several data, distributed in different ways (gamma, student, normal...you name it, depending on various factors). These data are scores from a survey done to assess their riskiness. With this data, I need to figure out a way to decide what's not as risky and what's very risky. I do not want to use any distribution/standard deviation method b/c the scores are widely apart, I want the outliers in and distribution is messy. I thought that the Sn would give me a reference to start building percentiles from there. Thoughts? $\endgroup$ – Bernardo Jan 27 '14 at 18:51
  • $\begingroup$ Your comment doesn't help. Could you edit your question to explain how you plan to use the Sn to " start building percentiles from there" --that part is really confusing. $\endgroup$ – user603 Jan 27 '14 at 18:54
  • $\begingroup$ Just to clarify, my goal is not to identify outliers. My goal is to keep all data in and decide the following: given the scores I have, what is a low score and what is a high score? What should earn the tag of high risk and what should earn the tag of low risk? Given that the scores are far too apart for the mean to make any sense, i figured that the Sn could tell me what's a good value to use as a yeardstick. If you think that's wrong, I'd love to hear any suggestions so I can start looking them up. Thank you!!! Updating Question now. $\endgroup$ – Bernardo Jan 27 '14 at 18:56
  • $\begingroup$ " i figured that the Sn could tell me what's a good value to use as a yeardstick" But how? Assume Sn(your dataset) gives a value of 4.23. How do you plan to use this number as a yardstick. Please edit your question to answer this (preferably with a formula). $\endgroup$ – user603 Jan 27 '14 at 19:01
  • 1
    $\begingroup$ Ok thank you very much. I will mark this as resolved and open a new thread asking for a suggested methodology. $\endgroup$ – Bernardo Jan 27 '14 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.