I have some data where I want to determine whether the shape of the probability distribution has changed compared to 10 years ago.

One example is that I have for various automobiles multiple measures of price at a given point in time from different used car dealers and online sellers.

make, model, year of manufacture, price

(there are further complications such as condition etc, but I've simplified the problem somewhat).

The data has a number of outliers. Is it possible to do a robust (to clarify, by robust I mean resistant to outliers) transformation that preserves the original distribution of values, but allows a comparison of probability distributions (with different means / sds) on a similar or identical scale such as [0,1].

Also, if I had an hypothesis such that maybe the distribution of prices had a different shape in the past. Is there any sensible way to combine the data from different makes and models so as to get a more accurate estimation of the shape of the distribution function, or is this completely nonsensical.

I can of course individually compare the same make, model, yr of manufacture but I'm looking at a large number of comparisons.


Of course, it is possible to rescale your multivariate data so as
to express it in some standardized basis. It is also possible to do that in a way that preserves the distribution of your data (read the correlations between your original variables). Finally it is also possible to do this standardization in a way that is not influenced by the outliers. Furthermore, you will get the added benefit that it will also reveal potential outliers as standing out from the main body of your data.

denote $\pmb X$ your $n$ by $p$ data matrix. Let $(\hat{\pmb\mu},\hat{\pmb\varSigma})$ be respectively, the robust estimate of location and scatter of $\pmb X$. You can then simply transform your data as

$$\pmb Z=\hat{\pmb\varSigma}^{-1/2}(X-\pmb1_n'\hat{\pmb\mu})$$

where $1_n$ is a $n$-vector of ones and $\pmb D^{-1/2}$ denotes the matrix inverse square root of $\pmb D$. This transformation preserves the distribution of your data, standardizes it and makes the outliers stand out.

To compute $(\hat{\pmb\mu},\hat{\pmb\varSigma})$, there are several algorithms, but they all do essentially the same thing. The simplest, oldest is the FastMCD. You will find an R (open source), good implementation of it (as well as that of some other competitors) here. Make sure to read the vignette: it's very well done.

You can of course also use $(\hat{\pmb\mu},\hat{\pmb\varSigma})$ to derive a measure of outlyingness in $[0,1]$ associated with each observations (telling you how much that observation departs, in the multivariate space, from the bulk of the data). The way to compute such an outlyingness index is to do:

$$\text{Out}(\pmb x_i)=\left(1+(\pmb x_i-\hat{\pmb\mu})'\hat{\pmb\varSigma}^{-1}(\pmb x_i-\hat{\pmb\mu})\right)^{-1}$$

where $\pmb x_i$ is the i-th observations. You can use this index as a form of center outward ranking of your data. Of course, this measure will also not be influenced by the outliers and so you can also use $\text{Out}(\pmb x_i)$ to reveal them (e.g. the outliers will have stand out by having a very low values of $\text{Out}(\pmb x_i)$).

  • $\begingroup$ Thanks for your comment. Actually I was originally thinking about the univariate case, but I should research and think some more about the multivariate approach. By the way what is the matrix D ? I've probably overlooked some keypoint. Thanks. $\endgroup$ Aug 22 '13 at 10:08
  • $\begingroup$ Sure. Just know that the 'one variable at a time' approach changes the correlation between the variables. You can use bivariate plots to convince you of this. If the correlation structure is meaningful to you (presumably it is, otherwise why collect all these variables?) I wouldn't standardize each variable independently. if $D$ is a given matrix, $D^{-1/2}$ is is the square root inverse of $D$ (I’ve seen people interpret $D^{-1/2}$ as an element-wise operator on the entries of $D$ so I write explicitly what it mean, in case). $\endgroup$
    – user603
    Aug 22 '13 at 10:18

The only standardizations that ensure the same limits are those based on the minimum and maximum such as (value $-$ minimum) / (maximum $-$ minimum) but these would usually be considered far from robust, if robust here means resistant to outliers, which is far from the only sense.

(value $-$ mean) / SD is very commonly used, but again that is not robust in the usual sense. (value $-$ median) / IQR is more robust.

You may be working with some other definition of robust, in which case you would need to spell it out here.

Your example of a hypothesis, "the distribution of prices had a different shape in the past", is rather general.

On a variety of grounds, I suspect that your problem would be simpler if you used a quite different approach and looked at logarithm of price. That should pull in outliers. That's not completely arbitrary because it is so common to talk of prices in general rising (sometimes falling) by some percentage, which corresponds to a shift of log price by the corresponding constant. Much here depends on whether you, and especially your likely readership, are comfortable with thinking on a logarithmic scale.

An alternative used by Howard Wainer for car prices is the reciprocal. An accessible summary of his article can be found at http://mahalanobis.twoday.net/stories/908248/

I'd expect that looking at a variety of graphs (histograms or even better quantile plots) would help to see what kinds of changes there have been to the distribution over time.

  • $\begingroup$ Thanks for your suggestions. I was thinking about doing something like a Z-score but with the median and Median Absolute Deviation, but I'm not sure. I clarified the question so that by robust I mean insensitive to outliers. I agree the hypothesis is rather vague ! $\endgroup$ Aug 20 '13 at 1:37
  • $\begingroup$ MAD will probably work in about the same way as IQR. Expect MAD to be about half IQR, but skewness could mess that up. $\endgroup$
    – Nick Cox
    Aug 20 '13 at 12:46
  • $\begingroup$ @Antonio2100: if you do univariate $z$ scores, you change the multivariate shape of your data. To preserve the multivariate shape of your data you should do a multivariate standardization (see my answer). $\endgroup$
    – user603
    Aug 20 '13 at 15:53
  • $\begingroup$ On the use of the log transform to 'pull the outliers'. This doesn't solve the problem. See the discussion here for more info. $\endgroup$
    – user603
    Aug 20 '13 at 15:59
  • $\begingroup$ @user603 Like you, I probably didn't answer the question in quite the way the OP may have expected. I take the real problem as being to understand the data and I suspect that transforming the data is just as likely to be as useful, and easier for some people to understand, compared with what you propose. Your solution is very interesting but this comment seems a little dogmatic to me. $\endgroup$
    – Nick Cox
    Aug 20 '13 at 16:07

No, it is not possible to do a robust transformation that preserves the original distribution of values. This is because any robust transformation (e.g., Winsorizing) will necessarily change the distribution (e.g., if the highest and lowest parameters are removed, then this changes both the shape of the distribution and the degree of variation).

A simple way of achieving your desired outcome is, within each group that you wish to compute the distribution, compute z = floor(5 * (x - min) / (max - min + 0.000001)) and then compare the proportion of observations in each of the 5 categories. Of course you could use more or fewer than 5 categories if you so wanted.

If you were concerned about a small number of very extreme observations you could first Winsorize the data.

  • $\begingroup$ The question is about standardization, not transformation. I'm happy to regard standardizations as linear transformations, but you need to make a case that the reverse is true, namely that transformations such as the Winsorizing you name are standardizations. I'd suggest that's at odds with typical usage in statistics. $\endgroup$
    – Nick Cox
    Aug 20 '13 at 15:30
  • $\begingroup$ Note that I do propose a transformation in my answer. That's presented as an alternative, not a standardization. $\endgroup$
    – Nick Cox
    Aug 20 '13 at 16:09
  • $\begingroup$ Thanks for pointing out the broken link. The question asks about 'transformation' not standardization. And, standardization of a variable doesn't really fix problems with the shape of the distribution anyway. My proposed method is essentially a robust empirical estimate of the distribution function, so it feels pretty much in line with the typical usage in statistics to me, but there is always a gulf of guessing between what the person asks and what we answer. ;) $\endgroup$
    – Tim
    Aug 28 '13 at 4:39
  • $\begingroup$ I think we can agree that the desiderata stated by the OP are contradictory: preserving the original distribution of values and transformation are antagonistic. $\endgroup$
    – Nick Cox
    Aug 28 '13 at 7:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.