2
$\begingroup$

As my data has a lot of outliers, using mean to standardize data doesn't seem to be optimal.

I'm experimenting with using median to classify outliers and stumbled upon robStandardize function from robustHD package.

What I struggle is how to intuitively explain the numbers produces by this kind of standardization.

For example, it's quite intuitive if we scale by mean and sd and the corresponding z-score tells us how far deviated is a value from its population mean, with standard deviation as a unit.

Although with median and median absolute deviation (mad), it's harder to interpret it.

Based on the standardization technique using median and mad, I want to classify outliers and potentially exclude them from regression analysis.

EDIT: As suggested by mkt in the comment, it'd be helpful to have some context. I'm dealing with e-commerce transaction prices. So each record will have the product SKU along with the transacted price. Throughout the year, the price of the different SKUs may vary over time which can be due to many events such as promotional period, clearing out inventory, etc.

I would expect the price variation is anchored around the normal price. For example, in the case of promotional period, sellers would at most give say 30% discount. For some reasons, prices may also increase should the sellers feel increasing price doesn't have impact on sales and profit.

My end objective is to calculate the demand curve based on number of sales or transactions at different price point. I would then use this demand curve to derive the optimal price for which they can maximize profit.

I have hundreds of SKUs and each SKU has their own price range. The challenge is that some of these prices are really just errors which I consider outliers. The outliers will have impact on the demand curve, and therefore may not capture accurate information. For example, based on the demand curve I may calculate the price-demand elasticity of the SKU. However if these outliers were to be introduced, they inevitably produce unexpected results.

Example below, I have profit-price curves for each different segments (represented by color lines) for the same product.

Business users would expect, while there maybe some variations in optimal recommended prices for each segment due to different demand curves, the variations shouldn't be varied by too much. In this case, we have optimal price for segment E (blue line) to be at USD269 but for segment B (green line) at USD118. Segment E has high outliers while segment B has low outliers.

The expected range of the price should have been around USD180-USD200. Notice that segment A (red line) and segment D (yellow line) are somewhat ok as there're no erroneous transacted prices for these two segments.

enter image description here

Thanks

$\endgroup$
  • 5
    $\begingroup$ It's rarely a good idea to exclude outliers. Especially if you think you have lots of them. Can you explain why you are trying to do this? $\endgroup$ – mkt - Reinstate Monica Oct 8 at 12:01
  • 3
    $\begingroup$ One method is familiar to you and one isn’t. I don’t know otherwise what you mean by “intuitive”. Median +/- MAD from median correspond to the quartiles of any symmetric distribution: does that help? By the way, I am no fan of outlier detection heuristics of this kind, but that’s a bigger story, $\endgroup$ – Nick Cox Oct 8 at 12:06
  • 1
    $\begingroup$ @mkt, I see your point but in my context, these outliers are simply errors. For example, let's say the median price of a product is USD100. It's not useful to include outliers with price USD3000 for the same product. Even USD300 should be considered as outliers as it's not normal for people to pay the same product 3x the normal price. I want to exclude them so that these outliers would have no impact on my pricing optimization which require some demand calculation at different price point. $\endgroup$ – Afiq Johari Oct 8 at 14:09
  • 1
    $\begingroup$ You should have some substantive criterion for deleting observations you don't care about. Otherwise using some threshold level of (value $-$ median) / MAD will just bite you. Prices so variable might be better off looked at on logarithmic scale. Otherwise I am not an economist. $\endgroup$ – Nick Cox Oct 8 at 14:16
  • 3
    $\begingroup$ "It's not useful to include outliers with price USD3000 for the same product. Even USD300 should be considered as outliers as it's not normal for people to pay the same product 3x the normal price." The danger here is that you have not articulated any clear criterion to distinguish what you consider an error from the remaining points. Why do you have errors that are 10X, 3X or 0.1X of the 'regular price', whatever that is? Editing this information into your question would help us give you good advice. Without knowing this, many of us are leery of offering guidance because it may be poor. $\endgroup$ – mkt - Reinstate Monica Oct 8 at 14:49
4
$\begingroup$

First, the median divided by the MAD is really no less intuitive than the mean divided by the SD. It's a little less common but "how far from the median" is just as clear as "how far from the mean". Neither SD nor MAD is very intuitive, but, if anything, I'd say MAD is more so. No squaring and square rooting to deal with.

Second, unless the method of recording prices of SKUs is really terrible (in which case there could be lots of errors) if you have a "lot" of outliers then I don't think they are really outliers. I think they are an indication that you are missing something. And, if the method of recording prices is so bad, then you have other problems. There will be other errors in the data that just aren't so clear.

Third, rather than deleting outliers (especially a lot of them) I think you should use methods that deal with them well. There is a whole field of robust statistics.

Fourth, the exception is when the data are clearly wrong. Some values are impossible. There are no 4 meter tall human beings, for instance. But, unless you have some information that you haven't put in your question, all you have is a sort of suspicion that prices shouldn't vary as much as yours do. You are surprised. One definition of an outlier is a "surprising point". That leads to:

Fifth: My favorite professor in grad school used to say "if you're not surprised, you haven't learned anything". Don't throw the surprise away. Figure out where it comes from.

$\endgroup$
  • 1
    $\begingroup$ Thanks Peter for your detailed answer. For different context, these outliers are still useful and actually they are being used as part of the overall analysis. However for this pricing optimization activity, they are clearly not at all. Maybe it wasn't explicit in the original question but these outliers were made clear by the data owner that they are simply errors as our vendor used to manually input the recommended price. In some cases, we made a lost due to these transactions. I guess that's the 'suprising' part after all :). $\endgroup$ – Afiq Johari Oct 9 at 12:08
  • 1
    $\begingroup$ (+1) This is my take too, and at the risk of sounding like a broken record, I will add, yet again, that prices are often better considered on logarithmic scale. $\endgroup$ – Nick Cox Oct 9 at 12:14
  • $\begingroup$ @NickCox I believe I've explored log scale for this but ended up not using it. If I remember it doesn't solve the outliers impact on the demand curve. Let me see if I can get back with the actual reasons. $\endgroup$ – Afiq Johari Oct 9 at 12:19
  • $\begingroup$ @AfiqJohari If you have that many errors that result in outliers, you probably have a bunch that result in "inliers". $\endgroup$ – Peter Flom - Reinstate Monica Oct 10 at 11:07
  • $\begingroup$ @PeterFlom thanks, learn something new today about 'inliers' :). At this point, I'm resolving this by asking the data owner on price range so that we can apply business rules to identify pricing error. I think it's more efficient that way for the time being. Potentially for more sophisticated application, may develop a mix of business rules and statistical validation to clean these outliers before using the data as input. Thanks again. $\endgroup$ – Afiq Johari Oct 11 at 1:42
1
$\begingroup$

There seems to be a hangup in the comments over "outlier removal," which is often done without thinking and can be a bad idea. You're just looking for error detection--removing errors, I agree, is perfectly reasonable in this context, because they don't inform demand at all, which is what you're interested in.

Here are three ways of going about this.

  1. You can just a priori decide what counts as an error for each product (i.e. no more than 50%-150% of "normal price," or median price if you don't know the "normal price). If you have a lot of substantive knowledge this is probably a reasonable approximate solution. Definitely do some sensitivity analysis for your cutoffs if you do this.
  2. You can model the distribution of prices parametrically and look for points with low p-values. For example, you could assume a Gaussian distribution. Then, if you have 100 data points, you might decide that anything with a p-value (the probability, under the estimated distribution, of a point as or more extreme appearing) less than, say, 1/10,000, is an error. You have to choose this threshold, again, based on substantive knowledge. Honestly, unless you have a lot of data and know a lot about how prices are distributed, this could be misleading (i.e. Gaussian vs. Cauchy vs. skewed distributions will give very different results). I'd probably do 1.
  3. If you have a large number of points labeled as true prices or errors, you could try to learn a model for which prices are errors. I'm guessing you don't have this data, and this would be a ton of work anyway. Again, 1. is probably the best option.
$\endgroup$
  • 2
    $\begingroup$ This is all very reasonable but it seems highly likely that #1 is the only game here. Unfortunately, the flavour of "you just need to devise your own criteria if you are determined to do this, and be careful" is about as much as can be said in generality. That's my "hangup" in comments here and elsewhere, and answers elsewhere: people do seem to want objective criteria that are guaranteed to work to weed out suspect data, but there is a bundle of good reasons why such criteria are elusive if not impossible. $\endgroup$ – Nick Cox Oct 9 at 12:11
  • $\begingroup$ Thanks Sheridan, I think 1. is the best options after all although not statistically 'sexy' enough if I can call it that way. That range of % can be further confirmed by the subject matter expert. $\endgroup$ – Afiq Johari Oct 9 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.