# Using standardization methods to exclude erroneous outlier values

As my data has a lot of outliers, using mean to standardize data doesn't seem to be optimal.

I'm experimenting with using median to classify outliers and stumbled upon robStandardize function from robustHD package.

What I struggle is how to intuitively explain the numbers produces by this kind of standardization.

For example, it's quite intuitive if we scale by mean and sd and the corresponding z-score tells us how far deviated is a value from its population mean, with standard deviation as a unit.

Although with median and median absolute deviation (mad), it's harder to interpret it.

Based on the standardization technique using median and mad, I want to classify outliers and potentially exclude them from regression analysis.

EDIT: As suggested by mkt in the comment, it'd be helpful to have some context. I'm dealing with e-commerce transaction prices. So each record will have the product SKU along with the transacted price. Throughout the year, the price of the different SKUs may vary over time which can be due to many events such as promotional period, clearing out inventory, etc.

I would expect the price variation is anchored around the normal price. For example, in the case of promotional period, sellers would at most give say 30% discount. For some reasons, prices may also increase should the sellers feel increasing price doesn't have impact on sales and profit.

My end objective is to calculate the demand curve based on number of sales or transactions at different price point. I would then use this demand curve to derive the optimal price for which they can maximize profit.

I have hundreds of SKUs and each SKU has their own price range. The challenge is that some of these prices are really just errors which I consider outliers. The outliers will have impact on the demand curve, and therefore may not capture accurate information. For example, based on the demand curve I may calculate the price-demand elasticity of the SKU. However if these outliers were to be introduced, they inevitably produce unexpected results.

Example below, I have profit-price curves for each different segments (represented by color lines) for the same product.

Business users would expect, while there maybe some variations in optimal recommended prices for each segment due to different demand curves, the variations shouldn't be varied by too much. In this case, we have optimal price for segment E (blue line) to be at USD269 but for segment B (green line) at USD118. Segment E has high outliers while segment B has low outliers.

The expected range of the price should have been around USD180-USD200. Notice that segment A (red line) and segment D (yellow line) are somewhat ok as there're no erroneous transacted prices for these two segments.

Thanks

• It's rarely a good idea to exclude outliers. Especially if you think you have lots of them. Can you explain why you are trying to do this?
– mkt
Oct 8, 2019 at 12:01
• One method is familiar to you and one isn’t. I don’t know otherwise what you mean by “intuitive”. Median +/- MAD from median correspond to the quartiles of any symmetric distribution: does that help? By the way, I am no fan of outlier detection heuristics of this kind, but that’s a bigger story, Oct 8, 2019 at 12:06
• @mkt, I see your point but in my context, these outliers are simply errors. For example, let's say the median price of a product is USD100. It's not useful to include outliers with price USD3000 for the same product. Even USD300 should be considered as outliers as it's not normal for people to pay the same product 3x the normal price. I want to exclude them so that these outliers would have no impact on my pricing optimization which require some demand calculation at different price point. Oct 8, 2019 at 14:09
• You should have some substantive criterion for deleting observations you don't care about. Otherwise using some threshold level of (value $-$ median) / MAD will just bite you. Prices so variable might be better off looked at on logarithmic scale. Otherwise I am not an economist. Oct 8, 2019 at 14:16
• "It's not useful to include outliers with price USD3000 for the same product. Even USD300 should be considered as outliers as it's not normal for people to pay the same product 3x the normal price." The danger here is that you have not articulated any clear criterion to distinguish what you consider an error from the remaining points. Why do you have errors that are 10X, 3X or 0.1X of the 'regular price', whatever that is? Editing this information into your question would help us give you good advice. Without knowing this, many of us are leery of offering guidance because it may be poor.
– mkt
Oct 8, 2019 at 14:49

First, the median divided by the MAD is really no less intuitive than the mean divided by the SD. It's a little less common but "how far from the median" is just as clear as "how far from the mean". Neither SD nor MAD is very intuitive, but, if anything, I'd say MAD is more so. No squaring and square rooting to deal with.

Second, unless the method of recording prices of SKUs is really terrible (in which case there could be lots of errors) if you have a "lot" of outliers then I don't think they are really outliers. I think they are an indication that you are missing something. And, if the method of recording prices is so bad, then you have other problems. There will be other errors in the data that just aren't so clear.

Third, rather than deleting outliers (especially a lot of them) I think you should use methods that deal with them well. There is a whole field of robust statistics.

Fourth, the exception is when the data are clearly wrong. Some values are impossible. There are no 4 meter tall human beings, for instance. But, unless you have some information that you haven't put in your question, all you have is a sort of suspicion that prices shouldn't vary as much as yours do. You are surprised. One definition of an outlier is a "surprising point". That leads to:

Fifth: My favorite professor in grad school used to say "if you're not surprised, you haven't learned anything". Don't throw the surprise away. Figure out where it comes from.

• Thanks Peter for your detailed answer. For different context, these outliers are still useful and actually they are being used as part of the overall analysis. However for this pricing optimization activity, they are clearly not at all. Maybe it wasn't explicit in the original question but these outliers were made clear by the data owner that they are simply errors as our vendor used to manually input the recommended price. In some cases, we made a lost due to these transactions. I guess that's the 'suprising' part after all :). Oct 9, 2019 at 12:08
• (+1) This is my take too, and at the risk of sounding like a broken record, I will add, yet again, that prices are often better considered on logarithmic scale. Oct 9, 2019 at 12:14
• @NickCox I believe I've explored log scale for this but ended up not using it. If I remember it doesn't solve the outliers impact on the demand curve. Let me see if I can get back with the actual reasons. Oct 9, 2019 at 12:19
• @AfiqJohari If you have that many errors that result in outliers, you probably have a bunch that result in "inliers". Oct 10, 2019 at 11:07
• @PeterFlom thanks, learn something new today about 'inliers' :). At this point, I'm resolving this by asking the data owner on price range so that we can apply business rules to identify pricing error. I think it's more efficient that way for the time being. Potentially for more sophisticated application, may develop a mix of business rules and statistical validation to clean these outliers before using the data as input. Thanks again. Oct 11, 2019 at 1:42

There seems to be a hangup in the comments over "outlier removal," which is often done without thinking and can be a bad idea. You're just looking for error detection--removing errors, I agree, is perfectly reasonable in this context, because they don't inform demand at all, which is what you're interested in.