As my data has a lot of outliers, using mean to standardize data doesn't seem to be optimal.
I'm experimenting with using median to classify outliers and stumbled upon robStandardize function from robustHD package.
What I struggle is how to intuitively explain the numbers produces by this kind of standardization.
For example, it's quite intuitive if we scale by mean and sd and the corresponding z-score tells us how far deviated is a value from its population mean, with standard deviation as a unit.
Although with median and median absolute deviation (mad), it's harder to interpret it.
Based on the standardization technique using median and mad, I want to classify outliers and potentially exclude them from regression analysis.
EDIT: As suggested by mkt in the comment, it'd be helpful to have some context. I'm dealing with e-commerce transaction prices. So each record will have the product SKU along with the transacted price. Throughout the year, the price of the different SKUs may vary over time which can be due to many events such as promotional period, clearing out inventory, etc.
I would expect the price variation is anchored around the normal price. For example, in the case of promotional period, sellers would at most give say 30% discount. For some reasons, prices may also increase should the sellers feel increasing price doesn't have impact on sales and profit.
My end objective is to calculate the demand curve based on number of sales or transactions at different price point. I would then use this demand curve to derive the optimal price for which they can maximize profit.
I have hundreds of SKUs and each SKU has their own price range. The challenge is that some of these prices are really just errors which I consider outliers. The outliers will have impact on the demand curve, and therefore may not capture accurate information. For example, based on the demand curve I may calculate the price-demand elasticity of the SKU. However if these outliers were to be introduced, they inevitably produce unexpected results.
Example below, I have profit-price curves for each different segments (represented by color lines) for the same product.
Business users would expect, while there maybe some variations in optimal recommended prices for each segment due to different demand curves, the variations shouldn't be varied by too much. In this case, we have optimal price for segment E (blue line) to be at USD269 but for segment B (green line) at USD118. Segment E has high outliers while segment B has low outliers.
The expected range of the price should have been around USD180-USD200. Notice that segment A (red line) and segment D (yellow line) are somewhat ok as there're no erroneous transacted prices for these two segments.