Detect outliers / detect classes

Question

Currently I have a dataset that contains several products with different prices and quantities. My goal is to detect if the given product was sold as a package or as a unit and I have used the Mahalanobis distance on the prices and a Chi squared distribution to identify the unit/package separation. An example would be :

product       price  quantity
cigarette       $1      1
cigarette      $10      1
chips          $5       1
chips          $5       1
chips         $4.5      1
chips          $5       1

In this example it would be clear that the cigarette at \$1 is a unit, and the cigarette with price \$10 would be a package. However I have cases such as the chips where the prices are "concentrated" around certain value (ex. \$5) and if I have thousands of those values then a price of \$4.5 would be detected as an outlier(false unit) when it is probable that someone could have sold a bag of chips for \$4.5 (maybe they were about to expire)

Answer 1

If it seems reasonable to assume that prices are normally distributed you can use gaussian mixture modeling to identify the clusters in your data. Consider the following example of chip sales.

Let's say that chips are sold in packs of 1, 5, or 20 with associated average costs of \$1, \$4, $10 respectively. You don't know this exactly and you want to separate sales into clusters based on the size of the pack sold. There is also variation in the price that each vendor charges so that the price for any unit size is normally distributed with some variance which is not equal across unit-sizes.

set.seed(777)
chips_1  = rnorm(n = 15, mean = 1,  sd = .25) # Sample of prices for 1-packs
chips_5  = rnorm(n = 30, mean = 4,  sd = 1.0) # Sample of prices for 5-packs
chips_20 = rnorm(n = 8,  mean = 10, sd = 1.0) # Sample of prices for 20-pack

chips = c(chips_1, chips_5, chips_20)

hist(chips, breaks = 12, freq = F, 
     main = "Histogram of Prices for Chips", 
     xlab = "Price ($)")
lines(density(chips, adjust = .5), col = "blue")

Using the mclust library we can fit a gaussian mixture model to the data. First, we compare the fits with different numbers of clusters via BIC.

library(mclust)
BIC <- mclustBIC(chips)
plot(BIC)

This plot indicates that the data is best explained by a mixture of three normal distributions. We can estimate the respective means and variances of these three distributions.

fit = Mclust(chips, G=3, model="V") 
summary(fit, parameters =T)
plot(fit, what="density", lwd = 2, xlab = "Price ($)")

The estimated parameters are summarized below. Additionally, every point in this example was correctly classified according to the distribution from which it was drawn. The fit$classification parameter details the predicted classifications.

$$\begin{array}{c|c|c|} & \text{Estimated} & \text{Actual} \\ \hline \text{Mean 1} & 1.039 & 1.000 \\ \hline \text{Mean 2} & 3.952 & 4.000 \\ \hline \text{Mean 3} & 10.37& 10.00 \\ \hline \text{Variance 1} & 0.050& 0.063 \\ \hline \text{Variance 2} & 1.149 & 1.000 \\ \hline \text{Variance 3} & 1.540 & 1.000 \\ \hline \end{array}$$