3
$\begingroup$

Is there any robust methodology to identify outliers in the discrete data distribution. I am specifically concerned with discrete geometrical distribution? P.S. Data transformation does not seem to work effectively.

enter image description here

$\endgroup$
  • 1
    $\begingroup$ Can you show us some plots. $\endgroup$ – user2974951 Feb 15 at 10:29
  • $\begingroup$ Please refer this link to see the kind of plot I am talking about. mathworld.wolfram.com/GeometricDistribution.html $\endgroup$ – Vivek Feb 15 at 10:52
  • 1
    $\begingroup$ The plot of your data! Not the geometric distribution. $\endgroup$ – user2974951 Feb 15 at 11:01
  • $\begingroup$ I have added the plot in the question. $\endgroup$ – Vivek Feb 15 at 12:10
  • 1
    $\begingroup$ The data is highly skewed and the x-axis extends to more than 100 counts. The likeliness of the data decreases as we move to the right. Is there any method which captures this mathematically like mean+3*std for a normal distribution.Also, Median is 1 in this case. $\endgroup$ – Vivek Feb 15 at 12:44
1
$\begingroup$

Just for fun, I'll add an answer expanding my comments with the idea of fitting a distribution, and then using either a plot or a chi-square goodness-of-fit test to see if the observed data differ from the proposed distribution. I'm not sure in what cases using this approach would tell you something practically important. And I'm pretty sure this isn't what the OP wanted as a method to identify "outliers".

Unfortunately, the answer relies pretty heavily on the code. But here's what I'm doing.

I made up some data: 214 observations of categories 1 to 8. And then, I fit a negative binomial distribution to these data.

Here, the plot may be the most valuable tool to see where the observed data differ from the proposed distribution. Category 1 stands out as the observed frequency is quite higher than the proposed. To me, Category 8 also stands out in that the frequency was 8 when we were expecting a frequency of 0 or 1.

enter image description here

We could also look at the standardized residuals from a chi-square goodness-of-fit test to see where the observed frequencies differ from the proposed. Because the sample size is relatively large, there will be several categories with a standardized residual greater than 1.96 or less than -1.96. So instead of using the 1.96 cutoff, we can look at where the absolute value of the standardized residuals are largest. Here again, it's for Categories 1 and 8 where the absolute values of the residuals are large.

###         1          2          3          4          5          6          7          8 
### 9.1635916 -3.5005318 -5.2217855 -2.8954578 -1.6837577 -0.4168148 -0.3410554 10.9815772 

R code:

Data = read.table(header=T, text="
Transactions  Frequency
 1             140
 2              40
 3              10
 4               8
 5               4
 6               3
 7               1
 8               8
")

Long = Data[rep(row.names(Data), Data$Frequency), "Transactions"]

library(fitdistrplus)

FD = fitdist(Long, "nbinom")

plot(FD)

TD = rnegbin(1e5, mu=FD$estimate[2], theta=FD$estimate[1])

TD1 = TD[TD > 0 & TD < 9]

Theoretical = prop.table(table(TD1))

Theoretical

   ###           1           2           3           4           5           6           7           8 
   ### 0.353693748 0.299581840 0.184562923 0.093152628 0.042960972 0.017597571 0.006110613 0.002339705 

Observed    = table(Long)

Observed

   ###   1   2   3   4   5   6   7   8 
   ### 140  40  10   8   4   3   1   8 

CST = chisq.test(Observed, p=Theoretical)

CST$stdres

   ###         1          2          3          4          5          6          7          8 
   ### 9.1635916 -3.5005318 -5.2217855 -2.8954578 -1.6837577 -0.4168148 -0.3410554 10.9815772 
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.