0
$\begingroup$

I have a data set with the dollars in transactions a store has done historically from noon-2pm on Wednesdays. Most of the time, the store does no volume during this time (16 observations with the value of 0), but when it does one or more transactions, since the minimum price for an item is around 400, those observations occupy a second mode around $800. What are some approaches one might take to estimate normal variability in such a situation with an eye towards using that approach for detecting outliers?

For the domain context, given this store's volume, I would roughly consider anything below $1700 normal variation, anything above 3000 as a significant outlier, and anything in between slightly anomalous, but not full fledged outliers. I'd like to find an approach that would allow me to accurately represent normal variability and outliers for a wide range of store sizes and activity levels that fundamentally follow a similar distribution.

I've tried using the standard deviation, but this doesn't seem to capture what I'd like to consider normal deviations vs. outlying deviations. For example, I would want to represent values close to the right hand mode as normal deviations from the mean, but the point estimates for mean and standard deviation return z-scores that are > 1 for new observations that fall around the right mode.

One approach I considered was just to use the non-zero right mode values to calculate the mean and standard deviation to calculate z-scores from, but that has the disadvantage of not incorporating information from the left side of the distribution. For example, for samples with relatively more 0 value observations, the probability of experiencing an extreme value should be lower than for a sample with relatively fewer 0 value periods.

My ultimate desire is to be able to observe new values in subsequent Wednesday noon-2pm periods and determine whether that new observation falls within what could be considered normal population variation vs. is an outlier.

Any suggestions are greatly appreciated!

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Just looking at the graph, I would classify 0 as an outlier. I would not remove the 0s from the data. I would hardly ever remove outliers; it is the special cases that contain the most information. You just need to make sure that the information is genuine and not some mistake occurring during data collection or cleaning.

Moreover, you need to make sure that you appropriately use that information. How to find out whether the information in outliers is genuine and what appropriately means depends on the context, which you did not give us.

Improve this answer
$\endgroup$
10
  • $\begingroup$ +1. With marked bimodality it is entirely likely, but not really a problem, that calculating the mean and SD will not add insight: you can see that by looking at and thinking about your histogram. Almost always there is, and otherwise there should be, a substantive story that explains bimodality as marked as you show -- you have sheep and goats mixed, or non-smokers and smokers, or whatever -- and it's that story and your substantive goals that drive what to do. As Maarten underlines, there is no context here to help us further. $\endgroup$
    – Nick Cox
    Feb 11, 2016 at 9:08
  • $\begingroup$ Thanks very much Maarten and Nick for your feedback. I've updated my question to provide better context. $\endgroup$
    – Eddie
    Feb 11, 2016 at 18:16
  • $\begingroup$ @Eddie Thanks, but what do you want to do with that variable? Do you just want to look at it or use it as an explained/dependent/left-hand-side/y-variable in some regression, or as explanatory/independent/right-hand-side/x-variable? $\endgroup$
    – Maarten Buis
    Feb 12, 2016 at 8:14
  • $\begingroup$ Thanks again @MaartenBuis for pushing me to further specify my objectives here. My ultimate desire is to be able to observe new values in subsequent two hour periods and determine whether that new observation falls within what could be considered normal population variation vs. is an outlier. In some ways you could consider this a regression problem where the transaction amount of a new observation is the dependent variable for which we want a reasonable confidence interval about what that new observation is likely to be (presumably n = chronological time period as the independent variable). $\endgroup$
    – Eddie
    Feb 12, 2016 at 18:26
  • $\begingroup$ @MaartenBuis The approach I was implying in my question would be more like a z-score - subtracting the expected value of a new observation from a newly observed value under evaluation and dividing by a measure of common variability to generate some kind of outlier score for the new observation. $\endgroup$
    – Eddie
    Feb 12, 2016 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.