Finding outliers on simple dataset?

Question

I'm pretty inexperienced in statistics and am wondering the best way to tackle a couple of simple questions.

I am given the following data on 1,000 hypothetical slot machines:

Attempts made, wins, payout, payout per attempt (win rate * payout)

Some slot machines have over 15,000 attempts, while others have as few as 50. Here is what I'm wondering:

1. What is the best way assess which machines have payout per attempt values that can be classified as outliers? Seeing as some machines have so few attempts.

2. How can I determine the statistical significance of a given payout per attempt value when comparing it to a specific number? For example if as a gambler I want to target machines with higher than X payout per attempts?

I am working with these values in Excel also. Thank you very much!

Answer 1

Assuming roughly identical payouts, about half of the machines will provide "above average" paybacks. If you increase your criteria from "above average" to "top 5%," then 5% or 50 will be in the top 5%; 10 will be the top 1% of paybacks. This isn't particularly insightful for the would-be gambler. Note that even if all the machines are genuinely identical, some will have to perform better than others, but those won't necessarily provide better results for a future game.

Another way to consider this problem is to calculate a mean payout for all 1000 machines over all tens, hundreds, thousands, etc of trials. The overall average based on this very large amount of data is probably pretty reliable. Then you can use a statistical test on each machine's payout and test for a mean response better than the overall mean.

Suppose that there are only two outcomes – you get all your money back plus 0.8 times your money half the time, and nothing the other half. The average payout is 0.5 x 1.8 + 0.5 x 0 = 0.9. Further, suppose that you observed a payout of ratio of 1.26 for a machine with 10 trials (i.e. 7 wins). What is the probability that the machine’s win ratio is really only 0.5 and that this level of winning was mere luck? The probability of winning at least 7 times with a 0.5 win rate is 17% sum(10 choose k, k = 7..10)/2^10, interesting, but hardly statistical proof. A payout ratio of 1.26 over 50 trials (i.e. 35 wins) is only about 0.3% likely, but this is still likely to happen in about 3 of the 1000 different machines. Moving the payout up to 1.44 (80% sample win rate), the odds start to drop off faster. 80% on 50 trials with a machine would only happen with probability 1.2x10^(-5) or 0.0012%, probably indicating a good one to hit. As you can see it takes a good bit of suspect data to be confident in this kind of outlier.

Or you could give up the slots and lose your money in the stock market like a “responsible” adult :)

Answer 2

You might try to detect the outliers using a robust method. If you are willing to assume that a majority (at least 50+$\epsilon$% of your data) is from the same distribution, then you can use this approach. In doing so, you will have to assume that there is no non-random difference between the behavior of the slot machines in the majority group: the outlier machines can behave in any strange way you like.

Many robust techniques exist, but multivariate ones tend to be computationally intensive, and even univariate ones can be complicated. Since you are restricted to Excel I propose the following.

If you want to find outliers for winrate*payout (denote this $\mathbf{X}$), and your majority is symmetrically distributed, you can use the univariate version of the Minimum Covariance Determinant (MCD). The multivariate version is more complicated, but the univariate can be obtained as follows. The goal will be to find a subset of $n/2 < h<n$ observations that are "clean" that has the smallest variance (so it is the tightest). Since the outliers are a minority, any set of $h$ points must have at least some good observations. Therefore, if the outliers are far enough from the good data, the variance of a set of outliers plus some good values totally $h$ observations should tend to have a higher variance than an $h$-subset from the good data.

1. Decide on a value of $h$ that seems reasonable to you, or perform this analysis with a range of $h$ values and see what looks reasonable.
2. Sort your values of $\mathbf{X}$ from smallest to largest. The reason you can do this instead of trying all combinations of your observations is that the $h$-subset with the smallest variance must be a contiguous set of the sorted observations. This simplification is why you can do it in Excel.
3. Compute a rolling variance of sets of $h$ observations.
4. Select the $h$-subset with the smallest covariance. This should have no outliers.
5. After you have an $h$ subset, you can define the 97.5% confidence interval around the mean of the observations in your $h$-subset and include other observations in this interval as good data. This is called reweighting.