4
$\begingroup$

I have a list of numbers and I am trying to detect a drastic point difference from the previous numbers and see if the pattern has changed. For instance,

x = [5000,5500,6250,4800,3950,7200,5500,800,1200,900,500,400,300,200]

Above, there is high spending until 800 and then it seems that there is high spending before the 800 and low spending after the 800. All in all after 800, the spending has decreased a good bit. I want to try and divide the list based on this drastic point and then check if there is a different pattern (i.e. there is high spending before and low spending after or if there is low spending before and low spending after or low spending before high spending after or high spending before and high spending after). Essentially, I am looking for this sort of inflection point and trying to detect if there are two different classes of numbers. I know I could set a threshold and check each number or assume a normal distribution and check standard deviations. Is there a better way to approach this problem from a more statistical point of view? Note that the numbers could be of larger scale, purchases could be tens of thousands of dollars and then drop to thousands of dollars and this should indicate a change. I'm wondering if there is some statistical based method that is better than using standard deviations and means.

For a set of numbers like x above the following does not work well

def reject_outliers(data, m = 2.):
    d = np.abs(data - np.median(data))
    mdev = np.median(d)
    s = d/mdev if mdev else 0.
    return data[s<m]
$\endgroup$
  • 3
    $\begingroup$ Possible duplicate of How can I group numerical data into naturally forming "brackets"? (e.g. income) $\endgroup$ – Nick Cox Jun 30 '17 at 17:16
  • $\begingroup$ I'd consider working on log scale here, with a trap for zeros, e.g. ln(1 + spending). $\endgroup$ – Nick Cox Jun 30 '17 at 17:16
  • $\begingroup$ I am intersted in a more time series approach i.e. the first k numbers before the inflection point compared to the last n numbers after the inflection point. Does the Jenks Natural Break take this into account, I am not familiar with it $\endgroup$ – turnerRocker Jun 30 '17 at 17:20
  • 1
    $\begingroup$ That's a distinction without a difference. For the method you need to have a series of measurements in sequence but it's immaterial whether the sequence is in time. That point is made in the first paragraph of my answer. FWIW, the terminology Jenks natural break is not one I use and it's in my view one to avoid as W.D. Fisher had the idea much earlier and "natural break" is a loaded term. Also, inflexions are not what the method is looking for, rather changes in level. $\endgroup$ – Nick Cox Jun 30 '17 at 17:23
  • $\begingroup$ Are there any other approaches for flat list data? $\endgroup$ – turnerRocker Jun 30 '17 at 18:27
4
$\begingroup$

You describe a circumstance in which spending might vary around a constant level (under the null hypothesis) or change suddenly to vary around a second level at some point during the series. One way to measure this is to compare the variance in spending under the two models. In the first case it would be the mean squared difference between spending and the overall average. In the second case, the differences would be relative to the two levels of spending for the best possible model: that is, among all possible points where the series could break, choose the one where the magnitude of the effect is largest. Let's call the resulting mean squared difference the "two-level variance."

A permutation test can do a fine job of assessing the data. Its null hypothesis is that the sequence of the data doesn't matter. The sampling distribution of the two-level variance therefore is obtained by examining all possible reorderings of the data and computing its value for each one. In practice there are so many reorderings that we sample from them randomly, thereby approximating the sampling distribution. The data will show significant evidence of a break when it is rare for the reordered data to exhibit a two-level variance that is as low (or lower) than the two-level variance of the data themselves.

Here, in the histogram at left, is an approximate permutation distribution of the two-level variance for the data given in the question. The vertical dotted red line marks the value of the two-level variance for the actual data. The p-value is very low because no permuted versions of the data exhibited a smaller two-level variance.

Figure

The plot at the right shows the (obvious) break where the symbols change between indexes 7 and 8.

Here is R code to reproduce (or extend) these calculations. It requires about one second to compute ten thousand values of the permutation distribution--more than enough for practical work. (The computation of the test statistic can be optimized and made to scale directly with the amount of data.)

#
# The two-level variance.
#
stat <- function(y) {
  r2 <- function(i) {
    r1 <- function(z) crossprod(z - mean(z))
    (r1(y[1:i]) + r1(y[-(1:i)])) / length(y)
  }
  r <- sapply(1:length(y), r2)
  i <- which.min(r)
  return(list(Value=r[i], Index=i))
}
#
# The data.
#
y <- c(5000,5500,6250,4800,3950,7200,5500,800,1200,900,500,400,300,200)
#
# Approximate the permutation distribution of the two-level variance.
#
set.seed(17)
sim <- replicate(1e4, stat(sample(y))$Value)
#
# Compute the two-level variance of the data, then compute its p-value.
#
stat.data <- stat(y)
sim <- c(sim, stat.data$Value)
p.value <- mean(sim <= stat.data$Value)
#
# Plot the results.
#
par(mfrow=c(1,2))
hist(sim, xlab="Statistic", main="Permutation Distribution")
abline(v=stat.data$Value, col="red", lwd=2, lty=3)
plot(y, main="Estimated Break", sub=paste("p =", signif(p.value, 3)))
points(y[1:stat.data$Index], pch=16)
par(mfrow=c(1,1))
$\endgroup$
  • $\begingroup$ x = [5000,200,6250,4800,100,7200,5500,800,1200,900,500,400,300,200] how would this sequence perform ? where two anomolous points existed in the first group . $\endgroup$ – IrishStat Jul 5 '17 at 21:00
  • $\begingroup$ @Irish That's a good example. A break is identified at the same location as before, but now (using 100,000 permutations) $p = 0.077$. $\endgroup$ – whuber Jul 5 '17 at 21:21
  • $\begingroup$ x = [5000,5500,6250,4800,3950,7200,5500,800,1200,6900,500,6400,300,200] two pulses possibly masking the level shift . In my experience you can;t ignore pulses while trying to detect a "break point" a.k.a. a "level shift" a.k.a. an "intercept change" . $\endgroup$ – IrishStat Jul 6 '17 at 0:34
  • $\begingroup$ would this (r) solution be easily extended to detect "break" based on correlation, instead of variance? $\endgroup$ – Maximilian Jul 11 '17 at 7:55
  • $\begingroup$ @Maximilian Yes, this is a very general approach. My reply at stats.stackexchange.com/a/59875/919 gives an account of applying permutation tests based on any statistic. $\endgroup$ – whuber Jul 11 '17 at 13:12
1
$\begingroup$

You could use a sequential t-test as described at the link below and implemented in Visual Basic for Application (Excel).

http://www.beringclimate.noaa.gov/regimes/

screenshot screenshot_2

$\endgroup$
  • $\begingroup$ a t test premises independent observations with no pulses , time trends and/or seasonal pulses . $\endgroup$ – IrishStat Jul 6 '17 at 16:36
  • $\begingroup$ +1 This appears to be a legitimate test, designed to detect level shifts and shifts in variance. Thank you for the reference and the example. @IrishStat appears to be referring to something altogether different (a Student t test), so although he is correct about its inapplicability, his comment does not apply to this answer. $\endgroup$ – whuber Jul 11 '17 at 13:16
  • $\begingroup$ It says "F-test" not "T-test" in the link you provided. $\endgroup$ – Tom Reilly Jul 12 '17 at 11:46
  • $\begingroup$ @TomReilly The "F-test" is for detecting shifts in variance. Both methods are discussed at the link. $\endgroup$ – Nat Jul 12 '17 at 21:40
0
$\begingroup$

You could consider k-means clustering.

Within group sum of squares

Clusters

# Data
x = c(5000,5500,6250,4800,3950,7200,5500,800,1200,900,500,400,300,200)

# Calculate within-group sum of squares for different number of clusters
n <- length(x)
wss <- rep(0, 6)
wss[1] <- (n - 1)*var(x)
for (i in 2:6) {
  wss[i] <- sum(kmeans(x, centers = i)$withinss)
}

# Plot WGSS vs number of groups
plot(1:6,
     wss,
     type = "b",
     xlab = "Number of groups",
     ylab = "Within groups sum of squares")

# Perform k-means clustering
clusters <- kmeans(x, centers = 2, iter.max = 1e4, nstart = 1e2)

# Create results data frame
xClustered <- data_frame(i = 1:length(x), x = x) %>% 
  mutate(cluster = factor(clusters$cluster))

# Plot clusters
ggplot(data = xClustered, aes(x = i, y = x, color = cluster, shape = cluster)) +
  geom_point() +
  theme(axis.text.x = element_text(angle = 90, hjust = 1)) +
  labs(x = "index", y = "x", color = "cluster", fill = "cluster")
$\endgroup$
  • 1
    $\begingroup$ (1) How do you use this approach to determine the statistical significance of a result? (2) How does it account for the sequence of values? You seem only to be clustering the values themselves, regardless of sequence--but that's not what a "changed spending pattern" means. $\endgroup$ – whuber Jul 5 '17 at 17:57
-1
$\begingroup$

Look to the work of Tsay and Balke on the idea of a "level shift". It is a deterministic variable that looks like this 0,0,0,0,0,0,0,1,1,1,1,1,1,etc.

Here are results using Autobox (I am affiliated with this company). An outlier was detected at period 6 and a level shift down 4,635 at period 8.

Y(T) = 5250.0 test
+[X1(T)][(- 4635.7 )] :LEVEL SHIFT 8 +[X2(T)][(+ 1950.0 )] :PULSE 6 + + [A(T)]

When running the tso() package it also confirms these results. enter image description here

$\endgroup$
  • 1
    $\begingroup$ Detecting an "outlier" in just a dozen data, together with using two parameters, seems assured to be over-fitting. The "pulse" at time 6 clearly is over-interpreting these data. Your "level shift" is not "deterministic," because the point at which the zeros become ones is stochastic and has to be estimated from the data. $\endgroup$ – whuber Jul 11 '17 at 13:14
  • 1
    $\begingroup$ (1) No, any decent test places the sixth point well inside. For instance, to use a crude but very conservative outlier test, a two-sided 95% Normal prediction interval based on points 1..5 and 7 (which is the relevant population before the level shift occurs) has an upper limit of 7331, exceeding the value of 7200. (2) The null hypothesis would be "is there a level shift," not "is there a level shift after the seventh value." The latter would amount to p-hacking, because it constructs the null based upon observing the data. One needn't consult any papers to appreciate these points. $\endgroup$ – whuber Jul 11 '17 at 14:01
  • $\begingroup$ After the detecting the level shift at time period 8, the recomputed two local means with the first mean reflecting the first 7 values and the second mean reflecting the last 7 values. The consequential pooled standard deviation was small enough to suggest a pulse at period 6. This was a close call to identify it, but once it was brought in it was found to be significant at the 99% level. 3 coefficients with 14 observed values is not overfitting in my view. The 0/1 variables are fixed variables and not stochastic and are all based on the work I cited by Tsay and Balke. $\endgroup$ – Tom Reilly Jul 11 '17 at 14:04
  • 1
    $\begingroup$ You are erring in computing the significance. Among other things, it is invalid to include the very value you wish to test! You must exclude it from the calculation of mean and SD. Clearly the 0/1 are not fixed by the question itself; if you think they are, then that introduces additional error to your tests of significance. $\endgroup$ – whuber Jul 11 '17 at 14:10
  • $\begingroup$ Period 5 is almost an outlier as well. $\endgroup$ – Tom Reilly Jul 11 '17 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.