1
$\begingroup$

I hope somebody in here can help me: I'm looking for some pointers as to how to distinguish random fluctuation from actual changes in trends, e.g.:

  • In a time series with measures taken at monthly (daily) intervals, for how many months (days) does the change have to be consistent (e.g. going from decreasing to increasing) before one can say that it is an actual change in trend and not just a random fluctuation?

  • How large (percentage-wise or in comparison with the variation) does the change have to be in order to be a real change and not just a random fluctuation?

Is there any any rules of thumb or hard rules on this?

Thanks!

EDIT: Example: If I have a time series with 60 observations, and the trend change (from decreasing to increasing, say) at observation 30, when will I be able to detect that it is a real change in the trend and not a random fluctuation? At observation 31? 32? 33? Something else?

r1 <- seq(1,30, by = 1)
r2 <- 1/r1
r3 <- sort(r2)
ex <- append(r2,r3)

   ex
[1] 1.00000000 0.50000000 0.33333333 0.25000000 0.20000000 0.16666667 0.14285714 0.12500000 0.11111111 0.10000000 0.09090909 0.08333333 0.07692308 0.07142857 0.06666667 0.06250000 0.05882353 0.05555556 0.05263158 0.05000000 0.04761905 0.04545455 0.04347826 0.04166667 0.04000000
[26] 0.03846154 0.03703704 0.03571429 0.03448276 0.03333333 0.03333333 0.03448276 0.03571429 0.03703704 0.03846154 0.04000000 0.04166667 0.04347826 0.04545455 0.04761905 0.05000000 0.05263158 0.05555556 0.05882353 0.06250000 0.06666667 0.07142857 0.07692308 0.08333333 0.09090909
[51] 0.10000000 0.11111111 0.12500000 0.14285714 0.16666667 0.20000000 0.25000000 0.33333333 0.50000000 1.00000000

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Do you have a data example? The following link may help if you use R: statmethods.net/advstats/timeseries.html $\endgroup$
    – Ruthger Righart
    Apr 14, 2015 at 11:53
  • $\begingroup$ I am using R :) $\endgroup$
    – SiKiHe
    Apr 14, 2015 at 12:41
  • 1
    $\begingroup$ You may be interested in this post. $\endgroup$
    – javlacalle
    Apr 14, 2015 at 21:09
  • $\begingroup$ @javlacalle: I don't think this question is about change detection and analysis in time series, but rather about trend analysis (please see my answer and comments). $\endgroup$
    – Aleksandr Blekh
    Apr 14, 2015 at 22:51
  • $\begingroup$ The question is exactly about change detection - hence my point that your answer below doesn't answer the question I asked! $\endgroup$
    – SiKiHe
    Apr 15, 2015 at 6:50

3 Answers 3

Reset to default
2
$\begingroup$

Expanding on @Aleksendr Blekh's answer. You can use the below code for decomposing trend, using the decompose function in R:

births <- scan("http://robjhyndman.com/tsdldata/data/nybirths.dat")
birthstimeseries <- ts(births, frequency=12, start=c(1946,1))
if(!require("TTR")) { install.packages("TTR");  require("TTR") }
birthstimeseriescomponents <- decompose(birthstimeseries)
birthstimeseriescomponents$seasonal # the seasonal component
birthstimeseriescomponents$trend # the trend component
birthstimeseriescomponents$random # random (irregular / remainder) component
# plot the seasonality, trend and random (epsilon) components
plot(birthstimeseriescomponents)

For changes in trends, relevant reading: Hidden Markov Models, Change Point Analysis.

This post may also be helpful to find fluctuations from the underlying trend. The approach here is fitting a polynomial instead of time series decomposition and reporting points that fall outside the confidence intervals.

Improve this answer
$\endgroup$
2
$\begingroup$

The multivariate time series tool mvtsplot may be useful to explore several time-series. This plot provides quick insight if values are high or low, before using other stats for examining trends and noise.

By default, the function uses three colors to divide the data in low, medium and high categories (purple=low values, grey=medium values, green=high values). This number can be changed using levels (for quartiles, use levels=4).

Expanding on your data I add a second example

r1 <- seq(1,30, by = 1)
r2 <- 1/r1
r3 <- sort(r2)
ex1 <- append(r2,r3)

ex2 <- c(1:60)^0.5

Plot to compare the series

plot(ex1, type="l", col="red", ylim=c(0,9))
lines(ex2, col="blue")

Plot using mvtsplot

library(mvtsplot)
dat<-as.matrix(cbind(ex1, ex2))
mvtsplot(dat, level=3, margin=TRUE)

enter image description here

MVTSPLOT

enter image description here

purple=low values grey=medium values green=high values.

Improve this answer
$\endgroup$
1
$\begingroup$

Basically, you have to perform trend analysis, which is time series exploratory technique, based on ARMA family of models, of which ARIMA is most likely the most popular one. However, for your purposes, I think that it might be enough to just perform time series decomposition, where, along with seasonality and cyclical pattern, trend is one of the main components. More details on time series decomposition as well as some examples can be found here. In regard to some existing rules of thumb for time series' minimum sample size, Prof. Rob J. Hyndman dismisses such guidelines as "mis­lead­ing and unsub­stan­ti­ated in the­ory or prac­tice" in this relevant blog post.

Improve this answer
$\endgroup$
4
  • $\begingroup$ This doesn't really answer my question, I'm afraid. I don't want to know the minimum number of observations I can fit a time series to, I want to know the minimum number of observations needed to distinguish a change in trends from a random fluctuation. $\endgroup$
    – SiKiHe
    Apr 14, 2015 at 12:30
  • $\begingroup$ @SiKiHe: I think that it does. The whole point of time series decomposition is to extract systematic components, i.e. trend, and ignore noise (random fluctuations). Citing from StatSoft textbook: "As in most other analyses, in time series analysis it is assumed that the data consist of a systematic pattern (usually a set of identifiable components) and random noise (error) which usually makes the pattern difficult to identify. Most time series analysis techniques involve some form of filtering out noise in order to make the pattern more salient.". (to be continued) $\endgroup$
    – Aleksandr Blekh
    Apr 14, 2015 at 22:45
  • $\begingroup$ @SiKiHe: (cont'd) Since you have to fit a model for time series decomposition to extract trend, my point about lack of valid rules of thumb for minimum sample size (# of observations), supported by referenced Prof. Rob J. Hyndman's blog post, still holds. $\endgroup$
    – Aleksandr Blekh
    Apr 14, 2015 at 22:48
  • $\begingroup$ STL-decomposition will do the same trick. Short of reading the R source code for decompose to extract the information on how they calculate the trend, the answer is useless. $\endgroup$
    – SiKiHe
    Apr 15, 2015 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.