3
$\begingroup$

I have time series data. One observation per time period, about 10 observations or so. When describing the data (I am interested in descriptive statistics), I'd like to say that there is either a "trend up", "trend down", or "no trend".

I'd like to have a general algorithm for determining if there is a trend. There should be good citations to support the use of this algorithm. I need to say "I am using technique X" and give a citation. This is an important part of my question. You can easily roll your own — what I need is something that's well established and citable and has a specific name.

"What exactly do we mean by trend?" you might ask. I do not have a clear answer to this. So this question is open to some interpretation. Interpret away.

Here is what I've thought of so far.

Joinpoint. Calculate a bunch of different piecewise regressions. Pick one. Calculate a bunch of different piecewise regressions. Pick one.

This seems like huge overkill. Specialized software. Lots of different regression models. Multiple hypothesis testing. Assumes the trend has a specific functional form — in other words, it's not descriptive. On the plus side, this is citable.

I just need a simple descriptive statistic.

Mann-Kendall test. I could use the Mann-Kendall test, comparing the latest observation to all the previous ones. Find the comparison with the lowest p-value. If $p-value < .05$, say there is a trend from that point on.

Problems. Hypothesis testing. Multiple hypothesis testing — I don't know how to correctly adjust the critical level in this case.

Correlation coefficient. I could just look at $r$ and say that there is a trend if $|r| > a$. I think this would be a bad approach in general, since not all trends are linear.

Effect size. Compare the latest observation with the lowest / highest observation in the series. If the effect size from this comparison is large enough, then I say that there is a trend.

This method is dangerous. It works well if the trend is "well-behaved". But if there is an outlier that is either very high or very low, it might give a bad result.

To fix this, instead of comparing to the highest / lowest observation, you could just compare to the observation $n$ time period ago. Say, latest observation versus the one 10 periods ago. Again, this would work well for "well-behaved" trends, but I could see potential problems here.

Integrate effect size. I think this is an improvement on the above. Compare the latest observation with all the previous ones. Assign a score to each comparison: if the effect size > $a$, score = +1; if the effect size < $-a$, score = -1; otherwise, score = 0. Add up the scores, and do something with that.

I could even vary the absolute value of the scores. When comparing with a recent observation, the absolute value of the score could be high; when comparing with an old observation, it could be low.

Smoothing. Because I have so few data points, I am cautious about smoothing or filters, though I have not looked into it properly.

|cite|improve this question
$\endgroup$
  • $\begingroup$ This "integrate effect size" idea is similar in concept to Common language effect size en.wikipedia.org/wiki/Effect_size#Common_language_effect_size . $\endgroup$ – Jessica Apr 22 '15 at 20:29
  • 1
    $\begingroup$ Since your objective is description, it is difficult to see why you would be concerned with correcting for multiple hypothesis testing. You can overcome the objection to effect size (it is not robust) by using a robust estimator, such as Theil-Sen. There is such an admixture of competing concerns expressed in this question (which, in itself, is understandable) that one is left wondering what, precisely, you mean by "descriptive statistics." Perhaps you could elaborate on that? $\endgroup$ – whuber♦ Apr 22 '15 at 22:45
  • 1
    $\begingroup$ Thanks @whuber. I looked up Theil-Sen, very interesting. By "descriptive statistics" I mean that I just want to make statements about the data itself. I do not want to be making inferences about future data. $\endgroup$ – Jessica Apr 23 '15 at 16:07
  • 1
    $\begingroup$ If you are an R user, or are willing to try it out, take at look at stl (seasonal decomposition by Loess). This approach might give you some ideas to help you narrow your options. $\endgroup$ – whuber♦ Apr 23 '15 at 16:17
3
$\begingroup$

First, I would suggest to have a look at the book of P. Brockwell and R. Davis, Introduction to Time Series and Forecasting, 2002, second edition, Springer, New-York. The first chapter deals with trend estimation, and tests for stationarity. It is not too technical and full of examples and the first chapter is really easy. If you need a more technical reference, you can have a look at Time Series: theory and methods, 1991, second edition, Springer, New-York, by the same authors. That is for the citation part of your question.

If you consider a time series $\{X_t\}$, you can always decompose it into a trend, a seasonal component and a (stationary) random noise, that is you can write $X_t = m_t + s_t + Y_t$, where $m_t$ is the trend component, $s_t$ is the seasonal component and $Y_t$ is a stationary series. First, I would suggest to test the stationarity of the observed series. You can use different tests to do so, based for instance on sample correlations. If there is evident deviation from stationarity, you could try to estimate a trend. There are many ways to do it. You've mentioned filtering, but you can also think about polynomial fitting (linear regression of your series using $t, t^2, t^3,\dots$ as explanatory variables) or exponential fitting (linear regression on the log of your observations with $t$ as explanatory variable). Finally, once you've done your trend estimation, test the stationarity of your residuals. If the residuals are not stationary you need some more sophisticated model for your trend.

You can use a blackbox algorithm to extract the trend and the seasonal component. As already mentioned before the stl function in R does the job, but at the price of a lack of interpretability (how the trend is extracted).

hope it helps!

|cite|improve this answer
$\endgroup$
1
$\begingroup$

In my opinion the answer to your question involves the one bullet point that apparently reflected your current minimum understanding i.e. Smoothing. Detecting a trend in a sequence of longitudinal vales requires a proper determination of the smoothing filter. Unusual values not representative of the typical auto-correlative structure need to be identified and their effect nullified. Unusual values may be one-time only pulses or seasonal pulses (which are in truth NOT unusual but systematic ). Furthermore before correctly identifying a reasonable filter/equation one needs to identify/remedy level shift changes(really intercept changes) in the history. Now that the data has been cleaned up one needs to use the auto-correlative structure to form an efficient ARIMA model. This ARIMA model may incorporate differencing operators and then may have a steady-state differential which could be classified as a trend. Alternatively the residuals from the ARIMA model may suggest/evidence a set of residuals which follow a linear trend suggesting the need for adding a deterministic series to your ARIMA model of the form 0,0,0,0,0,1,2,3,4,5 where a trend is suggested at period 6 . Of course there can be one or more of these trends e.g. x1=1,2,3,4,5,7,9,11,13,15,17 etc. which may be needed. I have programmed and have helped developed software which performs this tour-de-force. There is a 30 day free trial which could be useful to you in your work. The software is called AUTOBOX and I believe it's twofold trend detection procedures are unique in available software. Spline procedures do not work as the user has to pre-determine the number of splines and the length of each spline

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.