Analyzing up/down patterns in short time-series data

Question

I have not worked very frequently with time-series data, so am looking for some pointers as to how best proceed with this particular question.

Let's say I have the following data - graphed below:

Here there is year on the x-axis. The y-axis is a measure of 'inequality' e.g. it could be inequality in income in a country.

For this question, I am interested in asking whether there is an up/down nature to the data year on year (for want of a better description). In essence, I would like to ask whether if the inequality went up last year from the year previous to that, is it now likely to go back down? The size of ups/downs may be important to factor also.

I am thinking that something like wavelet analysis or Fourier analysis may help, though I have not used these before and I believe that a sample size such as this is too small.

Would be interested in any ideas/suggestions for me to follow up on.

EDIT:

These are the data for this chart:

#   year     value
#1  1956 0.9570912
#2  1957 1.0303563
#3  1958 0.9568302
#4  1959 1.1449074
#5  1960 0.8962963
#6  1961 1.0431552
#7  1962 0.8050077
#8  1963 0.8533181
#9  1964 0.9971713
#10 1965 1.0453083
#11 1966 0.8221328
#12 1967 1.0594876
#13 1968 1.1244195
#14 1969 1.0705498
#15 1970 0.8669457
#16 1971 0.8757319
#17 1972 1.0815189
#18 1973 1.1458959
#19 1974 1.2782848
#20 1975 1.0729718
#21 1976 1.1569416
#22 1977 1.2063673
#23 1978 1.1509700
#24 1979 1.1172020
#25 1980 1.0691429
#26 1981 1.0907407
#27 1982 1.1753854
#28 1983 0.9440187
#29 1984 1.1214175
#30 1985 1.2777778
#31 1986 1.2141739
#32 1987 0.9481722
#33 1988 1.1484652
#34 1989 0.7968458
#35 1990 1.1721074
#36 1991 1.1569523
#37 1992 0.8160300
#38 1993 0.9483291
#39 1994 1.0898612
#40 1995 0.8196819
#41 1996 1.0297017
#42 1997 1.0207769
#43 1998 0.9720285
#44 1999 0.8685848
#45 2000 0.9228595
#46 2001 0.9171540
#47 2002 1.0470085
#48 2003 0.9313437
#49 2004 1.0943982
#50 2005 1.0248419
#51 2006 0.9392917
#52 2007 0.9666248
#53 2008 1.1243693
#54 2009 0.8829184
#55 2010 0.9619517
#56 2011 1.0030864
#57 2012 1.1576998
#58 2013 0.9944945

Here they are in R format:

structure(list(year = structure(1:58, .Label = c("1956", "1957", 
"1958", "1959", "1960", "1961", "1962", "1963", "1964", "1965", 
"1966", "1967", "1968", "1969", "1970", "1971", "1972", "1973", 
"1974", "1975", "1976", "1977", "1978", "1979", "1980", "1981", 
"1982", "1983", "1984", "1985", "1986", "1987", "1988", "1989", 
"1990", "1991", "1992", "1993", "1994", "1995", "1996", "1997", 
"1998", "1999", "2000", "2001", "2002", "2003", "2004", "2005", 
"2006", "2007", "2008", "2009", "2010", "2011", "2012", "2013"
), class = "factor"), value = c(0.957091237579043, 1.03035630567276, 
0.956830206830207, 1.14490740740741, 0.896296296296296, 1.04315524964493, 
0.805007684426229, 0.853318117977528, 0.997171336206897, 1.04530832219251, 
0.822132760780104, 1.05948756976154, 1.1244195265602, 1.07054981337927, 
0.866945712836124, 0.875731948296804, 1.081518931763, 1.1458958958959, 
1.27828479729065, 1.07297178130511, 1.15694159981794, 1.20636732623034, 
1.15097001763668, 1.11720201026986, 1.06914289768696, 1.09074074074074, 
1.17538544689082, 0.944018731375053, 1.12141754850088, 1.27777777777778, 
1.21417390277039, 0.948172198172198, 1.14846524606799, 0.796845829569407, 
1.17210737869653, 1.15695226716732, 0.816029959161985, 0.94832907620264, 
1.08986124767836, 0.819681861348528, 1.02970169141241, 1.02077687443541, 
0.972028455959697, 0.868584838281808, 0.922859547859548, 0.917153996101365, 
1.04700854700855, 0.931343718539713, 1.09439821062628, 1.02484191508582, 
0.939291692822766, 0.966624816907303, 1.12436929683306, 0.882918437563246, 
0.961951667980037, 1.00308641975309, 1.15769980506823, 0.994494494494494
)), row.names = c(NA, -58L), class = "data.frame", .Names = c("year", 
"value"))

Answer 1

If the series is uncorrelated, unnecessarily taking differences injects auto-correlation . Even if the series is autocorrelated unwarranted differencing is inappropriate. Simple ideas and simple approaches often have unwanted side effects. The model identification process (ARIMA) starts with the original series and may result in differencing BUT it should never starts with unwarranted differencing unless there is a theoretical justification. If you wish you can post your short time series and I will use it to explain to you how to identify a model for this series.

After receipt of data:

The ACF of your data does not initially (or finally) indicate any ARIMA process here BOTH ACF and PACF

and here just ACF:
However there appears to be two level shifts in your data ... one at 1972 and the other at 1992 .. they appear to be nearly cancelling level shifts. A useful model might also include the incorporation of three unusual values at periods 1989,1959 and 1983. The equation then is
and here

with model statistics here:

The Actual/Fit and Forecast is here with the residual plot here suggesting model sufficiency . This is confirmed by the acf of the residuals . Finally the fit and the forecast summarizes the findings .
In summary the series ( probably a ratio) is without significant auto-regressive memory but does have some evidented deterministic structure (statistically significant). All models are wrong but some are useful (G.E.P. Box) .

After some discussion .. If one were to model differences then one would get the following model ... with ACTUAL/FIT and FORECAST . Forecasts look eerily similar ... the MA coefficient effectively cancels the differencing operator.

Answer 2

You can look at the up and downs as a random sequence, which is generated by some random process. For instance, let's assume that you're dealing with a stationary series $x_1,x_2,x_3,...,x_n \in f(x)$, where $f(x)$ is a probability distribution such as Gaussian, Poisson or anything else. This is stationary series. Now, you can create new variable $y_t$ such that $y_t=1:x_t<x_{t+1}$ and $y_t=0:x_t\ge x_{t+1}$, these are your ups and downs. This new sequence will form its own random sequence with interesting properties, see e.g. V Khil, Elena. "Markov properties of gaps between local maxima in a sequence of independent random variables." (2013).

For instance, look at ACF and PACF of your series. There's nothing here. This doesn't seem like ARIMA model. It looks like uncorrelated sequence of $x_t$.

This means that we could try applying known results for $y_t$, e.g. it's known that the average distance between two (up-down) pairs (or U-turns as some call them) is 3. In your data set the first peak (up-down) is 1957 and the last one is 2012, with 16 peaks in total. So, the average distance between peaks is 15/55=3.67. We know that the $\sigma=1.108$, and with 15 observations $\sigma_{15}=\sigma/\sqrt{15}=0.29$. So the mean distance between peaks is within $1.2\sigma_n$ from the theoretical mean.

UPDATE: on cycles

The graph in OP's question appears to suggest that there's some kind of long running cycle. There are several issues with this.

1. If you generate a random sequence, sometimes something like a cycle may appear just randomly. So, with 58 data points which are purely observational data, it's impossible to declare a cycle without some kind of economic theory behind it. Nobody's going to take it seriously without the economic reasoning. For that you definitely need exogenous variables, I'm afraid.
2. Check out this wonderful paper: The Summation of Random Causes as the Source of Cyclic Processes, Eugen Slutzky, Econometrica, Vol. 5, No. 2 (Apr., 1937), pp. 105-146. Basically, sometimes cycles are caused by some sort of MA process.
3. This could be an illusion. I use this trick quite often in presentations. I show the actual data, then draw lines, circles or arrows to mess with my audience's brains :) The extra lines trick the brain into seeing trends which may not be there at all, or to make them look much stronger than they actually are.

Answer 3

Aside 1: One thing we see is the appearance of a long cyclical trend in the data. This shouldn't affect the year-on-year analysis all that much* -- so for this very basic analysis I'll ignore that and treat the data as if they were homogeneous aside from the effect you're interested in.

*(it will tend to reduce the number of opposite direction movement from what you'd expect with homogeneity -- so it will tend to lower the power of this test somewhat. We could try to quantify that impact, but I don't think there's a strong need unless it seems likely to be big enough to make a difference -- if it's already significant, adjusting for something that would make the p-value a bit smaller would be a waste of effort.)

Aside 2: As expressed, your question seems to involve one-tailed alternative. I'll work on the basis that this is what you want.

Let's start with a simple analysis directed straight at your basic question, which seems to be along the lines of "is an increase more likely to be followed by a decrease?"

However, it's not as simple as it might first appear. In a stable series, with purely random data, an increase is more likely to be followed by a decrease. Note that the hypothesis we're considering involves three observations, which can be ordered in six possible ways:

Of those six ways, 4 involve a change in direction. So a purely random series (irrespective of the distribution) should see a flip in direction 2/3 of the time.

[This is closely related to a runs-up-and-down test, where you're interested in whether there are too many runs for it to be random. You could use that test instead.]

I assume your actual interest is in whether it's higher than the random 2/3 rather than whether it's more than 1/2 as you seemed to be asking.

$H_0:$ the series is "random"

$H_1:$ a shift up or down is more likely to be followed by a shift in the opposite direction than you'd see with a random series

Test statistic: proportion of shifts followed by shifts in the opposite direction.

Because our triples overlap, I believe we have some dependence between triples, so we can't treat this as binomial (we could if we split the data into non-overlapping triples; that would work fine).

Keeping that dependence in mind, we could still compute the distribution of the test statistic, but we don't need to in this case, because the observed proportion of direction reversed triples is just under the expected number of 2/3 for a random series, and we're only interested in more reversals than that.

So we don't need to compute any further -- there's no evidence at all of a tendency to reverse (up-down or down-up) more than you'd get with a random series.

[I really doubt the neglected mild cycle will have enough impact to move the expected proportion down anywhere near far enough for this to make a substantive difference.]

Answer 4

You could use a package called structural change which checks for breaks or level shifts in the data. I have had some success in automatically detecting level shifts for non-seasonal time series.

I converted your "value" into a time series data. and used the following code to check for level shifts or change points or break points. The package also has nice features such as chow test to do chow test to test for structural breaks:

require(strucchange)
value.ts <- ts(data[,2],frequency = 1, start = (1956))
bp.value <- breakpoints(value.ts ~ 1)
summary(bp.value)

Following is the summary from the breakpont function:

Breakpoints at observation number:

m = 1           36      
m = 2     16    31      
m = 3     16    36    46
m = 4     16 24 36    46
m = 5   8 16 24 36    46
m = 6   8 16 24 33 41 49

Corresponding to breakdates:

m = 1                  1991          
m = 2        1971      1986          
m = 3        1971      1991      2001
m = 4        1971 1979 1991      2001
m = 5   1963 1971 1979 1991      2001
m = 6   1963 1971 1979 1988 1996 2004

Fit:

m   0           1           2           3           4           5           6          
RSS   0.8599316   0.7865981   0.5843395   0.5742085   0.5578739   0.5559645   0.5772778
BIC -71.5402819 -68.5892480 -77.7080112 -70.6015129 -64.1544916 -56.2324540 -45.9296608

As you can see the function identified possible breaks in your data and selected two structural breaks at 1971 and 1986 as shown in the plot below based on BIC criterion. The function also provided other alternative break points as listed in the output above.

Hope this is helpful