# Analyzing up/down patterns in short time-series data

I have not worked very frequently with time-series data, so am looking for some pointers as to how best proceed with this particular question.

Let's say I have the following data - graphed below:

Here there is year on the x-axis. The y-axis is a measure of 'inequality' e.g. it could be inequality in income in a country.

For this question, I am interested in asking whether there is an up/down nature to the data year on year (for want of a better description). In essence, I would like to ask whether if the inequality went up last year from the year previous to that, is it now likely to go back down? The size of ups/downs may be important to factor also.

I am thinking that something like wavelet analysis or Fourier analysis may help, though I have not used these before and I believe that a sample size such as this is too small.

Would be interested in any ideas/suggestions for me to follow up on.

EDIT:

These are the data for this chart:

#   year     value
#1  1956 0.9570912
#2  1957 1.0303563
#3  1958 0.9568302
#4  1959 1.1449074
#5  1960 0.8962963
#6  1961 1.0431552
#7  1962 0.8050077
#8  1963 0.8533181
#9  1964 0.9971713
#10 1965 1.0453083
#11 1966 0.8221328
#12 1967 1.0594876
#13 1968 1.1244195
#14 1969 1.0705498
#15 1970 0.8669457
#16 1971 0.8757319
#17 1972 1.0815189
#18 1973 1.1458959
#19 1974 1.2782848
#20 1975 1.0729718
#21 1976 1.1569416
#22 1977 1.2063673
#23 1978 1.1509700
#24 1979 1.1172020
#25 1980 1.0691429
#26 1981 1.0907407
#27 1982 1.1753854
#28 1983 0.9440187
#29 1984 1.1214175
#30 1985 1.2777778
#31 1986 1.2141739
#32 1987 0.9481722
#33 1988 1.1484652
#34 1989 0.7968458
#35 1990 1.1721074
#36 1991 1.1569523
#37 1992 0.8160300
#38 1993 0.9483291
#39 1994 1.0898612
#40 1995 0.8196819
#41 1996 1.0297017
#42 1997 1.0207769
#43 1998 0.9720285
#44 1999 0.8685848
#45 2000 0.9228595
#46 2001 0.9171540
#47 2002 1.0470085
#48 2003 0.9313437
#49 2004 1.0943982
#50 2005 1.0248419
#51 2006 0.9392917
#52 2007 0.9666248
#53 2008 1.1243693
#54 2009 0.8829184
#55 2010 0.9619517
#56 2011 1.0030864
#57 2012 1.1576998
#58 2013 0.9944945


Here they are in R format:

structure(list(year = structure(1:58, .Label = c("1956", "1957",
"1958", "1959", "1960", "1961", "1962", "1963", "1964", "1965",
"1966", "1967", "1968", "1969", "1970", "1971", "1972", "1973",
"1974", "1975", "1976", "1977", "1978", "1979", "1980", "1981",
"1982", "1983", "1984", "1985", "1986", "1987", "1988", "1989",
"1990", "1991", "1992", "1993", "1994", "1995", "1996", "1997",
"1998", "1999", "2000", "2001", "2002", "2003", "2004", "2005",
"2006", "2007", "2008", "2009", "2010", "2011", "2012", "2013"
), class = "factor"), value = c(0.957091237579043, 1.03035630567276,
0.956830206830207, 1.14490740740741, 0.896296296296296, 1.04315524964493,
0.805007684426229, 0.853318117977528, 0.997171336206897, 1.04530832219251,
0.822132760780104, 1.05948756976154, 1.1244195265602, 1.07054981337927,
0.866945712836124, 0.875731948296804, 1.081518931763, 1.1458958958959,
1.27828479729065, 1.07297178130511, 1.15694159981794, 1.20636732623034,
1.15097001763668, 1.11720201026986, 1.06914289768696, 1.09074074074074,
1.17538544689082, 0.944018731375053, 1.12141754850088, 1.27777777777778,
1.21417390277039, 0.948172198172198, 1.14846524606799, 0.796845829569407,
1.17210737869653, 1.15695226716732, 0.816029959161985, 0.94832907620264,
1.08986124767836, 0.819681861348528, 1.02970169141241, 1.02077687443541,
0.972028455959697, 0.868584838281808, 0.922859547859548, 0.917153996101365,
1.04700854700855, 0.931343718539713, 1.09439821062628, 1.02484191508582,
0.939291692822766, 0.966624816907303, 1.12436929683306, 0.882918437563246,
0.961951667980037, 1.00308641975309, 1.15769980506823, 0.994494494494494
)), row.names = c(NA, -58L), class = "data.frame", .Names = c("year",
"value"))

• A very simple idea: how about taking the time series of differences and looking at the auto correlation coefficient(s)? – psarka Nov 23 '14 at 16:54
• The data are now revealed as inequality measured using Lorenz Asymmetry. See OP's comment below the answer by @IrishStat. But what's the nature of the smoother curve? Being coy about what you are showing usually just makes the question more cryptic, not usefully more general. – Nick Cox Nov 24 '14 at 9:06
• @jalapic fourier analysis on a short observational sample is almost always pointless. You may pick a cycle at 25 years wave length, but it's going to be very weak. You barely have one full cycle at best. Fourier was designed for physical data where you have repeatable or at least long series. – Aksakal Nov 24 '14 at 13:57
• You realise that the up down nature you describe fits well with the idea of regression to the mean. I.e. You will have that up down behaviour even if your series does not have memory in the arima sense. – Corone Nov 28 '14 at 9:44

If the series is uncorrelated, unnecessarily taking differences injects auto-correlation . Even if the series is autocorrelated unwarranted differencing is inappropriate. Simple ideas and simple approaches often have unwanted side effects. The model identification process (ARIMA) starts with the original series and may result in differencing BUT it should never starts with unwarranted differencing unless there is a theoretical justification. If you wish you can post your short time series and I will use it to explain to you how to identify a model for this series.

After receipt of data:

The ACF of your data does not initially (or finally) indicate any ARIMA process here BOTH ACF and PACF

and here just ACF:
However there appears to be two level shifts in your data ... one at 1972 and the other at 1992 .. they appear to be nearly cancelling level shifts. A useful model might also include the incorporation of three unusual values at periods 1989,1959 and 1983. The equation then is
and here

with model statistics here:

The Actual/Fit and Forecast is here with the residual plot here suggesting model sufficiency . This is confirmed by the acf of the residuals . Finally the fit and the forecast summarizes the findings .
In summary the series ( probably a ratio) is without significant auto-regressive memory but does have some evidented deterministic structure (statistically significant). All models are wrong but some are useful (G.E.P. Box) .

After some discussion .. If one were to model differences then one would get the following model ... with ACTUAL/FIT and FORECAST . Forecasts look eerily similar ... the MA coefficient effectively cancels the differencing operator.

• Many thanks for the suggestions. I'd be very interested in hearing more - I've added the raw data to the question. – jalapic Nov 23 '14 at 19:24
• @IrishStat suggested a model with 6 parameters. The entire data set is 58 observations. This makes it 10 observations per parameter. Even if this was not an economic data, I'd say that a sample is too small to support 6 parameter model. Since this is economic model, I'll say that pure time series model is not going to work. You need exogenous variables (GDP?) or some kind of structural model. – Aksakal Nov 23 '14 at 21:17
• The model could be easily simplified to ignore the three aberrant points (pulses). This would reduce to 3 coefficients .. a constant and two level shift indicators ... couldn't be simpler ! and would still provide a reasonable representation ... two level shifts. I agree that a pure time series model is not going to work. The model I presented is simply an ordinary regression model with two level shift indicators ... there is no memory thus it is not classically speaking a time series model as there is no ARIMA structure in the model as it is unwarranted. – IrishStat Nov 23 '14 at 21:29
• One can't cry "I need more variables" , although my experience tells me that this is so. This problem is what it is ! Solve it ! – IrishStat Nov 23 '14 at 21:37
• @aksakal, what irishstat model shows is it has 5 variables, if parsimony is of concern, you could simply turn off the pulses and just keep the level shifts which would address the op question. I know of no other methedology besides autobox that efficiently does this type of level shift detection prior to selecting a time series model – forecaster Nov 23 '14 at 22:29

You can look at the up and downs as a random sequence, which is generated by some random process. For instance, let's assume that you're dealing with a stationary series $x_1,x_2,x_3,...,x_n \in f(x)$, where $f(x)$ is a probability distribution such as Gaussian, Poisson or anything else. This is stationary series. Now, you can create new variable $y_t$ such that $y_t=1:x_t<x_{t+1}$ and $y_t=0:x_t\ge x_{t+1}$, these are your ups and downs. This new sequence will form its own random sequence with interesting properties, see e.g. V Khil, Elena. "Markov properties of gaps between local maxima in a sequence of independent random variables." (2013).

For instance, look at ACF and PACF of your series. There's nothing here. This doesn't seem like ARIMA model. It looks like uncorrelated sequence of $x_t$.

This means that we could try applying known results for $y_t$, e.g. it's known that the average distance between two (up-down) pairs (or U-turns as some call them) is 3. In your data set the first peak (up-down) is 1957 and the last one is 2012, with 16 peaks in total. So, the average distance between peaks is 15/55=3.67. We know that the $\sigma=1.108$, and with 15 observations $\sigma_{15}=\sigma/\sqrt{15}=0.29$. So the mean distance between peaks is within $1.2\sigma_n$ from the theoretical mean.

UPDATE: on cycles

The graph in OP's question appears to suggest that there's some kind of long running cycle. There are several issues with this.

1. If you generate a random sequence, sometimes something like a cycle may appear just randomly. So, with 58 data points which are purely observational data, it's impossible to declare a cycle without some kind of economic theory behind it. Nobody's going to take it seriously without the economic reasoning. For that you definitely need exogenous variables, I'm afraid.
2. Check out this wonderful paper: The Summation of Random Causes as the Source of Cyclic Processes, Eugen Slutzky, Econometrica, Vol. 5, No. 2 (Apr., 1937), pp. 105-146. Basically, sometimes cycles are caused by some sort of MA process.
3. This could be an illusion. I use this trick quite often in presentations. I show the actual data, then draw lines, circles or arrows to mess with my audience's brains :) The extra lines trick the brain into seeing trends which may not be there at all, or to make them look much stronger than they actually are.
• In 45 months on this board I have rarely commented about an answer BUT in this case your answer leaves me with a furrowed brow and speechless. Can you please explain your model , your tests of significance , tests regarding your assumptions , , a forecast with confidence intervals and a logical conclusion. – IrishStat Nov 23 '14 at 21:56
• This is not my model. The idea is simple: if you have uncorrelated sequence of $x_t$, then you can create another sequence from it, such as $y_t$, which will have 1 if $x_t$ went up compared to previous value, and 0 if it went down. In this case [1,0] pair in $y_t$ will indicate a local maximum (peak) in $x_t$, such as in years 1957 and 1959. The rest is easy, you can come up with all sorts of stats for $y_t$ sequence. I thought about this because OP has "up/down" in the title of question. The sequence $y_t$ has many interesting markovian features. – Aksakal Nov 23 '14 at 22:03
• IMHO his up/down was not period to period but over a longer horizon. Thanks for your explanation. There is nothing in his data to suggest an up/down sequence that is predictable. – IrishStat Nov 23 '14 at 22:08
• They are not predictable in this model. They are random but correlated – Aksakal Nov 23 '14 at 22:12
• I STRONGLY disagree they are uncorrelated ...within each of the three regimes : 1956-1972 .... 1973-1991 ... 1992-2013 ...given that you adjust/modify the three unusual points/readings . – IrishStat Nov 23 '14 at 22:20

Aside 1: One thing we see is the appearance of a long cyclical trend in the data. This shouldn't affect the year-on-year analysis all that much* -- so for this very basic analysis I'll ignore that and treat the data as if they were homogeneous aside from the effect you're interested in.

*(it will tend to reduce the number of opposite direction movement from what you'd expect with homogeneity -- so it will tend to lower the power of this test somewhat. We could try to quantify that impact, but I don't think there's a strong need unless it seems likely to be big enough to make a difference -- if it's already significant, adjusting for something that would make the p-value a bit smaller would be a waste of effort.)

Aside 2: As expressed, your question seems to involve one-tailed alternative. I'll work on the basis that this is what you want.

Let's start with a simple analysis directed straight at your basic question, which seems to be along the lines of "is an increase more likely to be followed by a decrease?"

However, it's not as simple as it might first appear. In a stable series, with purely random data, an increase is more likely to be followed by a decrease. Note that the hypothesis we're considering involves three observations, which can be ordered in six possible ways:

Of those six ways, 4 involve a change in direction. So a purely random series (irrespective of the distribution) should see a flip in direction 2/3 of the time.

[This is closely related to a runs-up-and-down test, where you're interested in whether there are too many runs for it to be random. You could use that test instead.]

I assume your actual interest is in whether it's higher than the random 2/3 rather than whether it's more than 1/2 as you seemed to be asking.

$H_0:$ the series is "random"

$H_1:$ a shift up or down is more likely to be followed by a shift in the opposite direction than you'd see with a random series

Test statistic: proportion of shifts followed by shifts in the opposite direction.

Because our triples overlap, I believe we have some dependence between triples, so we can't treat this as binomial (we could if we split the data into non-overlapping triples; that would work fine).

Keeping that dependence in mind, we could still compute the distribution of the test statistic, but we don't need to in this case, because the observed proportion of direction reversed triples is just under the expected number of 2/3 for a random series, and we're only interested in more reversals than that.

So we don't need to compute any further -- there's no evidence at all of a tendency to reverse (up-down or down-up) more than you'd get with a random series.

[I really doubt the neglected mild cycle will have enough impact to move the expected proportion down anywhere near far enough for this to make a substantive difference.]

• thanks -very interesting. A quick question regarding the runs test. Are you referring to the typical runs-test to test for randomness? From the above data, I can produce the following run: 1 0 1 0 1 0 1 1 1 0 1 1 0 0 1 1 1 1 0 1 1 0 0 0 1 1 0 1 1 0 0 1 0 1 0 0 1 1 0 1 0 0 0 1 0 1 0 1 0 0 1 1 0 1 1 1 0 with a 1 indicating the series going up, and a 0 it going down. Using runs.test from the tseries R package, this gives a test-statistic of 1.81 and a p of 0.07. Whilst I'm not too worried about these example data, I wonder if this is the sort of analysis you were referring to? – jalapic Nov 24 '14 at 2:12
• No, I believe that's runs-of-one-kind (such as for clumping of signs, say), not runs-up-and-down. It would be suitable for a somewhat different kind of hypothesis than this. – Glen_b -Reinstate Monica Nov 24 '14 at 3:05
• @Glen_b, the six possible ways of up-down and stuff like that is covered in literature which can be traced back from the paper I posted in my answer. Particularly, from your picture you can easily see that there's two way to have up-down sequence out of six combinations. This means that P=1/3 to encounter a local peak anywhere in the sequence, i.e. the average distance between peaks is exactly 3. – Aksakal Nov 24 '14 at 3:11
• @Aksakal That they're connected is no surprise, since they're considering closely related things (I can only upvote your answer once, sorry). I just decided to try to answer the most literal interpretation of the question in the most elementary way I could, because I thought it might prove illuminating to show that some pretty basic analysis requiring little mathematics beyond counting could give some useful results. – Glen_b -Reinstate Monica Nov 24 '14 at 3:17
• jalapic -- see this page which discusses runs-up-and-down and a particular version of runs-of-one-kind. It gives a handy normal approximation for runs up-and-down and should also help clarify the distinction. – Glen_b -Reinstate Monica Nov 24 '14 at 3:19

You could use a package called structural change which checks for breaks or level shifts in the data. I have had some success in automatically detecting level shifts for non-seasonal time series.

I converted your "value" into a time series data. and used the following code to check for level shifts or change points or break points. The package also has nice features such as chow test to do chow test to test for structural breaks:

require(strucchange)
value.ts <- ts(data[,2],frequency = 1, start = (1956))
bp.value <- breakpoints(value.ts ~ 1)
summary(bp.value)


Following is the summary from the breakpont function:

Breakpoints at observation number:

m = 1           36
m = 2     16    31
m = 3     16    36    46
m = 4     16 24 36    46
m = 5   8 16 24 36    46
m = 6   8 16 24 33 41 49

Corresponding to breakdates:

m = 1                  1991
m = 2        1971      1986
m = 3        1971      1991      2001
m = 4        1971 1979 1991      2001
m = 5   1963 1971 1979 1991      2001
m = 6   1963 1971 1979 1988 1996 2004

Fit:

m   0           1           2           3           4           5           6
RSS   0.8599316   0.7865981   0.5843395   0.5742085   0.5578739   0.5559645   0.5772778
BIC -71.5402819 -68.5892480 -77.7080112 -70.6015129 -64.1544916 -56.2324540 -45.9296608


As you can see the function identified possible breaks in your data and selected two structural breaks at 1971 and 1986 as shown in the plot below based on BIC criterion. The function also provided other alternative break points as listed in the output above.

Hope this is helpful

• The problem with the approach that you are using is that it tacitly assumes no Pulses and (probably) no ARIMA structure. My breaks are "on or around basis" 1972 and 1992 .... yours are 1971 and 1986, ignoring the 1989 anomaly. In summary the analyses are not that different with mine being robust to one time anomalies. – IrishStat Nov 24 '14 at 12:48
• @IrishStat you are right. Autobox does indeed identify corrects for pulses and then correctly identifies for level shift. One issue that I had with stuctural change package in R is that it works reasonably well for non seasonal time series, it has very limited use or of no use in Seasonal time series, which I'm sure Autobox handles very well. – forecaster Nov 24 '14 at 15:44