3
$\begingroup$

How do you predict data that contains multiple levels of nearly constant data?

Simple linear models even with weights (exponential) did not cut it.

I experimented with some clustering and then robust linear regression but my problem is that the relationship between these levels of constant data is lost.

Here is the data:

structure(list(date = structure(c(32L, 10L, 11L, 14L, 5L, 6L, 
1L, 2L, 12L, 9L, 19L, 13L, 4L, 17L, 15L, 3L, 18L, 7L, 8L, 21L, 
16L, 22L, 28L, 29L, 30L, 26L, 27L, 31L, 20L, 23L, 24L, 25L), .Label = c("18.02.13", 
"18.03.13", "18.11.13", "19.08.13", "19.11.12", "20.01.13", "20.01.14", 
"20.02.14", "20.05.13", "20.08.12", "20.09.12", "21.04.13", "21.07.13", 
"21.10.12", "21.10.13", "22.04.14", "22.09.13", "22.12.13", "23.06.13", 
"25.01.15", "25.03.14", "25.05.14", "26.02.15", "26.03.15", "26.04.15", 
"26.10.14", "26.11.14", "27.07.14", "27.08.14", "28.09.14", "28.12.14", 
"29.03.10"), class = "factor"), amount = c(-4, -12.4, -9.9, -9.9, 
-9.94, -14.29, -9.97, -9.9, -9.9, -9.9, -9.9, -9.9, -9.9, -9.9, 
-9.9, -9.9, -9.9, -4, -4, -11.9, -11.9, -11.9, -11.9, -11.98, 
-11.98, -11.9, -13.8, -11.64, -11.96, -11.9, -11.9, -11.9)), .Names = c("date", 
"amount"), class = "data.frame", row.names = c(NA, -32L))

regression for multiple levels

revisiting rollmedian

@Gaurav - you asked: Have you tried building a model with moving averages? as ARIMA didn't work - I did not try it. But I have now.

zoo::rollmedian(rollTS, 5)

Seems to get the pattern of the data. However I wonder now how to reasonably forecast it. Is this possible?

rollmedian

Improve this question
$\endgroup$
14
  • $\begingroup$ Pictures of data are pointless. Give us a reproducible example! $\endgroup$
    – thelatemail
    Commented Sep 28, 2015 at 6:24
  • $\begingroup$ If it is nearly constant, do you mean there are minor deviations? $\endgroup$
    – Gaurav
    Commented Sep 28, 2015 at 6:25
  • $\begingroup$ added the data. $\endgroup$
    – Georg Heiler
    Commented Sep 28, 2015 at 6:26
  • $\begingroup$ does -9,9 indicate -9.9? Or is it a tuple of -(9,9)? $\endgroup$
    – Gaurav
    Commented Sep 28, 2015 at 6:28
  • $\begingroup$ @Gaurav - it's -9.9. Look at the graph. $\endgroup$
    – thelatemail
    Commented Sep 28, 2015 at 6:30

3 Answers 3

Reset to default
3
$\begingroup$

Your data is a classic example of data where there is more noise than signal and therefore unpredictable, no matter what ever data mining /time series approach you use, it is going to give you poor predictions unless you know a priori by domain knowledge what $caused$ the level shifts and outliers. Also techniques like arima and exponential smoothing needs equally space time series which you do not have in your example. That said two reasonable approaches:

  1. Model it deterministically, again this needs knowledge of outliers
  2. Use last value for all future prediction ( this is simple exponential smoothing)
Improve this answer
$\endgroup$
4
  • $\begingroup$ There is no doubt that level shifts and outliers need to be explained BUT they first must be identified otherwise you would have nothing to explain. When dealing with a few data sets the "eye" can often identify the level shifts and the ouliers but with massive amounts of time series this needs to me automated. $\endgroup$
    – IrishStat
    Commented Sep 29, 2015 at 1:47
  • $\begingroup$ @irishstat my issue with this particular data is that automatically identifying outliers is not going to help in forecasting, the data is more noiser and we need to take a step back and See what we could and could not forecast. $\endgroup$
    – forecaster
    Commented Sep 29, 2015 at 2:13
  • $\begingroup$ @forecaster: maybe you are right. I will try to incorporate a priori domain knowledge in order to produce useful forecasts. $\endgroup$
    – Georg Heiler
    Commented Sep 29, 2015 at 8:45
  • $\begingroup$ @forecaster If one could "explain" the root cause of the level shift then one could presumably get a better forecast, The point is that there was a "level shift" and one needs to find out why. If one is not aware of the level shift then one will never look for the root cause. $\endgroup$
    – IrishStat
    Commented Sep 29, 2015 at 11:45
2
$\begingroup$

Call $Y$ the output and $U$ the piecewise constant function you would like to obtain. Your idea is to minimize something like:

$$ \min_U ||Y-U||^2_2 + \lambda P(U) $$ Where $P$ is a function that penalizes the derivative of $U$ (to minimize the number of levels). If you choose to enforce sparsity with a $L_1$-norm, you obtain : $$\min_U ||Y-U||^2_2 + \lambda \sum_i |U_{i+1}-U_i|$$ Which is the Group Fused LASSO. It is studied extensively in: The group fused Lasso for multiple change-point detection, by Kevin Bleakley and Jean-Philippe Vert.

More information is available here http://arxiv.org/pdf/1106.4199v1.pdf

Improve this answer
$\endgroup$
3
  • $\begingroup$ Is there an R package which already implements the proposed approach of Group Fused LASSO? So far I only could find packages like cghFLasso which only have fused LASSOs $\endgroup$
    – Georg Heiler
    Commented Sep 28, 2015 at 17:26
  • $\begingroup$ There is a matlab implementation here: cbio.ensmp.fr/~jvert/svn/GFLseg/html As for R, I haven't heard of any... $\endgroup$
    – RUser4512
    Commented Sep 28, 2015 at 17:29
  • $\begingroup$ Note that you can also implement your own gradient descent or use optimization packages ! $\endgroup$
    – RUser4512
    Commented Sep 28, 2015 at 19:43
2
$\begingroup$

I utilized AUTOBOX , a program (partially developed by me) designed for analyzing data like this. Using Intervention Detection procedures it automaticallyfound a model with a level shift and a few pulses. This is a series that should not be analyzed with ARIMA procedures because it is primarily deterministic.enter image description here . The Actual/Fit?forecast graph is here enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ This looks interesting. However, does it take into account the associated dates or does it treat the data as a regular timeseries? $\endgroup$
    – Roland
    Commented Sep 29, 2015 at 7:04
  • 1
    $\begingroup$ In this case it treats the data as a "regular time series" and finds that although there is period-to-period (ARIMA) structure it is not as important as a model that includes "dummy variables" . If one had dates then this might lead to incorporating/testing for daily effects, weekly effects.monthly effects or holiday effects. $\endgroup$
    – IrishStat
    Commented Sep 29, 2015 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.