0
$\begingroup$

I am new in the topic of time series data and found out about the STL decomposition which can smoothly show the trend in some time series data. My task is to detect trends given certain data.

At first, I went the complicated way and labelled data, where trends were and trained classification models on it and tested it with new data. Unfortunately this did not perform well. Now after reading more about time series data, I thought it might be a better solution to treat it as a time series problem rather than a classification problem.

Take this STL decomposition as an example:

enter image description here

Is there a way to 'detect' the location of the trend? I mean to put points in the plot of the trend around x=30 until x=52 (upgoing trend) or around x=45 until x=80 (down-going trend)? (Not manually & hardcoded obviously, but with an algorithm that detect a trend)

My data are measurement points of laptop components that are retrieved after each other (with a timestamp) but not necessary periodically with the same time interval. I hope this isn't a problem for this.

My data has this look:

0       0.2362
1       0.3939
2       0.3062
3      -0.2643
...
232    -0.3146
233    -0.1514
234    -0.1881
235    -0.0965
$\endgroup$
4
  • $\begingroup$ There are to different "trends" that are possible . See stats.stackexchange.com/questions/103193/… for a discussion that might be of help to you. You can conduct a search ( trial and error) to determine which kind of trend is more appropriate for your data. The data has to have a fixed time interval. $\endgroup$
    – IrishStat
    Sep 6, 2022 at 12:55
  • 1
    $\begingroup$ The STL decomposition assumes regularly sampled data, and you apparently have an irregular time series. The other problem is that if your time series goes up and down in a zigzag fashion, one could say that you have lots of "trends", a new one between every pair of successive data points. This is probably not useful. There are "optimal change point detection" algorithms, implemented in R. Can you edit your post to include some example data (timestamps and measurements), ideally the output of dput(your_data)? $\endgroup$ Sep 6, 2022 at 13:21
  • $\begingroup$ @StephanKolassa I added the 'look' of my data now. The timestamps are not included but you can imagine it as a column with irregular timestamps. The index is already ordered in chronological order. And yes I also meant that you would see lots of 'trends' of different sizes and slopes, but that is my task.. to detect them:/ $\endgroup$
    – josh
    Sep 6, 2022 at 13:44
  • $\begingroup$ Can you please provide more data, including timestamps? Per above, please run dput(your_data) and paste the output in your question. $\endgroup$ Sep 6, 2022 at 13:52

2 Answers 2

2
$\begingroup$

Your problem is more along the line of time series segmentation and changepoint detection. If you care for a readily available solution in R (or Matlab), there are numerous alternatives possible (e.g., strucchange, and changepoint), as summarized in the R CRAN task view on time series: https://cran.r-project.org/web/views/TimeSeries.html. If you want to implement your own, I find the easiest reference as a starting point is Keogh, Eamonn, Selina Chu, David Hart, and Michael Pazzani. "Segmenting time series: A survey and novel approach." (https://www.ics.uci.edu/~pazzani/Publications/survey.pdf). Apparently, assumptions of all sorts are made in different methods. As a practitioner, I think no single method is the best all the time: try and let the results tell which suits your purpose.

Allow me to use one package Rbeast written by me and available in R and Matlab ( more info at https://github.com/zhaokg/Rbeast) to illustrate the basic idea. It accepts regular or irregular data with or without a periodic component. It seeks to decompose a time series into trend and seasonality and simultaneously detect changepoints in the trend and seasonal components.

library(Rbeast)

# Nile:           an annual riverflow time series of the River Nile
# season='none':  no periodic/seasonal component so no time series decomposition is performed and only changepoint detection is done

o = beast(Nile, season='none')  
       
plot(o)

The time series is divided into two segments, with one changepoint. The time-varying probability of changepoint occurrence in the trend component is shown in the second subplot Pr(tcp). The sgnSlp subplot gives the probability of the slope being positive, zero, or negative over time.

enter image description here

Here is another example using the covid19 daily time series of new cases, with a periodic component of 7 days.

library(Rbeast)
data(covid19)

newcases = covid19$newcases
datenum  = as.numeric(covid19$date) 

o       = beast(newcases, start=min(datenum), deltat=1, freq=7) 

o$time  = as.Date(o$time, origin='1970-01-01') # Convert from integers to Date.
plot(o)

Here is the output. In this case, the time series was decomposed into the seasona/periodic component and the trend component. The locations of changepoints in the two components are indicated by the dash lines in the season and trend subplots. enter image description here

$\endgroup$
0
$\begingroup$

Easiest thing for 'plug and play' is to use Meta's Prophet since it fits for these trend changepoints. An example output of the trend from the example in the link: enter image description here Then just look at the change in the gradients for the trend component in the output dataframe. There are definitely better methods that perform better on benchmarks but it will get you started. If it irregularly sampled just pretend that it is and disable seasonality and reindex your time series to appear regular. Basically it will just be a trend changepoint algo at that point.

Alternatively you could try wild binary segmentation which involves exhaustively segmenting your data into 2 pieces and fitting a trend line to each then choosing the point where segmenting your data minimizes some criterion like mse, then repeating that process fitting on the residuals until a global criterion is minimized like AIC/AICC. If using python this is essentially what a package I wrote up uses if you use a 'linear' trend estimator with 'local' fit type: ThymeBoost.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.