5
$\begingroup$

I am trying to fit a baseline through my data, and I am not getting close enough with polynomial regression. I used gradient descent to set the parameters.

Are there any other ways or algorithms that could get me any closer? I want to make my baseline in blue look more similar to the actual data in red.

enter image description here

Improve this question
$\endgroup$

3 Answers 3

Reset to default
7
$\begingroup$

Looks like a time series with very strong (and fairly regular) seasonality. If you use R, you might want to look at function stl(), or fit a basic structural model (an easy entry point function is StructTS(), otherwise there are several packages which afford you more generality and better control of the model you want to fit).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'd add that you might want to log transform the outcome before modeling it, will likely make things easier. $\endgroup$
    – charles
    Commented May 14, 2014 at 14:32
  • $\begingroup$ so you would recommend using R ? I was trying to code all the thing by-myself and use octave, what i actually plan to do is write an algorithm that would do the job done in an already built in function like the one in R $\endgroup$
    – AnarKi
    Commented May 14, 2014 at 14:40
4
$\begingroup$

The answer is ARIMA models with Intervention Detection enabled. Intervention Detection will suggest level shifts/local time trends/seasonal pulses and pulses which are needed to aid the efficient identification/ robust identification of the ARIMA structure reflecting auto-regressive memory. Please post your data in column format and advise as to the frequency of measurement. It looks to me like you might be attempting to use a very dated procedure called Fourier (a pure deterministic structure i.e. no auto-regressive component ) which fits the data based upon an assumed structure but often (nearly always) doesn't deliver a good "explanation" of the data ... consequently a picture like the one that you presented. Kudos on asking the question !

IN RESPONSE TO COMMENTS BY w.huber AND OTHERS .....:

A point in clarification. Intervention Detection (ID) and Power Transforms(PT) like logs are forms of transformations. ID deals with adjusting vales for unspecified deterministic structure while PT deals with uncoupling error variance relationships with the expected value. The whole idea is to as little transformation as necessary much like a doctor prescribing the least form of treatment . As @w.huber correctly points out , you need to prove a dependence between the error variance and the expected value before you apply a PT. When (and why) should you take the log of a distribution (of numbers)? might help you.

Improve this answer
$\endgroup$
2
$\begingroup$

Think you have your answer.

But would add that it may be useful to log your data. Then consider doing regular (d) and/or seasonal (D) differencing. The resultant series should be much easier to model. I'm not confident d/D is necessary, but some form of transformation likely is necessary. Hard to tell from graph, but it appears that the volatility increases with time/linear trend.

The models suggested (by the other answers) will give you better forecasts and decomposition of the series, but with some transformation of the time series you can often fit a good-enough polynomial.

UPDATE:
(1) As mentioned below, it is unclear if the volatility is directly proportional to the level. If so log transformation is helpful. Otherwise perhaps not.
(2) Square root transformations are underused, but also often helpful in these settings.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ That's a good approach in principle. Evidence in favor of logarithms would be some indication that the amount of variation (perhaps as exhibited by the local cyclic amplitude) is directly proportional to the underlying value. I see little such evidence in this plot, although possibly the data might benefit from a gentler form of re-expression such as a square root (which does not share the same severe problems near $0$, either). $\endgroup$
    – whuber ♦
    Commented May 14, 2014 at 20:40
  • $\begingroup$ (+1) agreed. the blue overlay and outlier made it hard for me to judge visually. square root transformation is underused (or rather I don't think to use it often enough). $\endgroup$
    – charles
    Commented May 14, 2014 at 21:07
  • $\begingroup$ @whuber More correctly the directly proportional ratio is between the error variance and the expected value. This is the basis for the Box-Cox identification procedure where lambda can range from -1 to +1 $\endgroup$
    – IrishStat
    Commented May 15, 2014 at 17:41
  • $\begingroup$ @Irish Thank you for your comment. I do not understand your use of "more correctly" because you seem to be repeating a special case of what I said. (Procedures for identifying a Box-Cox transformation are more robust and general when they are based on estimates of dispersion and location that are more resistant to outliers then error variances and expectations, which is why I used those more general terms.) Also please note that although $\lambda$ is frequently found between $-1$ and $1$, it is not limited to that range (nor to any range in principle). $\endgroup$
    – whuber ♦
    Commented May 15, 2014 at 17:49
  • 1
    $\begingroup$ @Whuber we are good ... I just wanted to clarify that the term "amount of variation" means the error variance NOT the variance of the observed series and the term(too vague for me) "underlying value" is more precisely the "expected value" $\endgroup$
    – IrishStat
    Commented May 15, 2014 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.