4
$\begingroup$

I have a variable with sales data over time. It is very noisy at a disaggregate level but if you look at it as a whole, you can see a smoothing curve that follows a polynomial pattern. Is there a way to fit such curve and actually generate values of it over time in the dataset?

enter image description here

I tried many different options here including a moving average (which leaves a lot of noise still); I tried lowess smoother in Stata, which is on the opposite side - too neutral and leaves out important curve patterns. I also tried fitting a nonlinear function nl but to no avail.

The original goal of this task is to determine any infection points that the curve can have. In my mind, I am looking for a curve that would run in the middle of all the noise.

Improve this question
$\endgroup$
2
  • $\begingroup$ Do you need the smoothed function to be represented as a polynomial, or would you accept any function that smooths out the noise (not necessarily expressed in polynomial form). $\endgroup$
    – Underminer
    Jun 15, 2016 at 15:44
  • $\begingroup$ Have you tried filtering, through the Fourier transform/convolution? $\endgroup$
    – Firebug
    Jun 15, 2016 at 19:28

2 Answers 2

Reset to default
3
$\begingroup$

Since you mention the "polynomial pattern" in your question, try to fit your data using polynomial least squares fitting.

I tried to reproduce your data (more or less) and plotted a third degree least squares fit on the data. The result is in the graph below.

Actually, I used two goniometric functions to generate the data. The period and amplitude of the base cycle can be estimated by the fitted polynomial: half the period is the distance between the two extremes. The amplitude is half of the difference between the two extremes.

The advantage of this approach is that is it very easy to understand, calculate and apply (maybe at the cost of some mathematical inaccuracy).

There are two disadvantages of using a moving average over any fitting approach:

  • the moving average is always lagging. This results in shift in the direction of the past data (in the example below: the moving average is above the polynomial fit line; the trend is going down, so the moving average is lagging upwards in this case).
  • the moving average moves along with any spike in the data. This results in a more "wobbely" line compared to the fitting approach.

These effects can be clearly seen in the picture below (red = polynomial fitting; black = 20 period moving average).

Data and polynomial least squares fit

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ Because this is a time series, with an obvious seasonal structure and some serial correlation, caution would seem advisable. In particular a "classic" polynomial least squares fit would have several disadvantages compared to methods that are more suitable to time series or are more robust. $\endgroup$
    – whuber ♦
    Jun 15, 2016 at 21:24
  • $\begingroup$ The OP mentions polynomial-like patterns. Doing a moving polynomial least squares fit is a logical thing to try, I'd say. $\endgroup$
    – agtoever
    Jun 15, 2016 at 21:33
  • 1
    $\begingroup$ Possibly--but fitting it via ordinary least squares may be problematic. It certainly would produce incorrect standard errors of the parameters. $\endgroup$
    – whuber ♦
    Jun 15, 2016 at 22:02
  • 1
    $\begingroup$ This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? You can also turn it into a comment. $\endgroup$
    – gung - Reinstate Monica
    Jun 15, 2016 at 22:02
  • 1
    $\begingroup$ Moving averages are not always lagging. If you use points equally spaced on either side of the point (sometimes called 'central moving average') then the moving average is more representative of the current trend. $\endgroup$
    – Underminer
    Jun 20, 2016 at 12:52
7
$\begingroup$

Sounds like you just need to adjust the smoothing parameters (sometimes called bandwidth) to your liking. Either of these methods should be able to be tuned appropriately.

Moving average can be adjusted to account for more points (increasing smoothness).

Similarly, lowess smoothers have a smoothing parameter to increase or decrease smoothness.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.