4
$\begingroup$

We have the following data points in variable data pertaining to a problem that we are solving:

                9996792524
                8479115468
               11394750532
                9594869828
               10850291677
               10475635302
               10116010939
               11206949341
               11975140317
               11526960332
                9986194500
               11501088256
               11833183163
               13246940910
               13255698568
               13775653990
               13567323648
               14607415705
               13835444224
               14118970743

This corresponding date numbers are stored in a variable timevalues:

          735678.574305556
          735710.586805556
          735863.672916667
          735888.539583333
          735921.589583333
          735941.590972222
          735986.583333333
          736021.481944444
          736043.498611111
                  736063.5
          736083.504166667
              736223.35625
                 736250.45
          736278.452083333
          736314.327777778
          736356.239583333
          736383.209722222
              736411.10625
          736431.925694444

We fit a 9th degree polynomial to this data and then plot it as follows:

data9 = fit( timevalues, data, 'poly9', 'Normalize', 'on' );
plot(data9,timevalues,data);

enter image description here

Now we need to extrapolate this trend / polynomial into the future or for further values of timevalues on the X-axis. How do we do that?

UPDATE: Description of our problem We have bits per second observed on our border firewall device -- which is what these values are. There are a LOT of such values over 1 minute intervals in the last 4 years (more than a million). Not all values are useful because we just want to see how the trends in peaks is rising in time since we want to increase our load capacity before we hit 'max' some day. In other words, we are not interested in valleys or average values but 'peaks'. So we used the findpeaks() function in Matlab to find the peaks in our data (which is what the values above are). Now we are trying to fit a trend line on these peaks and extrapolate it to see how we need to increase capacity on border device.

$\endgroup$
  • 6
    $\begingroup$ You don't. This is about the worst possible way of extrapolating a time series that could be conceived. Consider changing your question into one where you describe your data and objectives and ask for reasonable ways of achieving those objectives, rather than proposing a method that is guaranteed to fail: the answers will be much more helpful. $\endgroup$ – whuber Jul 21 '16 at 20:26
  • $\begingroup$ @whuber Updated my question to include a general description of the problem in the end. Please advise on the best possible approach. Actual Matlab or R code might help better than theories. Thank you. $\endgroup$ – learnerX Jul 21 '16 at 20:38
1
$\begingroup$

Please let me caution you clearly: do not extrapolate a polynomial fit! There are few things in statistics more likely to lead to disaster. Any polynomial fit is a localised approximation to a function based on the Taylor polynomial. By necessity, the high-order terms will explode (either positively or negatively) as you go further outside the range of the fitted data. This means that extrapolation of a fitted polynomial curve lead you either to predict explosive positive or negative values.

$\endgroup$
0
$\begingroup$

Is this a machine learning dataset? What is it about? Be very careful about over-fitting. There can be a huge difference between fitting well a model and obtaining good forecasts. Again see more about it here.

Now, answering your question, first, you need to find the values for x you are interested on. The theory behind your model can guide you on this. You could select equally spaced values, or follow a pattern observed in your x data. Again, this depends on your interest.

Second, you need to predict values of y from these x. There are a few techniques. On the one hand, if you just want point forecasts, just plug the x-axis values you want to predict into the model, and you will get the point forecast. Alternatively, you can do bootstrapping to get a distribution of forecasts. In Matlab these techniques are already integrated with Machine Learning and related toolbox. Notice that they do not come with the standard Matlab. You might consider using R instead. It's free and there are plenty of toolbox to do bootstrapping available.

Finally, I want to stress again the problem of over-fitting. The way around this is applying cross-validation techniques. The idea is that you partition your data in such a way that the data from where you estimate the model is different from the data where you test the model. This method allows you to run out-of-sample prediction, which is regarded as the crux test for the capacity of the model to predict. In-sample prediction is a weak test for the model. In the end, cross-validation allows you to find the optimal point between fit and prediction accuracy. Just as before, cross-validation techniques are also integrated with Machine Learning and related toolbox in Matlab. Here again an alternative is to use R.

$\endgroup$
  • $\begingroup$ Ups, I read R and not Matlab. Sorry. Will update answer. $\endgroup$ – luchonacho Jul 21 '16 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.