2
$\begingroup$

I have a data set of gas prices. They are not evenly spaced in time, and have quite a few days missing.

Here is a sample of the data

 Date              Price
2016-05-18  0.9931994220059637
2016-05-19  0.9484317771304708
2016-05-20  0.93686097538978
2016-05-21  
2016-05-22  
2016-05-23  
2016-05-24  
2016-05-25  
2016-05-26  0.9416552772446175
2016-05-27  0.9312735217012867

Is there a widely used way to interpolate the data? I don't want to do polynomial or linear interpolation since gas prices are subject to perturbation.

$\endgroup$
4
$\begingroup$

Gaussian Process Modeling (GPM) is a non-parametric approach that is excellent at interpolating non-uniformly sampled data which have an expected temporal correlation structure.

There are numerous resources on GPM located here:

http://www.gaussianprocess.org

Including this excellent treatment on GPMs:

http://www.gaussianprocess.org/gpml/

There are GPM capabilities available in Python in SciKitLearn:

http://scikit-learn.org/stable/modules/gaussian_process.html

Using GPM requires you to have some "prior" that describes the correlation (which GPM then fine-tunes from the observed data by solving for kernel hyper-parameters). Arguably the simplest covariance model is the "squared exponential" kernel function which simply says that "things that are close together tend to have similar values", which basically causes GPM to act like a smoothing function. Changing the kernel function based on beliefs concerning temporal correlation can result in different behaviors (for instance, you can add a kernel component that reflects periodic repetition in the data).

An arguably simpler approach would be splines (typically cubic splines), which do smoothing interpolation in a specific way and are also happy to deal with nonuniformly-sampled data. This may suit your needs just fine if you just need some basic smooth interpolation and don't have a strong opinion about the form it should take.

The advantage of GPMs over splines is that they provide a lot more flexibility and control over the behavior of the interpolation based on prior beliefs about its specific temporal covariance structure and are generally more adaptable to the data in terms of inferring underlying structure. Splines are simpler to employ, but always behave in more or less the same fashion and have very limited ability to adapt to underlying structure in the data (apart from the constraint that they always pass through the observed anchor points and interpolate smoothly in a particular fashion).

Additionally, you may not want your interpolation function to always anchor to the observed data exactly (if you believe that the observations are noisy, for instance). In this case, GPMs also have the additional advantage that they can account for noise in the observation data.

(The Kalman filter can do similar things to GPM using "forward and backward smoothing", and has a lot in common with the GPM approach, but GPM is arguably more elegant in its formulation and application in this case, particularly if you don't have a requirement for strictly-causal or incrementally-updating behavior).

There are a number of interpolation capabilities available in Python (including univariate splines, which would be suitable to try with your data).

https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html

$\endgroup$
  • 2
    $\begingroup$ The Kalman and GPM approach would provide some idea of the interpolation's variability, while the spline idea is very easy. The Kalman (StructTS) and spline options are available in R as well. $\endgroup$ – Wayne Oct 24 '16 at 18:21
  • 1
    $\begingroup$ Never heard of GPM, but the ability to constrict meaningful confidence intervals is a great part of that procedure. Thanks, great answer. $\endgroup$ – Demetri Pananos Oct 25 '16 at 14:05
  • $\begingroup$ Definitely true @DemetriPananos, a major advantage of GPMs is that they produce a non-stationary statistical model for the data inclusive of uncertainty which is itself non stationary and data-informed. For certain classes of problems, they're a really powerful tool. $\endgroup$ – T3am5hark Sep 17 at 18:20
0
$\begingroup$

One novel way to interpolate missing values in an EVENLY SPACED TIME SERIES is to simultaneously identify the ARIMA model (just a weighted average without assumptions) and anomalies (unusual values like 1,9,1,9,1,9,5,9 or 1,9,1,9,1,9,1,13 . This is done via Intervention Detection methodology . The coefficients for the newly formed Intervention Series are then the adustments needed to replace the errant values with "what should have been" . In the two cases the adjustment for the errant value is -4 and -4 respectively. Your data set would be insufficient to form the ARIMA model as 5/10 ths (5 out of 10) are errant occluding/masking/hiding the underlying predictive signal/model/equation/weighted average necessary to provide the baseline.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.