5
$\begingroup$

I have some time series data, where each data point in the time series has a timestamp and a value. For example:

Time 1    value 5 
Time 2    Value 7
Time 3    Value 10

Now at a given time X, I want to obtain an approximation on the average of recent values in the time series. For example, consider that the time series is temperature readings for a machine. I want to calculate the average temperature readings in recent days. To do this, I must choose the number of data points to be used to calculate the average.

On the one hand, if I choose to use a smaller number of values such as the two last values of the time series to calculate the average, the average will not be very reliable because these data points may be outliers.

On the other hand, if I choose to use all previous values of the time series, this may not reflect recent trends in the time series, but the average would better tolerate outliers.

Thus, my question is how can a determine a minimum number of data points to calculate an average of recent values in a time series that is considered "reliable". The concept of "reliable" should be expressed in terms of a confidence interval, if possible.

Of course, the concept of "recent values" is fuzzy, but there are certainly some lower bound on the number of data points that should be use to calculate an average reliably.

Just to make my question clearer, I will tell you the solution that I have thought about, which is in my opinion not good. I have though about using the Hoeffding bound ( https://en.wikipedia.org/wiki/Hoeffding%27s_inequality), as it "provides an upper bound on the probability that the sum of independent random variables deviates from its expected value". But I think that it is inappropriate for my problem with a time series as events may not be independent or the time series may not be stationnary. Thus, what could be used as a better technique?

Edit: I have edited the question since some people have decided put the question on hold.

Improve this question
$\endgroup$
11
  • 1
    $\begingroup$ What is your definition of a "good" approximation? $\endgroup$
    – Dave2e
    Commented Jul 18, 2017 at 2:27
  • $\begingroup$ If we don't use enough data points, then the average is meaningless. For example, if the time series is temperature readings, using only 2 data points would probably be meaningless as these points could be outliers which would influence too much the average. Perhaps that we should use at least 10 data points to get something more stable... But I do not want to use a fixed value like 10. I would like to calculate something like a confidence interval on how good the average is approximated using a given amount of data points $\endgroup$
    – Phil
    Commented Jul 18, 2017 at 2:31
  • $\begingroup$ I also do not make any assumption about the distribution of values in the time series. $\endgroup$
    – Phil
    Commented Jul 18, 2017 at 2:34
  • $\begingroup$ So in other words, a good approximation for me would be an approximation that is not influenced too much by outlier values, and that also use enough data points to consider that the average as representative of the recent values in the time series. $\endgroup$
    – Phil
    Commented Jul 18, 2017 at 2:37
  • $\begingroup$ I have thought about using the Hoeffding bound for this. But I am not sure if it is a good idea $\endgroup$
    – Phil
    Commented Jul 18, 2017 at 2:43

3 Answers 3

Reset to default
4
$\begingroup$

I agree with Carl that fitting a model to the data would be an ideal solution, if it makes sense in your context. But I also want to suggest that the following way of thinking about your problem might be helpful.

Suppose that I have a time-varying process that looks like a random walk (so that the location is autocorrelated through time). Suppose I measure this position with some measurement error (which is an IID random variable) at equally spaced time points. I understand your question to mean:

How many time points should I average in order to get the best possible information about the process' current location?

If the random walk careens around wildly and the measurement error is small, then the answer might well be use only the most recent point. Using previous points in the average would introduce lots of extra variation due to the random walk, and a single point is already a reasonably good approximation of the process' true location. If the measurement error is large and the random walk moves slowly, then the answer will be use a lot of points. The random walk is relatively stationary, and you need to average over the noise in the measurement. Interestingly, if your measurement noise is extremely fat-tailed (i.e. Cauchy distributed), then the answer will always be use just the most recent point (because the average of multiple points does not provide a better approximation to the central tendency of that distribution than any single point does!).

It should be possible to work out the ideal number of points to use in special cases where the distribution followed by the random walk and the distribution of the measurement error are both known. However, this is precisely the case where a model, as suggested by Carl, would be useful.

Edit Carl's comment also made me realize that it's very likely that a weighted average (that weights more recent points more heavily) could outperform an average that introduces some hard-threshold cutoff for inclusion.

Improve this answer
$\endgroup$
2
  • $\begingroup$ OK, +1, as a second step perhaps when tendencies in the data are excluded. To exclude them with certainty is a matter of showing that the data is truly random, which implies knowing what the noise in the data is, e.g., Gaussian, Poisson, whatever. In that case, if the noise explains all of the data, then there is no latent information. $\endgroup$
    – Carl
    Commented Jul 25, 2017 at 0:30
  • $\begingroup$ +1 to your answer as well. The model fitting approach works well even in the pure noise case, I think (the model in this case fits a random walk with measurement error to the data, and thus decomposes the time-series into process and sampling noise), so even though there's no latent information to leverage, it still gives a good answer. My reason for posting this was just to try to answer the original question more narrowly--if I'm going to attack this issue by averaging recent points (and not even using a weighted average!), how many should I use? $\endgroup$
    – Jacob Socolar
    Commented Jul 25, 2017 at 0:35
2
+50
$\begingroup$

Q1: How many points?

All of them. The best way, is to fit a model to the data. If the model is a good one, i.e., if an accurate $T(t)$ can be found, where $T$ is the temperature as a function of $t$ time, then the problem is solved, just choose a $t$, and $T$ is predicted.

In a temperature time series this may require one or more delay terms in the auto regressive integrated moving average, ARIMA, sense. However, in general, use of a model is superior to taking averages.

Q2: What about outliers?

True outliers are rare. If needed outlier testing can be performed. More frequently, when the data is not normally distributed on the $T$-axis, a transformation of variables may be needed to make conditions more normal, among other examples using $\ln(T)$, or using $\dfrac{1}{T}$ or using $\sqrt{T}$ instead of plain old $T$ for the regression equation target.

Improve this answer
$\endgroup$
2
  • $\begingroup$ ARIMA is useful as far as an error correlation model. It may be worth mentioning useful model classes for the function $T(t)$ itself -- for example, penalized splines. $\endgroup$
    – eric_kernfeld
    Commented Jul 26, 2017 at 15:12
  • $\begingroup$ @eric_kernfeld Sure, why not? Another would be physically motivated models, which are highly dependent upon the physical circumstances, and, no specifics were given in the question. My point merely was that in general, modelling is needed to figure out what will work, and what will not. $\endgroup$
    – Carl
    Commented Jul 26, 2017 at 21:26
1
$\begingroup$

This is more a suggestion than a complete answer, but maybe you could use the median instead of the average so that outliers affect less the result.

Improve this answer
$\endgroup$
2
  • $\begingroup$ For various reasons, not all of them valid, the average is used as a default measure of data location. General usage of the median in place of the mean for unknown distributions is superior in that it avoids some problems but may not be optimal either. $\endgroup$
    – Carl
    Commented Jul 25, 2017 at 18:35
  • $\begingroup$ That is, one needs to model the data to obtain knowledge of the distribution of data. When that is done, the appropriate measure of location becomes more obvious. For example, for a lognormal distribution, the antilog of the mean logarithm of the values (i.e., geometric mean) would be a more appropriate measurement than the mean of the data. $\endgroup$
    – Carl
    Commented Jul 27, 2017 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.