7
$\begingroup$

Example of problem: Part of our research team is working on providing operationally wind power forecast. Usually, since there are different time scalse that interest forecast user, a forecast is issued every 15 min (it has even happened that 5 seconds was requirer) for every 15 minutes ahead up to serveral days. Obviously when you contract with a wind farm owner he has many wind farms, can be 10 or even 1000 wind farms depending of who you are working with. It is likelly that this number is going to be bigger in the near future. Now if the forecast user wants not only the point forecast but the forecast of 100 intesreting quantile (i.e. the whole marginal distribution) and if he wants you to store everything in a data base year after year, then you start tinking ..... ok that makes a lot of data. I'm not an international meteorological institute and cannot afford that.

Edit (according to @whuber's comment): to make it critical it can be tough of with a 5 second time frame. But anyway this is not meant to be the question, just an example (people coule provide other examples)

My question is : is there a compression theory/best practice for time series of distributions (i.e function of time with values in a space of distribution). I agree that you could work quantile by quantile and apply simple compression algorithm dedicated to signals (could be based on wavelet) but I am searhing for something more dedicated. For example if you know everything is well approximated by gaussians, the quantile by quantile approach would be stupid !

Edit (according to @whuber's comment): The question is really how to integrate the temporal dimension in a compression scheme, hence nI don't want to make a choice of a subset of quantiles that would be the same for ALL time. The remark with gaussian is not dedicated to the example, it is an example to illustrate a case when very good compression is required. An other trivial remark (from the theoretical side) would be: imagine you have a very complicated distribution that cannot be summarized with few quantiles and that you observe it for 1000 successive times. It would be sad to store it for each of these times.

Improve this question
$\endgroup$
4
  • 4
    $\begingroup$ Some thoughts: (i) Gaussians won't do it. You're interested in the right tail of the distributions (winds below a few km/h won't generate any power). (ii) You don't need 100 quantiles; you can interpolate. Probably 5-16 quantiles will do well. (iii) You only need a few bits per quantile (8 will do) because economically useful wind speeds lie in a small range. (iv) 1000 distributions / 15 min * 15 numbers/distribution * 5 E5 min/yr = 5 E8 values = 0.5 GB per year. That's worth about $0.50 today: you can afford it! $\endgroup$
    – whuber ♦
    Commented Feb 11, 2011 at 17:55
  • 1
    $\begingroup$ That's not a comment @whuber, that's an answer! $\endgroup$
    – onestop
    Commented Feb 11, 2011 at 21:35
  • $\begingroup$ @whuber, thanks good remark "Probably 5-16 quantiles will do well" I think that by this remark you enter the problem of compression (even if you deny it afterward). The remark about gaussian measure in my question is not to illustrate what I would do with wind power but rather to illustrate simple fact about compression of time series of distribution (rather more from a theoretical point of view). $\endgroup$
    – robin girard
    Commented Feb 11, 2011 at 21:39
  • 1
    $\begingroup$ Sorry, I didn't mean to appear to deny this is a compression problem. Any method of storing an empirical distribution is implicitly a compression method. The issue concerns how best to go about it while not wasting computing resources. I have established that storage is not the bottleneck, implying you should focus on methods that make subsequent analysis sufficiently efficient and accurate. $\endgroup$
    – whuber ♦
    Commented Feb 11, 2011 at 21:44

2 Answers 2

Reset to default
2
$\begingroup$

You could use any probabilistic time series model in combination with arithmetic coding.

You'd have to quantize the data, though. Idea: the more likely an "event" is to occur, the more bits for that event are reserved. E.g if $p(x_t = 1| x_{1:t-1}) = 0.5$ with $x_{1:t-1}$ being the history of events seen so far, then coding that event will cost you 1 bit, while all others have to use more bits.

Improve this answer
$\endgroup$
2
  • $\begingroup$ +1 I like the way you turn the problem with "the more likely an event is to occur the more bits for that event. But you should notice that often the tails of the distribution are somehow "precious" (in wind power forecast, there are relatively rare errors or events that cost a lot in terms of money) $\endgroup$
    – robin girard
    Commented Mar 17, 2011 at 13:37
  • $\begingroup$ But then you can use a heavy tailed distribution for that, right? $\endgroup$
    – bayerj
    Commented Mar 17, 2011 at 16:53
1
$\begingroup$

Your distribution is parametric, and you should just store the parameters that are sufficient statistics, if you can identify them. That includes the distribution family. For a time series, you can take advantage of autocorrelation and store the parameters of the predictive distribution conditional on its previous values.

The entropy of the prior (predictive) distribution of the parameters determines the upper bound for compression strength, but you may not need to compress them further. If you do, use arithmetic compression. Decreasing entropy, say by discretizing quantiles, will give greater compression.

Improve this answer
$\endgroup$
6
  • $\begingroup$ @sesqu "your distribution is parametric" can you explain why you say that? $\endgroup$
    – robin girard
    Commented Feb 12, 2011 at 12:14
  • $\begingroup$ @robin You said you have a forecast, which implies a model for a random variable. That model comprises a family of distributions for the variable and a function to identify the distribution you predict from the data. The parameter is the processed data that identifies the predictive distribution - without one, there would be no variability in your predictions. $\endgroup$
    – sesqu
    Commented Feb 12, 2011 at 13:00
  • $\begingroup$ could be non parametric density forecast ? $\endgroup$
    – robin girard
    Commented Feb 12, 2011 at 13:02
  • $\begingroup$ Sure, if your model family is, say, an equal mix of multinomials $N(\vec {data}, \vec 1)$, you will have to store all data, since that is your parameter. In this case, you will have to compress the data according to some prior. However, unless your model assumes subsequent time points are independent (and it doesn't, or you wouldn't have specified time series), you can still take advantage of the dependence to refine the prior for $X_t|data_t,X_{t-1}$. $\endgroup$
    – sesqu
    Commented Feb 12, 2011 at 13:35
  • $\begingroup$ @sesqu anyway, parametric modeling is a good way to compress the density so +1 for that $\endgroup$
    – robin girard
    Commented Mar 17, 2011 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.