7
$\begingroup$

I have a time series of proportions, $x_t = \frac{a_t}{b_t}$ i.e.

$x_1 = 2 / 30, x_2 = 1/10$, ...

I want to smooth $x_t$. Should I apply a smoothing function directly to $x_t$, or should I smooth $a_t$ and $b_t$ individually, then recompute the proportions?

My gut tells me you should smooth the individual time series, and then recompute. Suppose each of the three time series have noise (this is why we would be interested in smoothing to begin with), then maybe $a_t = f(t) + \epsilon_a$, $b_t = g(t) + \epsilon_b$. Then the proportions $x_t = \frac{f(t) + \epsilon_a}{g(t) + \epsilon_b}$, which could be smoothed by taking $\hat{x_t} = \frac{\hat{f(t)}}{\hat{g(t)}}$

Improve this question
$\endgroup$
3
  • $\begingroup$ Are a and b always going to be integers? $\endgroup$
    – Peter Flom
    Commented Dec 20, 2012 at 19:22
  • $\begingroup$ Sure, let's assume a and b are positive integers $\endgroup$
    – JCWong
    Commented Dec 20, 2012 at 19:31
  • 1
    $\begingroup$ A severe problem with smoothing before taking ratios is that the smoothed proportions could easily be invalid (they could exceed 1). Furthermore, the smoothing ought to account for relative uncertainty in the values. For instance, in the series $a=(1000,4,2000,3,3000,2,4000,1)$, $b=(10000,10,10000,10,10000,10,10000,10)$ the ratios are $(.1,.4,.2,.3,.3,.2,.4,.1)$, which would smooth to around a steady $0.25$, but the values alternate in reliability. Using the most reliable values would suggest the smooth ought to rise steadily from $0.1$ to $0.4$. $\endgroup$
    – whuber ♦
    Commented Dec 21, 2012 at 20:00

2 Answers 2

Reset to default
8
$\begingroup$

It would be erroneous to compute the ratios of the smoothed counts, because it's possible many of the ratios would not be true proportions--they could (easily) wind up outside the valid range from $0$ to $1$. (This happens in the example described below.)

Since the denominators $b_t$ vary substantially, and a count based on such a denominator typically has a variance proportional to the denominator, you should consider making weighted smooths of the ratios using the $b_t$ as the weights.

To study this, I created time series $a_t$ and $b_t$ which vary substantially and are temporally correlated. The $a_t$ have binomial distributions based on the counts in $b_t$, with regularly (sinusoidally) varying probabilities. These, and their ratio $a_t/b_t$, are shown in the first row of the figure below. (The dots for the observations are scaled so their areas are proportional to $b_t$: larger dots represent more reliable data.) Then I compute four possible smooths: the ratio of the smooths, the smoothed ratio, and the two weighted versions. These occupy the next two rows: one for the unweighted version and another for the weighted version. Finally, the bottom row shows scatterplots of these four smooths against the true probabilities.

In experimenting with various values of n (length of series, ranging from $40$ to $200$) and m (maximum possible value of $b_t$, ranging from $3$ to $600$), I find the weighted smoothed ratio is slightly--but visibly--better than the unweighted smoothed ratio, and both are consistently much better than the ratios of the smooths (weighted or not). Compare the bottom right scatterplot ("Wtd Smoothed Ratios") to its unweighted counterpart, second from the left at the bottom ("Smoothed Ratios"). Ideally, these plots would be diagonal lines. Notice the wild behavior of the ratios of smooths and their extreme departures from the ideal. That awfulness is equally apparent in the plots of smooths against the data shown in the first column (second and third rows).

Figure

n <- 80
m <- 30
z <- 1.1
p <- (1 + sin(2 * pi * 1:n / n)) / (1+z)
q <- (1 + sin(5/3 * 2 * pi * 1:n / n)) / (1+z)
b <- ceiling(0.001 + m * q)
scale <- sqrt(b/m)*3
set.seed(26)
a <- rbinom(n, b, p)

col = c(rgb(.8,.2,.2), rgb(.2,.2,.8), rgb(.2,.8,.2), "Black", "Gray")
par(mfrow=c(4,4))
plot(p, type="l", lwd=2, col=col[1], main="Actual")
plot(a, col=col[2], main="Numerators")
plot(b, col=col[3], main="Denominators")
plot(a/b, col=col[5], main="Observed Ratios", cex=scale)

a.smooth <- loess(a ~ as.vector(1:n))
b.smooth <- loess(b ~ as.vector(1:n))
a.b.smooth <- loess(a/b ~ as.vector(1:n))

plot(a.smooth$fitted / b.smooth$fitted, col=col[1], type="l", lwd=2,
     main="Ratio of Smooths")
points(a/b, col=col[5], cex=scale)
plot(a.smooth$fitted, col=col[2], type="l", lwd=2, main="Smoothed Numerators")
plot(b.smooth$fitted, col=col[3], type="l", lwd=2, main="Smoothed Denominators")
plot(a.b.smooth$fitted, col=col[4], type="l", lwd=2, main="Smoothed Ratios")
points(a/b, col=col[5], cex=scale)

a.smooth.w <- loess(a ~ as.vector(1:n), weights=b)
b.smooth.w <- loess(b ~ as.vector(1:n), weights=b)
a.b.smooth.w <- loess(a/b ~ as.vector(1:n), weights=b)

plot(a.smooth.w$fitted / b.smooth.w$fitted, col=col[1], type="l", lwd=2, 
     main="(Weighted version)")
points(a/b, col=col[5], cex=scale)
plot(a.smooth.w$fitted, col=col[2], type="l", lwd=2, main="(Weighted version)")
plot(b.smooth.w$fitted, col=col[3], type="l", lwd=2, main="(Weighted version)")
plot(a.b.smooth.w$fitted, col=col[4], type="l", lwd=2, main="(Weighted version)")
points(a/b, col=col[5], cex=scale)

plot(p, a.smooth$fitted / b.smooth$fitted, col=col[1], type="l", lwd=2, 
     main="Ratio of Smooths")
plot(p, a.b.smooth$fitted, col=col[1], type="l", lwd=2, main="Smoothed Ratios")
plot(p, a.smooth.w$fitted / b.smooth.w$fitted, col=col[1], type="l", lwd=2, 
     main="Ratio of Wtd Smooths")
plot(p, a.b.smooth.w$fitted, col=col[1], type="l", lwd=2, 
     main="Wtd Smoothed Ratios")
Improve this answer
$\endgroup$
1
$\begingroup$

I am no expert in time series, but no one has answered so.... A little playing on a Thursday afternoon:

#MAKE UP DATA
set.seed(16549678)
time <- 1:100
a <- round(runif(100, 5, 10) + rnorm(100,0,1))
b <-  round(a*4+rnorm(100))
x <- a/b
#PLOTS OF RAW DATA, RED IS FOR B
plot(time, a, ylim = c(0,50), type = 'l')
lines(x = time, y=b, col = 2 )

#SMOOTH THE DATA
a.lowess <- lowess(a~time)
b.lowess <- lowess(b~time)
x.lowess <- lowess(x~time)
x.abfirst <- a.lowess$y/b.lowess$y

#Compare
plot(x.lowess$y,x.abfirst)
    plot(time,x.lowess$y, ylim = c(.2,.3))
lines(time, x.abfirst, col = 2)

At least for this example, it doesn't seem to matter much.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ +1 For the numerical experiment. However, if you change the first lines by a <- round(runif(100, 5, 10) + rt(100,df=1)) b <- round(0.1*runif(100, 5, 10)+rt(100,df=1)), then you get very different results. $\endgroup$
    – user10525
    Commented Dec 20, 2012 at 21:38
  • $\begingroup$ @Procrastinator Interesting. So, perhaps the decision should be made on a case by case basis? In your example (at least when I run it) the lowess curve for b goes below 0, and b goes way below 0. In the comments above, a and b were to be considered positive integers. $\endgroup$
    – Peter Flom
    Commented Dec 20, 2012 at 21:40
  • 2
    $\begingroup$ That is right, I ran the code using different seeds. I think that the context gives us the answer: if the interest is on modelling the ratios, then the best model is the one on the ratios, which your code already does. Studying when the ratio of the models is similar to the model of the ratios might be difficult since it will depend on the underlying structure of the processes generating the data. $\endgroup$
    – user10525
    Commented Dec 20, 2012 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.