I have a 7 points dataset that looks like this:

dates <- as.Date(sprintf('2011-01-%s', 15:21))
vals  <- c(0.05816136, 0.31020050, 0.45542547, 0.52401006, 0.56796386, 0.59687058, 0.61741189)

dat <- data.frame(date=dates, val=vals)

plot(dat$date, dat$val, type='b')

The plot is given here.

plot of original data

I'd like to create a prediction based on this dataset and I approach it so:

transf   <- function(orig) 170 ** orig
untransf <- function(trnsf) log(trnsf, base=170)

# transform the dependent variable to linearity and do a linear fit
fit <- lm(I(transf(val)) ~ date, data=dat)

test.dates <- as.Date(sprintf('2011-01-%s', 15:27))
plot(test.dates, untransf(predict(fit, newdata=list(date=test.dates))), type='b')
points(dat$date, dat$val, col='red', type='b')

predicted vs true

However, as the second plot suggests the true values (red) follow slightly different shape than the so predicted ones. What's the problem with the approach I'm taking? Is there a general problem in transforming the data using an exponential function and then using the inverted prediction?

  • $\begingroup$ what are your vals? $\endgroup$ – Glen_b Sep 26 '13 at 10:19
  • $\begingroup$ If you condense the information of seven points to a two-parameter function, it would be mere chance to end up with exactly the same values. I don't know if this already answers your question though. $\endgroup$ – Michael M Sep 26 '13 at 10:19
  • $\begingroup$ @Glen_b is that relevant? Those are daily usage rates. $\endgroup$ – nikola Sep 26 '13 at 11:43
  • $\begingroup$ Yes, sometimes what is being modeled can make a big difference to suitable models (with count data turned into proportions, for example). Even without any more detail than 'usage rates' we can guess that the values must be non-negative, for example, which may suggest considering GLMs, perhaps. $\endgroup$ – Glen_b Sep 26 '13 at 15:16

There are a number of issues with such a strong transformation.

1) Means don't carry across nonlinear transformations (that is, $\text{E}[\varphi(x)] \neq \varphi[\text{E}(x)]$; indeed, for a convex function, $\varphi\left(\text{E}\left[X\right]\right) \leq \text{E}\left[\varphi(X)\right]$). In some cases the bias can be severe.

2) by ignoring the implied effect of the transformation on variance, the relative weights to observations can differ quite markedly (equal weights on the transformed scale imply unequal weights on the untransformed scale); since your model doesn't fit all that well given the smoothness of the data, the lack of fit to the relatively high-variance points can become strong.

My suggestion is to try a nonlinear least squares model to the original data.

You should incorporate any features you understand about the process into the model. For example if you expect it to have a horizontal asymptote eventually, you can incorporate that understanding; this kind of information is critical to reasonable behavior beyond the end of the sampling period. (For example, if you know that 1 is an upper bound, that information would be important.)

For example, here's an asymptotic exponential fit ($\text{E}(y) = \beta_0 + \beta_1 e^{-\beta_2 x}$):

x <- 1:7
valfit <- nls(dat$val~b0+b1*exp(-b2*x),start=list(b0=.7,b1=-1,b2=1))

Asymptotic exponential

It's not a perfect fit (I wouldn't want to extrapolate more than a day or so), but it's a good deal better than the fit in your plot (which is different data). Note that since you (presumably) chose the "170" based on the data, you effectively have three parameters there.

If you do use that particular model above, note that it has a self-start function (I think it's called SSAsymp), so you don't need to supply starting values as I did (I chose those values simply by looking at the plot of the data).

If you thought the function might eventually be linear, you might add a linear term to that asymptotic model, which produces predictions like so:


But with only 7 points so many parameters is hard to justify.

You may (and hopefully do) have a logical reason to choose a better function than mine, of course (and any well justified choice should certainly be used over what I have suggested, which is merely an example) -- but on the other hand I seriously doubt you have any good justification for taking 170th powers.

As @FrankHarrell suggests a regression spline on the untransformed data is well worth considering. In that case I'd probably suggest a natural regression spline (with low degrees of freedom).

Such an approach has a number of advantages.

Note that natural splines will be linear beyond the ends of the data, so out of sample predictions from a well-fitting model will be straight lines whose slope is close to the general trend at the end of the observed data.

Here's an example, which shows that 3df may (slightly) underfit, while 4 may overfit:

 library(splines)  # comes with R
 with(dat, {
   valnsfit3 = lm(val ~ ns(x, df=3))
   valnsfit4 = lm(val ~ ns(x, df=4))
   plot(date,   val, type='b')
   points(date, fitted(valnsfit3), col=3, type="l")
   points(date, fitted(valnsfit4), col=4, type="l")

Given such low sample size I'd probably have just used the smaller model, though if you had more data (around say 10 points) I'd probably go with 4df.

The code above generates the plot below; the fit in green has 3 df and the blue one has 4 df.

natural spline fits of degree 3 (green) and 4 (blue)

  • 3
    $\begingroup$ Also consider a regression spline on the untransformed data. $\endgroup$ – Frank Harrell Sep 26 '13 at 11:17
  • $\begingroup$ Thank you very much, that's a fantastically put answer, very informative and thorough. $\endgroup$ – nikola Sep 26 '13 at 11:32
  • $\begingroup$ The suggestion by @FrankHarrell of regression splines on untransformed data is a good one; I have included an example using natural splines above. $\endgroup$ – Glen_b Sep 26 '13 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.