I need to fit a model to an existing dataset such that I can use the parameters to replicate the best-fit curve to manage the behaviour of an application. I've been trying to fit a polynomial model, but cannot escape the problem of (I think) edge effects (?) - despite fitting the model to a broader-ranged dataset than my operational requirements (exemplified by newdat).

The following R script illustrates:

dat = read.csv('https://gist.githubusercontent.com/geotheory/10ad6b2051e69213f81ccf2366938cda/raw/485c98381579d74ec581e34c1ebfa80b66b76d69/poly-test.csv')

# polynomial model
pm = lm(vy ~ poly(vx, degree=20, raw=TRUE), data = dat)

newdat = data.frame(vx = 10^seq(4,6,.02))
newdat$vy = predict(pm, newdata = newdat)
#> Warning in predict.lm(pm, newdata = newdat): prediction from a rank-
#> deficient fit may be misleading

plot(dat, log='x', pch=16, cex=.6)
lines(newdat, col='red', lwd=3)

enter image description here

I've experimented with different degrees but nothing works.

If another model is better suited I'd be happy to take advice. Grateful for suggestions.

  • 2
    $\begingroup$ These points simply don't exhibit polynomial behavior (not even on this log-linear plot); using degree 20 (!) is going to cause all kinds of overfitting problems, as you can see. What does vy represent, how is it measured, and how do you expect it to vary with vx? $\endgroup$ – whuber Jan 23 '19 at 23:23
  • $\begingroup$ Hmm. I've used this approach for similar problems in the past. Perhaps I need to fit to fewer data points. Explaining vx/vy is going to be too painful - save to say I'm experimenting to resolve an problem to do with subsetting datasets that are typoically log-ish but not quite log-ish enough for a simple log model. $\endgroup$ – geotheory Jan 23 '19 at 23:49
  • $\begingroup$ It occurs to me this question has paved the way to an unforgivably bad stats joke. $\endgroup$ – geotheory Jan 24 '19 at 21:25
  • $\begingroup$ There are many seemingly simple functions that cannot be will approximated by polynomials. Try restricted cubic splines instead. $\endgroup$ – rep_ho Jan 24 '19 at 22:02
  • $\begingroup$ @rep_ho @Glen_b Yes it probably does belong better out of sight by default. Think log polynomial.. $\endgroup$ – geotheory Jan 24 '19 at 23:50

You are fitting a linear vx but plotting it on a log scale. Try taking the log in your lm model fit and then plot.

 pm = lm(vy ~ poly(I(log(vx)), degree=5, raw=TRUE), data = dat)

enter image description here

  • $\begingroup$ Very nice, thanks Dave. What is the role of I() here? The model seems to work fine without it. $\endgroup$ – geotheory Jan 24 '19 at 9:37
  • $\begingroup$ Glad it work out for your. The I() function is the AsIs function. From help, "it is used to inhibit the interpretation of operators such as "+", "-", "*" and "^" as formula operators..." . I thought I needed it to properly handle the log function, but was mistaken. $\endgroup$ – Dave2e Jan 24 '19 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.