20
$\begingroup$

Context:

From a question on Mathematics Stack Exchange (Can I build a program), someone has a set of $x-y$ points, and wants to fit a curve to it, linear, exponential or logarithmic. The usual method is to start by choosing one of these (which specifies the model), and then do the statistical calculations.

But what is really wanted is to find the 'best' curve out of linear, exponential or logarithmic.

Ostensibly, one could try all three, and choose the best fitted curve of the three according to the best correlation coefficient.

But somehow I'm feeling this is not quite kosher. The generally accepted method is to pick your model first, one of those three (or some other link function), then from the data calculate the coefficients. And post facto picking the best of all is cherry picking. But to me whether you're determining a function or coefficients from the data it is still the same thing, your procedure is discovering the best...thing (let's say that which function is -also- another coefficient o be discovered).

Questions:

  • Is it appropriate to choose the best fitting model out of linear, exponential, and logarithmic models, based on a comparison of fit statistics?
  • If so, what is the most appropriate way to do this?
  • If regression helps find parameters (coefficients) in a function, why can't there be a discrete parameter to choose which of three curve families the best would come from?
Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I have added the model-selection tag for your convenience: linking through it will produce a large number of directly relevant threads. Other tags worth looking at include aic. You should eventually discover that the mathematical statement of this problem is missing two essential elements: a description of how and why the points might deviate from a theoretical curve and an indication of the cost of not getting exactly the right curve. Absent those elements, there are many different approaches that can produce different answers, showing that "best" is ill-defined. $\endgroup$
    – whuber ♦
    Commented Oct 5, 2013 at 16:07
  • 1
    $\begingroup$ You could set aside a percentage of your data to do validation on the model and pick the model that fits that set of validation data best. So you would in essence have three distinct sets to split your data into 1. the data to train a single model 2. data that validates each model that allows you to select the best model and 3. your actual final validation data that is not touched. $\endgroup$
    – kleineg
    Commented Aug 9, 2018 at 13:51
  • 1
    $\begingroup$ @kleineg That sounds like the right direction. The choice of model (eg between lin/exp/log) is like a single model hyperparameter, which are in some ways just another stage of regular parameters, and stepping into it by separate train/validate/test stages could be generalized. $\endgroup$
    – Mitch
    Commented Aug 9, 2018 at 14:34
  • $\begingroup$ Relevant: {A subtle way to overfit](johndcook.com/blog/2015/03/17/a-subtle-way-to-over-fit) - choosing between multiple model functions (eg exp vs linear vs log) is just another parameter. You could think of it as a hyperparameter (which would need a validation step) or a regular parameter in a complicated function of combination (where it would be tested in a test step). $\endgroup$
    – Mitch
    Commented Aug 29, 2019 at 12:44

4 Answers 4

Reset to default
13
$\begingroup$
  • You might want to check out the free software called Eureqa. It has the specific aim of automating the process of finding both the functional form and the parameters of a given functional relationship.
  • If you are comparing models, with different numbers of parameters, you will generally want to use a measure of fit that penalises models with more parameters. There is a rich literature on which fit measure is most appropriate for model comparison, and issues get more complicated when the models are not nested. I'd be interested to hear what others think is the most suitable model comparison index given your scenario (as a side point, there was recently a discussion on my blog about model comparison indices in the context of comparing models for curve fitting).
  • From my experience, non-linear regression models are used for reasons beyond pure statistical fit to the given data:
    1. Non-linear models make more plausible predictions outside the range of the data
    2. Non-linear models require fewer parameters for equivalent fit
    3. Non-linear regression models are often applied in domains where there is substantial prior research and theory guiding model selection.
Improve this answer
$\endgroup$
8
$\begingroup$

This is a question that is valid in very diverse domains.

The best model is the one that can predict data points that were not used during the parameter estimation. Ideally one would compute model parameters with a subset of the data set, and evaluate the fit performance on another data set. If you are interested in the details make a search with "cross-validation".

So the answer to first question, is "No". You cannot simply take the best fitting model. Image you are fitting a polynomial with Nth degree to N data points. This will be a perfect fit, because all the model will exactly pass on all data points. However this model will not generalize to new data.

When you do not have enough data to go through the cross-validation procedure in a sound manner, then you can use metrics such as AIC or BIC. These metrics punishes simultaneously the amplitude of residuals and the number of parameters in your model but makes strong assumptions on the generative processes of your data. As these metrics are sensitive to over-fitting, they can be used as a proxy for model selection.

Improve this answer
$\endgroup$
5
$\begingroup$

Since plenty of people routinely explore the fit of various curves to their data, I don't know where your reservations are coming from. Granted, there is the fact that a quadratic will always fit at least as well as a linear, and a cubic, at least as well as a quadratic, so there are ways to test the statistical significance of adding such a nonlinear term and thus to avoid needless complexity. But the basic practice of testing many different forms of a relationship is just good practice. In fact, one might start with a very flexible loess regression to see what is the most plausible kind of curve to fit.

Improve this answer
$\endgroup$
10
  • 3
    $\begingroup$ Whether quadratic fits better, will depend on how you have operationalised good fit. In particular, if you use a measure of fit that penalises models with more parameters (e.g., AIC), then, for example, fit can be worse for quadratic versus linear. $\endgroup$
    – Jeromy Anglim
    Commented Apr 8, 2011 at 3:28
  • 9
    $\begingroup$ @rolando, perhaps I am misunderstanding, but, frankly this sort of (unqualified) advice is precisely the kind of thing that, as statisticians, we spend so much time "fighting" against. Particularly, if the OP is interested in anything beyond simple curve fitting, e.g., prediction or inference, it is very important to understand the implications of the "just try whatever you can think of" approach to statistics. $\endgroup$
    – cardinal
    Commented Apr 8, 2011 at 12:46
  • 2
    $\begingroup$ I'm having trouble reconciling these comments with the tradition of Anscombe, Tukey, Mosteller, Tufte, and Cleveland, which emphasizes the need to visualize and explore data and to size up the shape of each relationship before building a model, establishing coeffiencients, or generating other statistics. $\endgroup$
    – rolando2
    Commented May 19, 2011 at 18:40
  • 9
    $\begingroup$ There is a lot of controversy regarding their approaches. An over-simplified way to summarize these issues is that if you want to learn about patterns and make new discoveries that need later validation, exploratory analysis is appropriate. If you want to draw inference (reason from particular sample to general population using P-values, confidence intervals, etc.) then not so much. $\endgroup$
    – Frank Harrell
    Commented May 21, 2011 at 12:34
  • 4
    $\begingroup$ This is the most productive comment thread I've seen on CV, especially the exchange b/t rolando2 (3^) & @FrankHarrell. I also find both approaches very appealing. My own resolution is to plan what to test beforehand & only fit/test that model for the sake of drawing firm conclusions, but also thoroughly explore the data (w/o believing the results necessarily hold) for the sake of discovering what might be true & planning for the next study. (Should I run another study & check something, would it be interesting/important?) The key is your beliefs about the results of these analyses. $\endgroup$
    – gung - Reinstate Monica
    Commented Feb 18, 2012 at 20:38
4
$\begingroup$

You really need to find a balance between the science/theory that leads to the data and what the data tells you. Like others have said, if you let yourself fit any possible transformation (polynomials of any degree, etc.) then you will end up overfitting and getting something that is useless.

One way to convince yourself of this is through simulation. Choose one of the models (linear, exponential, log) and generate data that follows this model (with a choice of the parameters). If your conditional variance of the y values is small relative to the spread of the x variable then a simple plot will make it obvious which model was chosen and what the "truth" is. But if you choose a set of parameters such that it is not obvious from the plots (probably the case where an analytic solution is of interest) then analyze each of the 3 ways and see which gives the "best" fit. I expect that you will find that the "best" fit is often not the "true" fit.

On the other hand, sometimes we want the data to tell us as much as possible and we may not have the science/theory to fully determine the nature of the relationship. The original paper by Box and Cox (JRSS B, vol. 26, no. 2, 1964) discusses ways to compare between several transformations on the y variable, their given set of transformations have linear and log as special cases (but not exponential), but nothing in the theory of the paper limits you to only their family of transformations, the same methodology could be extended to include a comparison between the 3 models that you are interested in.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.