16
$\begingroup$

Can anybody please explain to me the real difference(s) between regression analysis and curve fitting (linear and nonlinear), with an example if possible?

It seems that both try to find a relationship between two variables (dependent vs independent) and then determine the parameter (or coefficient) associated with the models being proposed. For example, if I have a set of data like:

Y = [1.000 1.000 1.000 0.961 0.884 0.000] 
X = [1.000 0.063 0.031 0.012 0.005 0.000]

Can anybody suggest a correlation formula between these two variables? I am having a difficulty understanding the difference between these two approaches. If you prefer to support your answer with other data sets, it's OK since that one seems hard to fit (perhaps only for me).

The above data set represents the $x$ and $y$ axes of a receiver operating characteristic (ROC) curve, where $y$ is the true positive rate (TPR) and $x$ is the false positive rate (FPR).

I am trying to fit a curve, or do a regression analysis as per my original question, not sure yet, among these points to estimate the TPR for any particular FPR (or vice-versa).

First, is it scientifically acceptable to find such a curve fitting function between two independent variables (TPR and FPR)?

Second, is it scientifically acceptable to find such a function if I know that the distributions of the actual negative and the actual positive cases are not normal?

$\endgroup$
  • 1
    $\begingroup$ Terms are (unfortunately) used differently by different people & in different contexts. Can you link to / provide an example where people are distinguishing between them? $\endgroup$ – gung May 8 '15 at 17:31
  • $\begingroup$ That's what I am trying to figure out, how they are different and how I can distinguish between them. $\endgroup$ – Ali Sultan May 8 '15 at 17:34
  • 1
    $\begingroup$ Fair enough, but did somebody tell you they were supposed to be different? $\endgroup$ – gung May 8 '15 at 17:36
  • 2
    $\begingroup$ On this site some people have used "curve fitting" in senses that cannot be considered regression. For instance, some of them view estimating a density as a form of "curve fitting" to a histogram. $\endgroup$ – whuber May 8 '15 at 18:39
22
$\begingroup$

I doubt that there is a clear and consistent distinction across statistically minded sciences and fields between regression and curve-fitting.

Regression without qualification implies linear regression and least-squares estimation. That doesn't rule out other or broader senses: indeed once you allow logit, Poisson, negative binomial regression, etc., etc. it gets harder to see what modelling is not regression in some sense.

Curve-fitting does literally suggest a curve that can be drawn on a plane or at least in a low-dimensional space. Regression is not so bounded and can predict surfaces in a several dimensional space.

Curve-fitting may or may not use linear regression and/or least squares. It might refer to fitting a polynomial (power series) or a set of sine and cosine terms or in some other way actually qualify as linear regression in the key sense of fitting a functional form linear in the parameters. Indeed curve-fitting when nonlinear regression is regression too.

The term curve-fitting could be used in a disparaging, derogatory, deprecatory or dismissive sense ("that's just curve fitting!") or (almost the complete opposite) it might refer to fitting a specific curve carefully chosen with specific physical (biological, economic, whatever) rationale or tailored to match particular kinds of initial or limiting behaviour (e.g. being always positive, bounded in one or both directions, monotone, with an inflexion, with a single turning point, oscillatory, etc.).

One of several fuzzy issues here is that the same functional form can be at best empirical in some circumstances and excellent theory in others. Newton taught that trajectories of projectiles can be parabolic, and so naturally fitted by quadratics, whereas a quadratic fitted to age dependency in the social sciences is often just a fudge that matches some curvature in the data. Exponential decay is a really good approximation for radioactive isotopes and a sometimes not too crazy guess for the way that land values decline with distance from a centre.

Your example gets no explicit guesses from me. Much of the point here is that with a very small set of data and precisely no information on what the variables are or how they are expected to behave it could be irresponsible or foolish to suggest a model form. Perhaps the data should rise sharply from (0, 0) and then approach (1, 1), or perhaps something else. You tell us!

Note. Neither regression nor curve-fitting is limited to single predictors or single parameters (coefficients).

$\endgroup$
  • 2
    $\begingroup$ "Curve-fitting" connotes something a-theoretical (eg, lowess) to me. Economists sometimes deride a-theoretical function fitting as 'charting', which sounds similar to some usages of curve-fitting. I think that it (eg lowess) has both pros & cons, when understood correctly. It's hard to know how somebody meant the terms distinctly w/o more context, though. $\endgroup$ – gung May 8 '15 at 17:55
  • 1
    $\begingroup$ @gung I think there is similar part-jocular, part-serious usage across several natural (and unnatural) sciences. One of the issues is that given enough parameters, you necessarily have a lot of wiggle room. I'm reminded of time series models that allow not just ARIMA but also sinusoidal terms and steps, ramps and spikes wherever the data suggest. $\endgroup$ – Nick Cox May 8 '15 at 18:00
  • $\begingroup$ I second @gung, curve fitting has a more nonparametric connotation, at least to me. $\endgroup$ – Christoph Hanck May 8 '15 at 18:22
  • 1
    $\begingroup$ @ChristophHanck Please don't bring "nonparametric" into this! The discussion is muddy enough already! $\endgroup$ – Nick Cox May 8 '15 at 18:33
  • 1
    $\begingroup$ @gung: Thinking of smoothing splines and RKHS methods in general as the backbone of "curve-fitting" for example I feel "curve fitting" to be much more theoretical than "regression". (+1 to NickCox for this answer) $\endgroup$ – usεr11852 May 9 '15 at 0:46
8
$\begingroup$

In addition to @NickCox's excellent answer (+1), I wanted to share my subjective impression on this somewhat fuzzy terminology topic. I think that a rather subtle difference between the two terms lies in the following. On one hand, regression often, if not always, implies an analytical solution (reference to regressors implies determining their parameters, hence my argument about analytical solution). On the other hand, curve fitting does not necessarily imply producing an analytical solution and IMHO often might be and is used as an exploratory approach.

$\endgroup$
  • 2
    $\begingroup$ Can't something with an analytical solution be used for exploratory reasons too? I don't think I get the opposition you are making. $\endgroup$ – amoeba May 9 '15 at 13:08
  • $\begingroup$ @amoeba: Analytical solutions certainly can be used for exploratory research as well. However, the point I am making is about the most popular implied essence of the terms in question. $\endgroup$ – Aleksandr Blekh May 9 '15 at 23:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.