8
$\begingroup$

We have fitted a nonlinear function to observed data. The next step should be the assessment of the goodness of fit of this function (like $R^2$ for linear models).

What are the usual ways to measure this?

Edit 1:

The fitting was performed as follows:

  1. Perform a linear regression with independent variables A and B.
  2. Calculate distribution's parameters from regression parameters. (The distribution is nonlinear and has variable C as an input.)
  3. Assess goodness of fit of nonlinear distribution by comparing estimated to observed data.

Edit 2:

Examples for the steps mentioned above:

  1. Regression model: $log(y) = \beta_0 + \beta_1 \centerdot log(a) + \beta_2 \centerdot log(b)$
  2. $\rho = -\frac{\beta_0}{\beta_1}$ and $\theta = \beta_2$ for the following nonlinear distribution: $f(a) = \rho \centerdot a^{-\theta}$
  3. Assess the goodness of fit of $f(a)$ with a given set of $(a, f(a))$ observations.
$\endgroup$
  • 5
    $\begingroup$ "Goodness of fit" depends on how the fit was performed. For instance, the appropriate GoF measure for a maximum likelihood estimator ought to differ from the GoF measure for a least squares estimator when the random component is not an additive Normal variable. If you have a specific problem in mind, then you might therefore wish to indicate how you performed the fitting. $\endgroup$ – whuber Sep 8 '14 at 15:58
  • 2
    $\begingroup$ You might be interested in: Magee, L. (1990). $R^{2}$ measures based on Wald and likelihood ratio joint significance tests. The American Statistician, 44(3):250–253, and Pesaran, M. H. and Smith, R. J. (1994). A generalized $R^{2}$ criterion for regression models estimated by the instrumental variables method. Econometrica, 62(3):705–710. $\endgroup$ – Alexis Sep 8 '14 at 16:59
  • 1
    $\begingroup$ @whuber I have added the description of the performed steps in the questions above. $\endgroup$ – Marco Sep 9 '14 at 7:16
  • $\begingroup$ @Alexis Thanks for the references, I will have a look at them. $\endgroup$ – Marco Sep 9 '14 at 7:16
  • 1
    $\begingroup$ @whuber Would it be a feasible way to calculate the correlation between $f(a)$ and the given observations to assess the goodness of fit? $\endgroup$ – Marco Sep 11 '14 at 9:31
2
$\begingroup$

There maybe more to it, but to me it seems that you just want to determine goodness-of-fit (GoF) for a function f(a), fitted to a particular data set (a, f(a)). So, the following only answers your third sub-question (I don't think the first and second are directly relevant to the third one).

Usually, GoF can be determined parametrically (if you know the distribution's function parameters) or non-parametrically (if you don't know them). While you may be able to figure out parameters for the function, as it appears to be exponential or gamma/Weibull (assuming that data is continuous). Nevertheless, I will proceed, as if you didn't know the parameters. In this case, it's a two-step process. First, you need to determine distribution parameters for your data set. Second, you perform a GoF test for the defined distribution. To avoid repeating myself, at this point I will refer you to my earlier answer to a related question, which contains some helpful details. Obviously, this answer can easily be applied to distributions, other than the one mentioned within.

In addition to GoF tests, mentioned there, you may consider another test - chi-square GoF test. Unlike K-S and A-D tests, which are applicable only to continuous distributions, chi-square GoF test is applicable to both discrete and continuous ones. Chi-square GoF test can be performed in R by using one of several packages: stats built-in package (function chisq.test()) and vcd package (function goodfit() - for discrete data only). More details are available in this document.

$\endgroup$
0
$\begingroup$

Well, in Machine Learning the thing called Cross Validation is performed pretty often for purpose of model testing (test if that type of a model with these hyper-parameters - like number of degrees of freedom or whatever - fits your problem) - you split your data several times into train and test data sets, then run optimization over training set and compute whatever quality over tests data. The most confidential way is to run so called "QxT-fold cross validation". The pseudocode could could like:

cv_values = []
for t in range(T):
    split = randomsplit(data, number_of_parst = Q)
    for test_id in range(Q):
        model.fit(split[:test_id] + split[test_id + 1:] # test on everything excepting test_id
        cv_values.append(model.test(split[test_id]))

cv_values.mean() # whatever
$\endgroup$
  • $\begingroup$ Thanks for your answer. So, how does your model.test(...) function look like? $\endgroup$ – Marco Sep 9 '14 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.