0
$\begingroup$

Can a nonlinear model be more parsimonious than a linear model? How to prove it mathematically?

$\endgroup$
4
$\begingroup$

There are cases in statistical modelling where a non-linear model with only a small number of parameters fits the data well, but any attempt to apply a linear model would require many more parameters to get a decent fit. This could occur, for example, in a periodic regression model:

$$Y_i = \beta_0 + \alpha \sin(2 \pi \phi (x_i-\mu)) + \sigma \varepsilon_i \quad \quad \quad \varepsilon_i \sim \text{IID N}(0, 1).$$

Here the frequency parameter $0 < \phi < 1$ enters the model in a non-linear way (the amplitude and phase angle parameters are linearisable), and there is no corresponding linear model that exactly achieves this same form. It is possible to create a linear regression model that approximates the sinusoidal signal (e.g., with a power series with a large number of powers) but this will only be a good approximation over the data range if you have a lot of parameters. For example, you might decide to approximate this signal as a sum of periodic signals with harmonic frequencies, yielding the linear regression model:

$$Y_i \approx \beta_0 + \sum_{i=1}^m \Bigg[ \beta_{1s} \sin \Bigg( \frac{2 \pi i}{m} x_i \Bigg) + \beta_{1c} \cos \Bigg( \frac{2 \pi i}{m} x_i \Bigg) \Bigg] + \sigma \varepsilon_i \quad \quad \quad \varepsilon_i \sim \text{IID N}(0, 1).$$

In this particular case you have a non-linear model with only $4$ unknown coefficient parameters and a linear approximation with $1+2m$ unknown coefficient parameters. If $m$ is reasonably large then the linear approximation is less parsimonious in the sense that it has more parameters. This would be the kind of case where it is reasonable to say that a non-linear model is more parsimonious than its corresponding linear approximation.

$\endgroup$
0
$\begingroup$

The functional space of nonlinear functions includes any linear function, so linear functional space is always more parsimonious than the nonlinear functional space; I am not sure exact definition of parsimony though.

$\endgroup$
0
$\begingroup$

Parsimony means the simplicity of a model to be accurate. At times models with simple parameters are more accurate at prediction than models with many parameters. We use simple models to avoid over fitting.

$\endgroup$
  • $\begingroup$ I don't think this helps much, as parsimony and simplicity are close siblings, but accuracy is not implied by either. Sure, overfitting and underfitting too imply that we seek parsimony combined with accuracy to the extent possible, which may mean compromise. $\endgroup$ – Nick Cox May 2 '18 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.