# Flexible and inflexible models in machine learning

I came across a simple question on comparing flexible models (i.e. splines) vs. inflexible models (e.g. linear regression) under different scenarios. The question is:

In general, do we expect the performance of a flexible statistical learning method to perform better or worse than an inflexible method when:

1. The number of predictors $p$ is extremely large, and the number of observations $n$ is small?
2. The variance of the error terms, i.e. $σ^2 = \text{Var}(e)$, is extremely high?

I think for (1), when $n$ is small, inflexible models are better (not sure). For (2), I don't know which model is (relatively) better.

• Generalization error is far from trivial. Unfortunately rules of thumb don't help much in this regard. – Marc Claesen Sep 4 '13 at 20:28
• Looks like this is from James, Witten, Hastie, Tibshirani's Introduction to Statistical Learning – Noel Evans Jan 16 '15 at 16:18
• 1. A flexible method would overfit the small number of observations. 2. A flexible methods fits to the noise in the error terms and increases variance. – Zanark Apr 29 at 13:35

In these 2 situations, comparative performance flexible vs. inflexible model also depends on:

• is true relation y=f(x) close to linear or very non-linear;
• do you tune/constrain flexibility degree of the "flexible" model when fitting it.

If relation is close to linear and you don't constrain flexibility, then linear model should give better test error in both cases because flexible model likely to overfit in both cases.

You can look at it as that:

• In both cases data doesn't contain enough information about true relation (in first case relation is high dimensional and you have not enough data, in second case it corrupted by noise) but
• linear model brings some external prior information about true relation (constrain class of fitted relations to linear ones) and
• that prior info turns out to be right (true relation is close to linear).
• While flexible model doesn't contain prior information (it can fit anything), so it fits to noise.

If however true relation is very non-linear, it's hard to say who will win (both will loose :)).

If you tune/constrain degree of flexibility and do it in a right way (say by cross-validation), then flexible model should win in all cases.

Of course it depends on the underlying data which you should always explore to find out some of its characteristics before trying to fit a model but what I've learnt as general rules of thumb are:

• A flexible model allows you to take full advantage of a large sample size (large n).
• A flexible model will be necessary to find the nonlinear effect.
• A flexible model will cause you to fit too much of the noise in the problem (when variance of the error terms is high).

Well, for the second part, I think more more flexible model will try to fit the model hard and training data contains a high noise, so flexible model will also try to learn that noise and will result in more test error. I know the source of this question as I'm also reading the same book :)

For the first part, I would expect the inflexible model would perform better with a limited number of observations. When n is very small, both models (whether it's flexible or inflexible) would not yield good enough prediction. However, the flexible model would tend to overfit the data and would perform more poorly when it comes to a new testset.

Ideally, I would collect more observations to improve the fitting, but if that is not the case, then I would use the inflexible model, trying to minimize a test error with a new testset.

For the second question I believe the answer is both of them will perform equally (assuming that those errors are irreducible, i.e., this error). More information is provided in An Introduction to Statistical Learning on page 18 (topic: Why estimate $$f$$) where the author explains saying

The accuracy of $$Y$$ as a prediction for $$Y$$ depends on two quantities, which we will call the reducible error and the irreducible error. In general, $$\hat f$$ will not be a perfect estimate for $$f$$, and this inaccuracy will introduce some error. This error is reducible because we can potentially improve the accuracy of $$\hat f$$ by using the most appropriate statistical learning technique to estimate $$\hat f$$. However, even if it were possible to form a perfect estimate for $$f$$, so that our estimated response took the form $$\hat Y = f(X)$$, our prediction would still have some error in it! This is because $$Y$$ is also a function of $$\epsilon$$, which, by definition, cannot be predicted using $$X$$. Therefore, variability associated with $$\epsilon$$ also affects the accuracy of our predictions. This is known as the irreducible error, because no matter how well we estimate $$f$$, we cannot reduce the error introduced by $$\epsilon$$.

• I don't understand this. – Michael R. Chernick Mar 31 '17 at 12:41