# Flexible and inflexible models in machine learning

I came across a simple question on comparing flexible models (i.e. splines) vs. inflexible models (e.g. linear regression) under different scenarios. The question is:

In general, do we expect the performance of a flexible statistical learning method to perform better or worse than an inflexible method when:

1. The number of predictors $p$ is extremely large, and the number of observations $n$ is small?
2. The variance of the error terms, i.e. $σ^2 = \text{Var}(e)$, is extremely high?

I think for (1), when $n$ is small, inflexible models are better (not sure). For (2), I don't know which model is (relatively) better.

• Generalization error is far from trivial. Unfortunately rules of thumb don't help much in this regard. – Marc Claesen Sep 4 '13 at 20:28
• Looks like this is from James, Witten, Hastie, Tibshirani's Introduction to Statistical Learning – Noel Evans Jan 16 '15 at 16:18
• 1. A flexible method would overfit the small number of observations. 2. A flexible methods fits to the noise in the error terms and increases variance. – Zanark Apr 29 '19 at 13:35

In these 2 situations, comparative performance flexible vs. inflexible model also depends on:

• is true relation y=f(x) close to linear or very non-linear;
• do you tune/constrain flexibility degree of the "flexible" model when fitting it.

If relation is close to linear and you don't constrain flexibility, then linear model should give better test error in both cases because flexible model likely to overfit in both cases.

You can look at it as that:

• In both cases data doesn't contain enough information about true relation (in first case relation is high dimensional and you have not enough data, in second case it corrupted by noise) but
• linear model brings some external prior information about true relation (constrain class of fitted relations to linear ones) and
• that prior info turns out to be right (true relation is close to linear).
• While flexible model doesn't contain prior information (it can fit anything), so it fits to noise.

If however true relation is very non-linear, it's hard to say who will win (both will loose :)).

If you tune/constrain degree of flexibility and do it in a right way (say by cross-validation), then flexible model should win in all cases.

Of course it depends on the underlying data which you should always explore to find out some of its characteristics before trying to fit a model but what I've learnt as general rules of thumb are:

• A flexible model allows you to take full advantage of a large sample size (large n).
• A flexible model will be necessary to find the nonlinear effect.
• A flexible model will cause you to fit too much of the noise in the problem (when variance of the error terms is high).

Well, for the second part, I think more more flexible model will try to fit the model hard and training data contains a high noise, so flexible model will also try to learn that noise and will result in more test error. I know the source of this question as I'm also reading the same book :)

For the first part, I would expect the inflexible model would perform better with a limited number of observations. When n is very small, both models (whether it's flexible or inflexible) would not yield good enough prediction. However, the flexible model would tend to overfit the data and would perform more poorly when it comes to a new testset.

Ideally, I would collect more observations to improve the fitting, but if that is not the case, then I would use the inflexible model, trying to minimize a test error with a new testset.

For each of parts (a) through (d), indicate whether i. or ii. is correct, and explain your answer. In general, do we expect the performance of a flexible statistical learning method to perform better or worse than an inflexible method when :

The sample size n is extremely large, and the number of predictors p is small ?

Better. A flexible method will fit the data closer and with the large sample size, would perform better than an inflexible approach.

The number of predictors p is extremely large, and the number of observations n is small ?

Worse. A flexible method would overfit the small number of observations.

The relationship between the predictors and response is highly non-linear ?

Better. With more degrees of freedom, a flexible method would fit better than an inflexible one.

The variance of the error terms, i.e. σ2=Var(ε), is extremely high ?

Worse. A flexible method would fit to the noise in the error terms and increase variance.

Taken from here.

1. If n is small and p is very large , we have a small observation set in which the flexible model might find non-existing relationships due to high number of predictors .

2. If the var of error terms is very high , the flexible models will go ahead and try to fit the unexplained error terms , so we should use a rather inflexible method.

Part a: Since the sample (training data) is small, both the models will not very well capture the true underlying relationship compared to the case where the sample size is large as a large sample means that the training data resembles very closely the underlying population data. So, the test data is likely to be very different from the sample data in this case.

With test data (data that we really care about), flexible models will underperform as they are fitted to a small training dataset. With a large number of predictors, the over-fitting will again be very high (much higher in flexible models compared to inflexible models) and a change in the input data can give very unreliable and inaccurate results. Again this will ensure flexible models under-perform the inflexible models.

Part b: If the variance of error terms is very high, the flexible models will try to fit the irreducible error (noise) in the model. This would be the case with the inflexible models as well but the results will be very drastic in the case of flexible models. So we should use an inflexible method in such a case.

For the second question I believe the answer is both of them will perform equally (assuming that those errors are irreducible, i.e., this error). More information is provided in An Introduction to Statistical Learning on page 18 (topic: Why estimate $$f$$) where the author explains saying

The accuracy of $$Y$$ as a prediction for $$Y$$ depends on two quantities, which we will call the reducible error and the irreducible error. In general, $$\hat f$$ will not be a perfect estimate for $$f$$, and this inaccuracy will introduce some error. This error is reducible because we can potentially improve the accuracy of $$\hat f$$ by using the most appropriate statistical learning technique to estimate $$\hat f$$. However, even if it were possible to form a perfect estimate for $$f$$, so that our estimated response took the form $$\hat Y = f(X)$$, our prediction would still have some error in it! This is because $$Y$$ is also a function of $$\epsilon$$, which, by definition, cannot be predicted using $$X$$. Therefore, variability associated with $$\epsilon$$ also affects the accuracy of our predictions. This is known as the irreducible error, because no matter how well we estimate $$f$$, we cannot reduce the error introduced by $$\epsilon$$.

• I don't understand this. – Michael R. Chernick Mar 31 '17 at 12:41