0
$\begingroup$

I have seen articles on the “curse of dimensionality” and why reducing the number of features can help with overfitting, but imagine we are interested in cases with significantly less features. Why is it that a classification model with fewer features can perform better than one with more features when the feature set is already small (for the sake of example, let's say 10 features)?

$\endgroup$
2
$\begingroup$

Overfitting can occur when we include ten features in a model on dataset $A$ just as easily as it can occur when we include a thousand features on dataset $B$. (This is not the "curse of dimensionality" as such.)

Let's simulate some data with a single "true" predictor and ten unrelated predictors, then check what happens on a holdout set if we include the unrelated predictors in a model. We will use a sample size of $n=30$ in both the training and the test sample and use the Mean Squared Error to assess the predictions.

MSEs

We see that the MSE goes down as we include the single relevant predictor; then, as we add completely unrelated predictors, it goes up again. This is overfitting.

This is a numerical example; the same holds for classification, I'd just rather not go into discussions of what KPI to use in classification.

nn <- 30
n_preds <- 10
n_sims <- 2e3

mses <- matrix(NA,nrow=n_sims,ncol=n_preds+1,dimnames=list(NULL,0:n_preds))
pb <- winProgressBar(max=n_sims)
for ( ii in 1:n_sims ) {
    setWinProgressBar(pb,ii,paste(ii,"of",n_sims))
    set.seed(ii)
    dataset <- data.frame(matrix(runif(2*nn*n_preds),nrow=2*nn,
      dimnames=list(NULL,paste0("X",1:n_preds))))
    dataset$yy <- dataset$X1+rnorm(2*nn)

    model <- lm(yy~1,dataset[1:nn,])
    mses[ii,1] <- mean((yy[-(1:nn)]-predict(model,dataset[-(1:nn),]))^2)

    for ( jj in 1:n_preds ) {
        model <- lm(paste0("yy~",paste0("X",1:jj,collapse="+")),dataset[1:nn,])
        mses[ii,jj+1] <- mean((dataset$yy[-(1:nn)]-predict(model,dataset[-(1:nn),]))^2)
    }
}
close(pb)

plot(colMeans(mses),type="o",pch=19,
  xlab="Number of predictors",ylab="Mean squared prediction error")
$\endgroup$
3
  • $\begingroup$ Hi Stephan, thanks for your response! I have typically seen overfitting in the context of high dimensionality. Does overfitting occur with lower dimensional data because it is essentially picking up on "noise" from unrelated predictors? $\endgroup$
    – Jane Sully
    Jul 10 '18 at 12:17
  • 2
    $\begingroup$ Yes, exactly. Overfitting can occur with a single predictor. I edited my answer to illustrate. $\endgroup$ Jul 10 '18 at 12:45
  • $\begingroup$ Awesome, thanks for the example and explanation. That was super helpful in illustrating the overall concept. $\endgroup$
    – Jane Sully
    Jul 11 '18 at 3:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.