Why fewer feature classification models can perform better? I have seen articles on the “curse of dimensionality” and why reducing the number of features can help with overfitting, but imagine we are interested in cases with significantly less features. Why is it that a classification model with fewer features can perform better than one with more features when the feature set is already small (for the sake of example, let's say 10 features)?
 A: Overfitting can occur when we include ten features in a model on dataset $A$ just as easily as it can occur when we include a thousand features on dataset $B$. (This is not the "curse of dimensionality" as such.)
Let's simulate some data with a single "true" predictor and ten unrelated predictors, then check what happens on a holdout set if we include the unrelated predictors in a model. We will use a sample size of $n=30$ in both the training and the test sample and use the Mean Squared Error to assess the predictions. 

We see that the MSE goes down as we include the single relevant predictor; then, as we add completely unrelated predictors, it goes up again. This is overfitting.
This is a numerical example; the same holds for classification, I'd just rather not go into discussions of what KPI to use in classification.
nn <- 30
n_preds <- 10
n_sims <- 2e3

mses <- matrix(NA,nrow=n_sims,ncol=n_preds+1,dimnames=list(NULL,0:n_preds))
pb <- winProgressBar(max=n_sims)
for ( ii in 1:n_sims ) {
    setWinProgressBar(pb,ii,paste(ii,"of",n_sims))
    set.seed(ii)
    dataset <- data.frame(matrix(runif(2*nn*n_preds),nrow=2*nn,
      dimnames=list(NULL,paste0("X",1:n_preds))))
    dataset$yy <- dataset$X1+rnorm(2*nn)

    model <- lm(yy~1,dataset[1:nn,])
    mses[ii,1] <- mean((yy[-(1:nn)]-predict(model,dataset[-(1:nn),]))^2)

    for ( jj in 1:n_preds ) {
        model <- lm(paste0("yy~",paste0("X",1:jj,collapse="+")),dataset[1:nn,])
        mses[ii,jj+1] <- mean((dataset$yy[-(1:nn)]-predict(model,dataset[-(1:nn),]))^2)
    }
}
close(pb)

plot(colMeans(mses),type="o",pch=19,
  xlab="Number of predictors",ylab="Mean squared prediction error")

