So, I understand overfitting (bonus question: precise statistical definition of overfitting?). You don't want to match the noise in your sample.
What I don't understand is why this requires a human to hide data from the algorithm. Doesn't this imply that you don't have the best algorithm to begin with if extra information will make your model worse? I know that you could program the computer to compute the AIC/BIC/etc and hide info from itself, but it still seems like more information shouldn't make models worse.
Put another way, removing a variable is equivalent to hardcoding its coefficient to zero. It seems quite strange that for all these variables, exactly zero would happen to be best coefficient estimate given the available data. Surely, given some information, we could improve upon this naive guess, even if only a tiny bit.
Why don't we have algorithms which can produce models involving all the variables, even those whose effect size is too small to be reliably detected from the sample, and estimate a coefficient for them without overfitting and making the fit on the holdout set worse?