I am learning a statistical model, which includes a very large amount of parameters, which results in the risk of over-fitting. If I first learn the model parameters from the data, and then simply remove some of the parameters according to whichever criterion, would I be reducing the chance for overfitting?
On the one hand, less parameters - less overfitting is supposed to be true.
On the other hand one could claim that once the multi-parameter model was already fit, the parameters themselves were already learned incorrectly - and so reducing the number of parameters now does not help.
I should note that I have been led to believe that the former is true, though I'm not sure why.