Can eliminating parameters reduce overfitting? I am learning a statistical model, which includes a very large amount of parameters, which results in the risk of over-fitting. If I first learn the model parameters from the data, and then simply remove some of the parameters according to whichever criterion, would I be reducing the chance for overfitting? 
On the one hand, less parameters - less overfitting is supposed to be true.
On the other hand one could claim that once the multi-parameter model was already fit, the parameters themselves were already learned incorrectly - and so reducing the number of parameters now does not help.
I should note that I have been led to believe that the former is true, though I'm not sure why.
 A: While removing parameters of the model and the relearning the weights will reduce overfitting (albeit at the potential cost of underfitting the data) simply removing the parameters after learning without any retraining will have highly unpredictable, and most likely detrimental, effects.
As for the question of "why not?", I think the question of "why?" is more appropriate. That is, your working hypothesis is that removing parameters will improve test error in an overfit model.  I don't see any reason why you would expect this to be true.
A: Overfitting does not always hurt prediction.  In one type of overfitting the variables are highly correlated. So they could be almost functionally related perhaps in a linear fashion.  Suppose that X1=2X2+5X3 exactly without error.  Then you could use any two of the variables in the model and get exactly the same result.  The equations can look very different and still both fit and predict well.  With another type of overfitting, it can lead to a poorer model than one with fewer parameters (when the problem is not collinearity) but rather the inclusion of too many variables results in fitting the noise as well as the signal.  For example suppose we have a response that is a simple linear function of time observed with random mean zero noise.  To fit the line we are given 5 pairs (t, f(t)) at distinct times t, where the function f(t) is "truly" f(t) =at+b and is observed with an additive noise component e(t).  It we fit a straight line we may get a good (but not perfect) linear fit to these five points. The reason it is not perfect is because of the noise.  So the 5 points do not all fall on a single straight line. However if we take a fourth degree polynomial of the form f(t) =a1 +a2 t +a3 t4$^2$ +a4 t$^3$ + a5 t$^4$ we can take the five values of f(t) on the left-hand side of the equation and plug in the five corresponding values of t on the righthand side we will have 5 linear equations in five unknowns that leads to a unique set of values for a1, a2, a3, a4, a5.  So by overfitting we can get a wiggly polynomial to fit the data perfectly.  But this function will not interpolate, extrapolate or predict well. Variable subset selection procedures will take out variables that may be highly correlated with other variables in the model and also remove variables that really have no relationship to the response.  So yes they can reduce or eliminated both types of overfitting problems.  when you drop variables from the model you do not keep the original parameter estimate that came about with the other variables included.  You will refit the new model that contains fewer parameters and the coefficients should change.  So there is no dilemma.
A: This leads you to regularization or alternatively to cross-validation.
