Thoughts on model self-penalization amidst difficult parameter estimation

Question

It is well accepted that one should account for model complexity when performing model comparisons, and the general procedure is to penalize more complex models more strongly. While this makes sense when the parameters of a given model are easily estimated (i.e. analytically, as with the mean, variance, etc), it occurs to me that if parameter estimation is a more difficult endeavour then more complex models may to some degree self-penalize. That is, if parameter estimation requires search of a parameter space, presumably larger parameter spaces are more difficult to search and therefore any finite search algorithm is more likely (as the parameter space expands) to terminate prior to finding the point of global maximum likelihood.

Has this idea been considered in the statistical literature at all?

Answer 1

Short answer: From what I know, no.

You need to consider two different aspects of a model: the parameter space and the prediction space. Model complexity only concerns the model's prediction space. As far as I know, there is no measure of model complexity that also takes the parameter space into account.

Furthermore, I think your premise seems to be wrong. While it is definitely true, that for larger models (i.e., bigger parameter space) it is more difficult to obtain the global minima, I dont now of any real case where the added complexity led to lower likelihoods. Although I must admit that I also had huge model that failed to obtain the known best estimates (e.g., when trying to fit a multinomial processing tree model in which all participants have their own model but are fitted simultaneously in one fit). But these are rare and avoidable cases.

So, more complex models do in general not lead to worse fits and etsimates and it is difficult why the parameter space should be incorporated into complexity measures. Note also that different optimization algorithms exist. For very complex models you could use something like evolutionary optimization or simulated annealing which may fair better than traditional Simplex or Newton based algorithms.

Furthermore, if more complex models should lead to worse fit it would be accounted for by some measures of complexity such as normalized maximum likelihood as it would use the same fitting algorithm for assessing the complexity as for fitting itself (and hence the complexity measure would only reflect the reduced complexity due to problems in fitting). The same is true for other measures such as model mimicry or cross-validation.

However, on a more philosophical level you might bring up an interesting point on the price of differently sized parameter spaces. Shouldn't we take the time the fitting procedure takes also into account when assessing models' performance? For example, in a recent paper we had one specification of a signal detection model that we were unable to fit using ordinary fortran double data types due to regular buffer underfolws. We now could redo this analyses with arbitrary precission arithmetics (e.g., with rmpfr or mpc), but fitting would take us ages (fitting with fortran already costed us a few month and we just threw the results away). Hence we refrain from doing so. This problem of model complexity should also be considered formally, but I have never heard of a way of doing so.