Profile Likelihood: why optimize all other parameters while tracing a profile for a partitcular one?

Question

Profile likelihood is sometimes used to get estimates for the confidence limits of parameters from an n-dimension parameter fit to a model. It can be used for example instead of Monte Carlo estimation. I don't understand the intuition of the algorithm itself. See section 4.4 of the paper "Parameter uncertainty in biochemical models described by ordinary differential equations", by Vanlier et al. (2013), Math Biosci.

Assume a model has been optimized and a minimum located. According to the algorithm, a parameter is selected and slowly changed. After each change, all the other unchanged parameters are re-optimized at the new value of the changed parameter. The chi-square at this new optimized point is recorded. This is repeated until a chi-square profile is obtained. This process can be applied to each parameter in turn and the change in chi-square can be used to define a confidence region for the particular parameter.

I'd like to understand the intuition as to why the other parameters must be optimized as we profile the selected parameter? Why for example couldn't we just change the parameter (leave the other parameters fixed) and observe how the chi-square changes away from the optimum? Wouldn't that tell us how the curvature changed and therefore give us information on how confident we are in the parameter?

Answer 1

You can think of the profile confidence interval as an inversion of the likelihood ratio test; you are comparing a model in which your parameter of interest is allowed to vary against a set of nested models in which the parameter of interest is fixed. Your confidence interval is the set of values for which the parameter is fixed and the likelihood ratio fails to reject. In the likelihood ratio test, you compare the likelihood at the MLE for each model. Therefore, you must optimize all free parameters of both the full model and the nested model. If you change one parameter, then it is very likely that the values of the other parameters at the optimum will change, so you can't just "recycle" them from the full model.

To further explore this, consider something like a regression model in which two covariates are highly collinear. From what we learned from studying linear regression, we should know that confidence interval for either individual predictor should be very wide, since it is hard to differentiate between the effect of the first and second variable, given their collinearity. Now, if we tried to make a profile confidence interval for predictor one, but fixed predictor two, we would (mistakenly) get a very narrow confidence interval; this would be the equivalent to fixing the second coefficient, subtracting out it's effect and then computing the confidence interval for the first coefficient without including the second covariate in your model.

In a nutshell, you need to allow the other parameters to vary to account for the fact that the uncertainty in your parameter of interest may be tied to the uncertainty in other parameters in your model.