1
$\begingroup$

Suppose I am estimating a parameter using maximum likelihood using numerical methods. I go through many steps until eventually I find a value that leads to the maximum. I then want to do inference on this value - for example I want to do a hypothesis test that it's not actually zero. This is just classic theory, carry out the motions and move on.

Now suppose I do one step of forward stepwise regression. I select the best predictor based on how much the sum of squares is reduced and compare it to a Chi-Squared (1 df) distribution. There is a big problem here - I cherry picked the best one so the distribution isn't like a Chi-Squared at all.

Why is the second case selective inference but the first is not? In both cases I tried many different possibilities and picked the best one. Shouldn't the maximum likelihood estimate also be adjusted for cherry picking when doing inference?

Source: http://www.pnas.org/content/112/25/7629.short

$\endgroup$
2
$\begingroup$

The difference is that when you're calculating the MLE you already know which feature $x$ you want find the estimated parameter, $\hat{\beta}$, for. However, when you're doing forward stepwise regression, you're looking to find the set of features that are most important (which I know I haven't defined formally).

In short, finding the MLE is a parameter estimation approach when you know what features you have in your model, forward stagewise is a feature selection approach. Forward stagewise is a good example of a very greedy method.

$\endgroup$
1
$\begingroup$

In the first case (maximum likelihood estimation), you take into account the uncertainty around the estimated parameter value in the inference you make.

In the second case (model selection), if you then naively use the selected model, as if it were the model you intended all along, you are ignoring the uncertainty around the model.

A rough equivalent to naive forward model selection would be, if in the first case would be if you picked the parameter value identified by maximum likelihood for some parameter and then decided that you had always known that this is the exact true value of that parameter (i.e. no uncertainty around it). E.g. you want to estimate the mean $\mu$ of a normal distribution and there is also the variance parameter $\sigma^2$. So, you estimate these parameters, then declare that you have always known that $\sigma^2=\hat{\sigma}^2$ and then you re-estimate $\hat{\mu}$ for a fixed known $\sigma^2=\hat{\sigma}^2$.

An approximate equivalent to maximum likelihood estimation within a single model when it comes to multiple models is for example AIC model averaging across the candidate models (or at least those that come close to minimizing the AIC). More on this topic can be found e.g. in the book by Burnham and Anderson ("Model Selection and Inference: A Practical Information-Theoretic Approach").

$\endgroup$
1
  • $\begingroup$ I've been looking for JUST that kind of book! Thanks so much for that reference. $\endgroup$
    – badmax
    Dec 13 '17 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.