4
$\begingroup$

I have fitted respectively a zero-knot, a one-knot and a two-knot linear spline to my data, and I need some index of model performance for model selection. The crucial point is that the splines are fitted with robust linear regressions, specifically with Huber estimations, which makes the usual estimator of prediction error like AIC invalid. So my problem is:

  1. What criterion should I use to perform model selection on my zero, one and two-knot splines? Can I use SSE?
  2. I may also need to compare a model using Huber estimation and a model using Tukey's bisquare estimation. What criterion should I use?
$\endgroup$
18
  • 1
    $\begingroup$ As far as I'm aware of, model selection here is an open issue. I have been studying this for some time and I'll write soon a brief answer with some possible pointers. $\endgroup$
    – utobi
    Commented Feb 18, 2023 at 8:00
  • $\begingroup$ @utobi Thank you very much. $\endgroup$
    – zyy
    Commented Feb 18, 2023 at 8:01
  • 2
    $\begingroup$ You could use cross-validation or bootstrap, however you need to make sure that the loss functions are not affected by outliers itself. You may google for robust cross-validation and robust bootstrap. $\endgroup$ Commented Feb 18, 2023 at 10:51
  • 1
    $\begingroup$ SSE is strongly affected by outliers. $\endgroup$ Commented Feb 19, 2023 at 11:10
  • 1
    $\begingroup$ There is no single "correct" criterion. As others said, RMSS will be affected by outliers, for that reason, I suggested MAE as the "standard" alternative. $\endgroup$
    – usεr11852
    Commented Feb 28, 2023 at 23:08

1 Answer 1

6
$\begingroup$

Robust estimation via Huber's loss function can be seen as a kind of M-estimation, that is, an estimator derived as the solution of an estimating equation

$$ \Psi(\theta;y) = 0. $$

The function $\Psi$ is typically a sum of $n$ contributions and is in these contributions where Huber's loss function enters in. Other possible losses are possible, you mentioned Tukey's bisquare, but there are many others.

Related names for M-estimation theory are the generalized method of moments (GMM) and the generalized estimating equation (GEE). The point is that robust inference can be seen as a kind of GEE or GMM thus it inherits all their issues and open problems.

In particular, while parameter inference in GEE and GMM is widely developed, mostly dominated by Wald-type inferential procedures, the estimation of a model, or model selection is problematic due to the lack of a parametric model.

The problem may arise in two different situations.

(1) Loss-based estimation. For the sake of robustness, we are not willing to assume a fully parametric model and we instead chose to estimate the parameters by minimising a suitably defined loss function. Typically this loss function doesn't come from a parametric model, thus the likelihood function is not available and all the likelihood-based machinery (likelihood ratio tests, AIC, BIC, etc.) cannot be immediately applied. You could think of getting a surrogate of likelihood by exponentiating the loss but this won't work since the surrogate likelihood is driven by an unknown proportionality constant. Thus AIC and BIC computed on this likelihood would be driven by some arbitrary constant so they are useless.

(2) Estimating-equation estimation. Typically in GEE and GMM but also when you consider joint estimation of regression parameters and the scale parameter using Huber's and other loss functions, there is no joint loss function anymore. Thus, if you are still tempted by the exponentiated loss, now that option is definitely out of reach!

To overcome the issue in the case in which a loss function is available, some scholars have tried to build surrogates of the likelihood function and then compute a BIC or an AIC on these surrogates. As I wrote above, simply exponentiation the loss doesn't work but you can do something more sophisticated than exponentiation. The work by Pan (2001) Akaike's information criterion in generalized estimating equations goes in this direction. Pan gets a surrogate likelihood with theoretical guarantees by means of the quasi-likelihood (McCullagh and Nelder, 1994 Generalized Linear Models, 2nd edition, p. 325) and builds an AIC from this likelihood.

I'm not aware of an R implementation of this but a stata implementation can be found in Qui (2007) QIC program and model selection in GEE analyses, The Stata Journal 7, Number. 2, pp. 209-220.

Surely there are others, but another related concept is the quadratic inference function advanced by B.G. Lindsay and his coauthors; see, e.g., Qu, Lindsay, Li (2000) Improving Generalised Estimating Equations Using Quadratic Inference Functions, Biometrika 87, pp. 823-836.

Some of these issues as well as other alternative approaches are discussed in the book by Hertier et al. (2009) Robust Methods in Biostatistics, Wiley, ISBN: 978-0-470-02726-4; another useful and more up-to-date reference on robust inference is the book by Maronna et al. (2018) Robust Statistics: Theory and Methods (with R), 2nd Edition, Wiley, ISBN: 978-1-119-21466-3.

$\endgroup$
6
  • 1
    $\begingroup$ Thank you. It really help me a lot. I will try google some R implementation of the methods you mentioned. $\endgroup$
    – zyy
    Commented Mar 2, 2023 at 8:55
  • $\begingroup$ +1. Assuming one is happy to use LOOCV, under a Huber loss, wouldn't that be equivalent (or at least qualitatively equivalent) to AIC for such a robust model? Are you suggesting using $\Psi$ as a surrogate for the quasi-likelihood? I am asking as we need first and second derivatives of $Q$ but many of the robust losses have discontinuities. $\endgroup$
    – usεr11852
    Commented Mar 2, 2023 at 12:40
  • $\begingroup$ @usεr11852 (1) Assuming one is happy to use LOOCV, under a Huber loss, wouldn't that be equivalent (or at least qualitatively equivalent) to AIC for such a robust model?: I believe the answer is: roughly yes. The problem though with LOOCV I guess is that you are may not get finite estimates at each iteration. But again, Huber loss is available as long as you estimate only location or location and scale separately; if you estimate them jointly no loss is available. (2)_Are you suggesting using $\Psi$ as a surrogate for the quasi-likelihood?_ $\endgroup$
    – utobi
    Commented Mar 2, 2023 at 19:49
  • $\begingroup$ No, as I wrote $\Psi$ is an estimating equation, i.e. a sort of "score function = 0" equation in the case of a fully specified model. $\endgroup$
    – utobi
    Commented Mar 2, 2023 at 19:51
  • 1
    $\begingroup$ Thank you for the clarifications. $\endgroup$
    – usεr11852
    Commented Mar 2, 2023 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.