I want to make a logistic regression with N independent variables, via cross validation (K splits). I want to use the resulting betas in order to put it in a formula to predict future cases (no yet in my data).

Provided K splits, I would get K sets of resulting $?\beta$s, so... the 'final' betas are the average of each $\beta_i$? Because it is clear the average error should be obtained from the average of results, but it is not so clear the average $\beta$s would correspond to the average error.


I think there might be some confusion here...

Cross-validation lets you choose from a family of models by estimating the prediction error for each one of them. One then takes the model whose estimated error is minimal.

Say you have a family of models $f_\alpha$ indexed by a hyperparameter $\alpha$. Cross-validation provides a function $CV(\hat{f},\alpha)$, where $\hat{f}$ denotes the models trained with CV, for fixed $\alpha$. You then pick $\hat{\alpha} \in \operatorname{argmin} CV(\hat{f},\alpha)$ and retrain the model $f_\alpha$ with all the data. This gives you $\hat{f}_\alpha$ with some specific set of parameters.

In logistic regression these parameters are your betas, and the hyperparameter $\alpha$ is the regularization coefficient for standard Tykhonov regularization, and it is typically $C$ or $\lambda$, ($C=1 / \lambda$).

For a good explanation of model selection see Hastie, Tibshirani, Friedman "The Elements of statistical learning", Chapter 7.

  • 2
    $\begingroup$ I kinda disagree with your definition of CV. CV aims at assessing how the results of a model will generalize to an independent dataset. The same statistical analysis (same model, same (hyper)parameters) is applied sequentially to different subsets of a dataset (with numerous variations on how-to). You seem to be saying CV is applied across a series of models. Apologies in case I misunderstood your reply, $\endgroup$ – tagoma Dec 18 '17 at 11:35
  • 1
    $\begingroup$ Yes, I meant what you say. By estimating how different models will generalise, one can choose among them. CV is performed for each of them in turn. For the case of logistic regression see e.g. what sklearn does $\endgroup$ – Miguel Dec 18 '17 at 11:40
  • $\begingroup$ Thanks a lot. I found the procedure I looked for here mingchen0919.github.io/learning-apache-spark/… $\endgroup$ – Hernan Perez Cortes Dec 18 '17 at 11:55
  • $\begingroup$ That link details the procedure I described (not that I'm claiming any merit for CV!). They use two hyperparameters though: a penalty coefficient and the $p$ for the $p$-norm. $\endgroup$ – Miguel Dec 18 '17 at 12:01
  • $\begingroup$ Also, if you don't accept my answer, you could post your comment as one, with some explanation as to what is being described in that site and accept it, for future users. It seems like CV for logistic regression is a common question here. $\endgroup$ – Miguel Dec 18 '17 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.