0
$\begingroup$

I have developed a regression model, and found some issues with the overall process. As I do not have depth knowledge on regression, I need some expert advice.

I have n observations (4700+) from eight cases (C) for few years, and p candidate predictors (p=18). Assuming that the true model is a linear combination of m variables (9 i.e. A-I, see below from p and 6 as dummy variables) among the p candidate predictors. Is there an upper bound to m to identify a model? I heard that if m is too large compared to C, there are chances of over-fitting. I have few questions:

  1. How strict the rule is that the m should be less than cases(C)? Or the observations(n)? Is dummy variable a considered under this rule, too? Can I break this rule? If so, can anyone please suggest any reference?

  2. I did not use stepwise regression, rather I applied my knowledge to keep useful variables that are most relevant, observing the significance level and finally checking the VIF. Is it necessary to use stepwise regression.

  3. I am planning to validate this (Or, the updated one) by using leave-one-out cross-validation. If I leave one case, the new regression models may have new sets of m variables where all the variables may not be the same as the final model below. Is that right? Or, Do I have to keep the same variables in each leave out model that I have in the final model in order to estimate the MSE for validation.

Thanks for your help.

enter image description here

Kind Regards,

$\endgroup$
  • $\begingroup$ It sounds like you have repeated measures on the same cases, if so, regression is not appropriate as it assumes independent data. You can look into multilevel models. It looks like you use R so that would be lme4 or nlme packages. Both have been discussed here a lot. $\endgroup$ – Peter Flom - Reinstate Monica Jan 8 '14 at 10:48
  • $\begingroup$ Thank you very much for your prompt help. What if I use only one year's observation for eight cases with no repetition. How these problems will be approached then. I assume the modelling will not be within multi-level domain? $\endgroup$ – user37022 Jan 8 '14 at 10:55
  • $\begingroup$ If you have 8 cases then you should have at most 1 independent variable. But you have much more data and you certainly shouldn't give that data up. $\endgroup$ – Peter Flom - Reinstate Monica Jan 8 '14 at 10:58
3
$\begingroup$

How strict the rule is that the m should be less than cases(C)? Or the observations(n)? Is dummy variable a considered under this rule, too? Can I break this rule? If so, can anyone please suggest any reference?

  • Observations $n$ vs. cases $C$: a very simple model of this could be a "hierarchical" data structure with is a variation $\sigma^2_C$ between cases, and an independent variation $\sigma^2_n$ between the (repeated) observations for each case. In this scenario, what matters is which of the two contributes the major variation: if $\sigma^2_C \gg \sigma^2_n$, you have an effective sample size of $C$, if $\sigma^2_C \ll \sigma^2_n$ you are fortunate and the effective sample size is almost $n$. I usually encounter situations with effective sample size $\approx C$, though occasionally, the $n$ repeated measurements I have add up to a bit of information as well.
    In any case, $C$ is a conservative estimate of the sample size.

  • As Peter Flom explained, there are sophisticated models for such situations. However, you may get away with more simple modeling (such as lm) if you can reasonably apply the simple variance model I outlined above and judge that one of the two sources of variance dominates the other. If $\sigma^2_C$ dominates, weighting the rows so that each case in the end gets the same weight may be enough. If you are lucky and $\sigma^2_n$ dominates, then the linear model should be OK.
    Even though lm may not be the most appropriate model (i.e. you may get better performance with more sophisticated modeling), you can still validate the model you got and find out how good that model is.

  • Violating the rule: Such rules are rules of thumb. In practice, you may have no choice but to violate it. The risk you run if you have too few cases is that your model overfits the data and does not generalize well to new cases. There are several things you can do to measure whether you are fortunate and there are no bad consequences or whether you do have a problem here. One symptom of overfitting is that the model and its predictions become unstable with respect to slight changes of the training data (e.g. exchange one case against another).
    This you can easily measure during a resampling validation scheme by comparing the different surrogate models you calculate.

    Here's a paper where we discuss this: Beleites, C. and Salzer, R. Assessing and improving the stability of chemometric models in small sample size situations, Anal Bioanal Chem, 2008, 390, 1261-1271. DOI: 10.1007/s00216-007-1818-6

I did not use stepwise regression, rather I applied my knowledge to keep useful variables that are most relevant, observing the significance level and finally checking the VIF. Is it necessary to use stepwise regression.

IMHO: a very wise decision.
IMHO stepwise regression is rarely a wise direction to take, whereas applying your knowledge about problem and data can help tremendously. It is one of the most valuables types of information you have about the problem.
In any case, stepwise regression needs tremendous sample sizes, otherwise you have a high risk of overfitting. As you already know that your number of cases is far below the recommended sample size, I'd keep my fingers away from optimization techniques (such as selecting an optimal set of variables).

I am planning to validate this (Or, the updated one) by using leave-one-out cross-validation. If I leave one case, the new regression models may have new sets of m variables where all the variables may not be the same as the final model below. Is that right? Or, Do I have to keep the same variables in each leave out model that I have in the final model in order to estimate the MSE for validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.