1
$\begingroup$

I'm a bit confused. I am doing an analysis where there are about 70 observations of my dependent variable. I'm planning to do a multiple linear regression or multivariate logistic regression to see whether there is an association between my variable of interest (X) and dependent variable, whilst adjusting for other potential confounders.

In most similar studies in the literature related to my project I've read, people don't bother with evaluating model performance (e.g. measures of calibration, discrimination, cross validation, bootstrapping).

My research question is quite exploratory and is moreso focusing on explaining my data rather than trying to build a robust prediction model.

My thinking is that since I'm dealing with a small sample size and my question is more explanatory than predictive, I shouldn't bother with testing my regression model's performance and doing re-sampling, etc.

Is that right? Any help would be much appreciated.

Edited to add: the dependent variable is a total score on a questionnaire. I am particularly interested in investigating a possible relationship between a novel clinical measure and score on this questionnaire. Im planning to do a multiple linear regression to control for known cofounders. However, I am limited to 5-7 variables due to the 10:1/15:1 rule. Whilst there’a probably about 15-20 other variables I’d like to include in the regression in an ideal world I realise that’s not feasible because I only have 70 participants. Im hoping to do a multiple factor analysis to at least look at the relationships between all the key study variables though.

$\endgroup$
7
  • 5
    $\begingroup$ Competent analysts always do bother--but most journal editors don't see much value in devoting space to reporting those details. It's a little like removing the scaffolding after finishing erecting a building: although the scaffolding is gone, it was essential for constructing a building that will not fall down. By analogy, the question facing you is whether you need to build a home or just a lean-to for the night. But even in the latter case you would want to take measures to make it a little waterproof and robust enough to last the day. $\endgroup$
    – whuber
    Mar 3, 2022 at 16:49
  • 2
    $\begingroup$ Please edit your question to say more about the nature of your dependent variable and how many predictors/confounders you might have. Depending on such details, 70 observations might be more than enough or not even close to enough for an "exploratory" study. The small extra cost of performance evaluation is more than outweighed by the benefits. You don't want your "exploration" to lead you into an alligator-filled swamp, unless you are hunting alligators. $\endgroup$
    – EdM
    Mar 3, 2022 at 17:24
  • $\begingroup$ I would say that exploratory analyses require more playing with the data than explanatory or predictive, not less. $\endgroup$ Mar 3, 2022 at 21:18
  • $\begingroup$ @gung That makes sense--but my experience is the opposite. When I am conducting explanatory or predictive analyses in real applications, often a lot is riding on the result. That forces me to work hard to check that the calculations are correct; that I haven't overlooked something, or made implicit assumptions, or overfit, etc., as well as checking for evidence of important violations of the underlying assumptions in the analysis; and to do this for all the models I might have considered along the way. That's a lot of work and almost always is greater than the initial exploratory effort. $\endgroup$
    – whuber
    Mar 4, 2022 at 14:50
  • $\begingroup$ @whuber, that's what I meant by "more playing with the data". $\endgroup$ Mar 4, 2022 at 18:55

1 Answer 1

0
$\begingroup$

There are several way to handle the problem of too many predictors of interest given the size of a data set. Frank Harrell deals with that extensively in his course notes, book, and other publications.

One solution is to combine each set of related predictors into a combined score before looking at the outcomes. This page emphasizes that approach among others.

Another is to keep your main predictor of interest unpenalized while penalizing the predictors that you're trying to account for. This paper illustrates that approach. Ridge regression will generally be superior to LASSO, as it keeps all the predictors in the model without the arbitrary predictor selection in LASSO.

Those methods for handling the too-many-predictors problem benefit from evaluating the performance of the modeling approach, for example via an extension of the optimism bootstrap implemented in the validate() function of Harrell's rms package. That would directly address your supervisor's fears about instability of the modeling process. With a penalty chosen by cross-validation, the effective number of predictors (with penalization taken into account) will tend to be reduced to a number consistent with the size of the data set, minimizing modeling instability in a way that you can document.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.