Are you always supposed to evaluate the performance of regression models? I'm a bit confused. I am doing an analysis where there are about 70 observations of my dependent variable. I'm planning to do a multiple linear regression or multivariate logistic regression to see whether there is an association between my variable of interest (X) and dependent variable, whilst adjusting for other potential confounders.
In most similar studies in the literature related to my project I've read, people don't bother with evaluating model performance (e.g. measures of calibration, discrimination, cross validation, bootstrapping).
My research question is quite exploratory and is moreso focusing on explaining my data rather than trying to build a robust prediction model.
My thinking is that since I'm dealing with a small sample size and my question is more explanatory than predictive, I shouldn't bother with testing my regression model's performance and doing re-sampling, etc.
Is that right? Any help would be much appreciated.
Edited to add: the dependent variable is a total score on a questionnaire. I am particularly interested in investigating a possible relationship between a novel clinical measure and score on this questionnaire. Im planning to do a multiple linear regression to control for known cofounders. However, I am limited to 5-7 variables due to the 10:1/15:1 rule. Whilst there’a probably about 15-20 other variables I’d like to include in the regression in an ideal world I realise that’s not feasible because I only have 70 participants. Im hoping to do a multiple factor analysis to at least look at the relationships between all the key study variables though.
 A: There are several way to handle the problem of too many predictors of interest given the size of a data set. Frank Harrell deals with that extensively in his course notes, book, and other publications.
One solution is to combine each set of related predictors into a combined score before looking at the outcomes. This page emphasizes that approach among others.
Another is to keep your main predictor of interest unpenalized while penalizing the predictors that you're trying to account for. This paper illustrates that approach. Ridge regression will generally be superior to LASSO, as it keeps all the predictors in the model without the arbitrary predictor selection in LASSO.
Those methods for handling the too-many-predictors problem benefit from evaluating the performance of the modeling approach, for example via an extension of the optimism bootstrap implemented in the validate() function of Harrell's rms package. That would directly address your supervisor's fears about instability of the modeling process. With a penalty chosen by cross-validation, the effective number of predictors (with penalization taken into account) will tend to be reduced to a number consistent with the size of the data set, minimizing modeling instability in a way that you can document.
