8
$\begingroup$

I'm having a hard time getting on the same page as my supervisor when it comes to validating my model. I have analyzed the residues (observed against the fitted values) and I used this as an argument to discuss the results obtained by my model, however my supervisor insists that the only way to validate a model is to make a random subset of my data, generate the model with 70% of it and then apply the model on the remaining 30%.

The thing is, my response variable is zero inflated (85% of it, to be more rpecise) and i prefer not to create a subset as it is already very difficult to converge to a result.

So, my question is: what are the possible (and scientifically acceptable) ways to validate a model? Is subsetting data the only way? If possible, reference your questions with articles/books so I can use it as an argument when presenting my alternatives.

$\endgroup$
  • $\begingroup$ If you've picked the model based on all the data, though, that still doesn't count... $\endgroup$ – Aaron - Reinstate Monica Dec 9 '18 at 2:22
  • $\begingroup$ You mean I should choose my model based on a subset of my data? How can properly analyze the distribution of my data if I don't use all of it? $\endgroup$ – Eric Lino Dec 10 '18 at 21:29
  • $\begingroup$ Yes, that's exactly what I mean -- if you want to have data to truly validate your model, that data needs to be held out when making the choice of analysis. This is what Wikipedia calls the training and validation sets. If you use your validation set to fit hyper-parameters, you'd even need a third set (the test set) to test your final model against. $\endgroup$ – Aaron - Reinstate Monica Dec 11 '18 at 20:36
  • $\begingroup$ Not that I'm advocating for that; in your case, you are perhaps better off using more traditional diagnostic and model selection methods, which is of course what you're asking for (and alas, I don't have the time to answer properly). $\endgroup$ – Aaron - Reinstate Monica Dec 11 '18 at 20:38
  • 2
    $\begingroup$ The answer you're looking for is probably in section 5.3 of Frank Harrell's Regression Modeling Strategies. $\endgroup$ – Aaron - Reinstate Monica Dec 11 '18 at 20:44
6
$\begingroup$

To start, I would suggest that it is usually good to be wary of statements that there is only one way to do something. Splitting an obtained sample into a "training" and a "testing" data set is a common approach in many machine learning/data science applications. Oftentimes, these modeling approaches are less interested in hypothesis testing about an underlying data generation process, which is to say they tend to be somewhat atheoretical. In fact, mostly these sorts of training/testing splits just want to see if the model is over-fitting in terms of predictive performance. Of course, it is also possible to use a training/testing approach to see if a given model replicates in terms of which parameters are "significant," or to see if the parameter estimates fall within expected ranges in both instances.

In theory, validating or invalidating models is what science, writ large, is supposed to be doing. Independent researchers, separately examining, generating, and testing hypotheses that support or refute arguments about a theory for why or under what circumstances an observable phenomenon occurs - that is the scientific enterprise in a nut shell (or at least in one overly long sentence). So to answer your question, to me, even training/testing splits are not "validating" a model. That is something that takes the weight of years of evidence amassed from multiple independent researchers studying the same set of phenomena. Though, I will grant that this take may be something of a difference in semantics about what I view model validation to mean versus what the term validation as come to mean in applied settings... but to get back to the root of your question more directly.

Depending on your data and modeling approach, it may not always be appropriate from a statistical standpoint to split your sample into training and testing sets. For instance, small samples may be particularly difficult to apply this approach to. Additionally, some distributions may have certain properties making them difficult to model even with relatively large samples. Your zero-inflated case is likely fits this description. If the goal is to get at an approximation of the "truth" about a set of relations or underlying processes thought to account for some phenomenon, you will not be well-served by knowingly taking an under-powered approach to testing a given hypothesis. So perhaps the first step is to perform a power analysis to see if you would even be likely to replicate the finding of interest in your subsetted data. If it is not appropriately powered, that you could be an argument against the testing/training split.

Another option is to specify several models to see if they "better" explain the observed data. The goal here would be to identify the best model among a set of reasonable alternatives. This is a relative, not an absolute, argument you'd be making about your model in which you are admitting that there may be other models that could be posited to explain your data, but your model is the best of the tested set of alternatives (at least you hope so). All models in the set, including your hypothesized model, should be theoretically grounded; otherwise you run the risk of setting up a bunch of statistical straw men.

There are also Bayes Factors in which you can compute the weight of evidence your model provides, given your data, for a specific hypothesis relative to alternative scenarios.

This is far from an exhaustive list of options, but I hope it helps. I'll step down from the soapbox now. Just remember that every model in every published study about human behavior is incorrect. There are almost always relevant omitted variables, unmodeled interactions, imperfectly sampled populations, and just plain old sampling error at play obfuscating the underlying truth.

$\endgroup$
  • $\begingroup$ I appreciate all the time you spent writing such a in-depth answer, Matt. However, I feel that although it helps me in a conceptual level, it lacks some reference that I will very much need to discuss this approach with my supervisor. Would you happen to have any papers/books on the odds of subsetting data? If not possible, would you recommend a R package on which i can perform this power analysis you spoke of? $\endgroup$ – Eric Lino Dec 10 '18 at 3:33
  • $\begingroup$ For R packages and power it depends on your model (pwr, simsem, etc). There is not a single answer. Also in terms of the odds of subsetting your data I think that is just another way of asking about power if I understand you correctly. If you are gravitating toward the power bit I would recommend concentrating on your weakest effect and seeing what the minimal sample size would have to be to replicate it - a sort of worst case scenario. $\endgroup$ – Matt Barstead Dec 10 '18 at 11:37
  • $\begingroup$ I see. Well, I'm using the glmmadmb package, developed by Ben Bolker & others. My response variable is zero inflated (number of people with a specific rare disease) and my independent variables include normal, non-normal and zero inflated distributions. Since I'm dealing with a time series, I used "year" as a grouping factor and it seemed like a good idea to explore the ZIGLMM family of models. Does this information help you in helping me? $\endgroup$ – Eric Lino Dec 10 '18 at 21:27
  • 2
    $\begingroup$ You may want to check out the simR package. To my knowledge it is the most flexible existing package for power analyses with linear and generalized linear models. Green, P., & MacLeod, C. J. (2016). SIMR: An R package for power analysis of generalized linear models by simulation. Methods in Ecology and Evolution. $\endgroup$ – Matt Barstead Dec 10 '18 at 21:41
10
$\begingroup$

Data splitting is in general a very non-competitive way to do internal validation. That's because of serious volatility - different 'final' model and different 'validation' upon re-splitting, and because the mean squared error of the estimate (of things like mean absolute prediction error and $R^2$) is higher than a good resampling procedure such as the bootstrap. I go into this in detail in my Regression Modeling Strategies book and course notes. Resampling has an additional major advantage: exposing volatility in feature selection.

$\endgroup$
  • $\begingroup$ still believe the OP's main motivation is to know if his ZIP model is adequate = residual check, not model / feature selection or predictive performance, but maybe he can clarify himself $\endgroup$ – Florian Hartig Dec 12 '18 at 16:04
  • $\begingroup$ Yes, @FlorianHartig is correct! However bootstrapping interests me (if not for this study, for knowledge gathering) and I'll defnitely look your website for future reference. Thank you very much for the input. $\endgroup$ – Eric Lino Dec 12 '18 at 18:47
6
$\begingroup$

I think the answers here diverge because the question is somewhat unclear, foremost: what do you mean by "validation"?

A 70/30 split (or a cross-validation for that matter) is usually performed to assess the predictive performance of a model or an entire analysis chain (possibly including model selection). Such a validation is particularly important if you are comparing different modelling options in terms of their predictive performance.

It's another case entirely if you don't want to select models, and are also not interested in predictive performance as such, but you are interested in inference (regression estimates / p-values), and want to validate if your model / error assumptions of the GLMM are adequate. In this case, it would be possible to predict to the hold-out and compare predictions to observed data, but the by far more common procedure is to do a residual analysis. If you need to prove this to your supervisor: this is basically what every stats textbooks teaches to do right after the liner regression.

See here for how to run a residual analysis for GLMMs (including zero-inflation with glmmTMB, which I would prefer over glmmadmb) with the DHARMa package (disclaimer: I am the maintainer).

$\endgroup$
  • $\begingroup$ Thank you for such a clear, yet insightful answer. My initial case was the second example you provided; I'm not interested in assessing the predictive performance of my model, but only quantify the underlying relations between my response variable and my independent ones. I'm not sure I understood exactly what you mean by "predict to the hold out". Are you referring to the predicted values that are generated on the model object output after you run it? $\endgroup$ – Eric Lino Dec 12 '18 at 18:29
  • $\begingroup$ I mean that you calculate residuals / bias by comparing predictions vs. observations on the hold-out (= validation) data $\endgroup$ – Florian Hartig Dec 14 '18 at 12:28
2
$\begingroup$

The short answer is yes, you need to assess your model's performance on data not used in training.

Modern model building techniques are extremely good at fitting data arbitrarily well and can easily find signal in noise. Thus a model's performance on training data is almost always biased.

It is worth your time to explore the topic of cross validation (even if you are not tuning hyperparameters) to gain a better understanding of why we hold out data, when it works, what assumptions are involved, etc. One of my favorite papers is:

No unbiased estimator of the variance of k-fold cross-validation

$\endgroup$
  • 7
    $\begingroup$ This is not correct in general. The bootstrap badly outperforms data spitting in terms of mean squared errors on quantities such as $R^2$. You do need data not used in training but this can be different data for each resample. $\endgroup$ – Frank Harrell Dec 12 '18 at 14:25
  • $\begingroup$ Are you suggesting to bootstrap on held out data? $\endgroup$ – Chris Dec 13 '18 at 1:19
  • $\begingroup$ Read up on the Efron-Gong optimism bootstrap which is the standard bootstrap model validation method and the one implemented in the R rms package validate and calibrate functions. With this bootstrap there are no one-time decisions about holding out data. As my RMS book and course notes describe in detail, the amount of overfitting is estimated by seeing how much a model developed in a bootstrap sample falls apart when applied to the (overlapping) original full sample. The bootstrap has to repeat all modeling steps afresh for each iteration. $\endgroup$ – Frank Harrell Dec 13 '18 at 12:44
  • $\begingroup$ Interesting. I've looked through your notes and Efron's paper. It feels like for models that are quick to fit we might realize some advantages employing specific versions of the bootstrap. $\endgroup$ – Chris Dec 14 '18 at 1:54
  • $\begingroup$ And the only thing that would make the bootstrap seem to be slow (although it would still be faster than cross-validation, which requires 100 repeats of 10-fold cross-validation to provide stability) is to compare it with single data splitting which provides only an illusion of stability. $\endgroup$ – Frank Harrell Dec 14 '18 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.