Timeline for Can overfitting and underfitting occur simultaneously?
Current License: CC BY-SA 4.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 27, 2020 at 13:15 | comment | added | gung - Reinstate Monica | The question states, "$Z$ are some regressors (perhaps partly overlapping with $X$ but not necessarily equal to $X$)". I interpreted that to be the dataset. The distinction I'm making b/t the true DGP & your dataset, is that there can be relevant variables in the DGP that aren't in your dataset. There's nothing that the model / modeler can do when the required info just isn't there. As a result, it doesn't sound right to me to say a model is underfit if it wasn't given all the relevant information. But, it does seem reasonable to say it's underfit if it didn't extract the info it was given. | |
| Nov 27, 2020 at 11:30 | comment | added | Stephan Kolassa | @gung-ReinstateMonica: $Z$ is the design matrix we use in modeling, as per the original question. (So I don't expect a model to look at possible transformations, like squaring a column.) I don't think there is all that much difference between "underfitting wrt the truth" vs. "underfitting wrt to the information in the dataset", because if something is in the dataset, it is a fortiori also in the truth, and if something is in the truth but not in the dataset, it can't have been all that truthy to start with. (Details in the argument to be filled in by the reader...) | |
| Nov 26, 2020 at 17:41 | comment | added | gung - Reinstate Monica | But I think of underfitting as relative to the information available in the dataset (consider this recent thread where subset selection typically leads to worse models). | |
| Nov 26, 2020 at 17:41 | comment | added | gung - Reinstate Monica | Hmmm, perhaps I misinterpreted your answer. What is $Z$? I interpreted it as the dataset. So when the model is fit to the dataset, "$x_2$ is unknown to $g(Z)$, so we will have underfitting". To me, that reads as, when the data available don't have all the relevant variables (which they never will in practice), the model will be underfit. This sounds like conceiving of underfitting as relative to the truth, which is certainly a justifiable stance. | |
| Nov 26, 2020 at 16:35 | comment | added | Stephan Kolassa | @gung-ReinstateMonica: I think I'm still not understanding. When I think about underfitting, I indeed assume the information to be both in the DGP and in the data (I am still a little confused about the difference). We may actually be on the same page here. And I wouldn't blame a model for underfitting if it doesn't see data in the first place, but the modeler, who provided the data to the model. And actually I wouldn't "blame" anyone at all, since per Box, underfitting models can be far more useful than correct ones... | |
| Nov 26, 2020 at 12:45 | comment | added | gung - Reinstate Monica | @StephanKolassa, certainly it's in the DGP. As I understand the way you are using "underfit", the information is in the DGP, but not in the dataset you have access to. The way I use the term is where the information is in both the DGP & the data, but the model misses it. I would call such a model underfit, but I'd hardly blame a model as having a flaw when it never had access to the requite information in the first place--that seems unfair to the model. | |
| Nov 26, 2020 at 10:29 | comment | added | Stephan Kolassa | @gung-ReinstateMonica: I think I'm not completely understanding the distinction you make. What would it mean for information to be in the dataset, but not in the DGP? In your example with a curvilinear relationship of $x_2$, I would say that the information about this relationship is indeed in the DGP. | |
| Nov 25, 2020 at 16:59 | comment | added | gung - Reinstate Monica | I think this is a great answer (+1), & perhaps I'm being too pedantic, but it seems to me that working from the definition used, essentially all models have to underfit by definition. I recognize the application of Box's famous dictum here, but I think it may be useful to distinguish between "wrong model" in that sense & "underfit model" in the common setting. | |
| Nov 25, 2020 at 16:55 | comment | added | gung - Reinstate Monica | There's something of a philosophical issue regarding what "underfitting" means. Here, you seem to take it as meaning that there is information in the data generating process that is not extracted by the model (because you don't have it in your dataset). Another, stricter, possibility is that it means information in your dataset that isn't extracted by your model. Consider a case where $x_2$ is in your dataset, but the relationship is curvilinear & no polynomial terms are used for $x_2$ (that would be somewhat inexplicable, given the overzealousness WRT $x_1$, but whatever...). | |
| Sep 21, 2020 at 14:19 | comment | added | Richard Hardy | Thank you, that makes sense. I am a conditional believer, conditioning on the field of application. In economics and finance, certainly yes. In gene expression data, perhaps no. | |
| Sep 21, 2020 at 13:47 | comment | added | Stephan Kolassa | I'm a firm believer in so-called "tapering effect sizes", i.e., there are always influences with weaker and weaker effects, and we can't model them all (e.g., because of the bias-variance tradeoff per Sextus). As such, we always underfit. Therefore, I honestly don't really think changing the definitions to make the two concepts mutually exclusive is very useful. (Also, I don't quite see how it could be done and still leave us with recognizable concepts.) | |
| Sep 21, 2020 at 10:55 | comment | added | Richard Hardy | The example illustrates the general formulas $f(X)$ and $g(Z)$ nicely. I guess, my problem is that overfitting and underfitting are often loosely defined. I am trying to see whether I should alter the definitions to make them mutually exclusive or proceed as is, yielding the surprising idea of simultaneous over- and underfitting. | |
| Sep 21, 2020 at 10:51 | history | answered | Stephan Kolassa | CC BY-SA 4.0 |