Timeline for Statistical inference under model misspecification
Current License: CC BY-SA 3.0
31 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 7, 2019 at 13:36 | answer | added | Christian Hennig | timeline score: 2 | |
| Apr 23, 2017 at 17:58 | history | edited | kjetil b halvorsen♦ |
edited tags
|
|
| Apr 13, 2017 at 12:44 | history | edited | CommunityBot |
replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
|
|
| Mar 15, 2017 at 22:04 | comment | added | user78229 | For what it's worth, this AER article Prediction Policy Problems by Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ziad Obermeyer specifically addresses concerns with inference vs prediction, "We argue that there are also many policy applications where causal inference is not central, or even necessary." (cs.cornell.edu/home/kleinber/aer15-prediction.pdf) | |
| Mar 15, 2017 at 19:42 | comment | added | Richard Hardy | @hejseb, hmm, probably not everyone knows this. People around you may be statistically literate, but there is some selection bias. | |
| Mar 15, 2017 at 19:28 | answer | added | Aksakal | timeline score: 4 | |
| Mar 15, 2017 at 19:05 | comment | added | Alexis | @DJohnson Relevant lit differentiating prediction and explanation:$$\phantom{0}$$ Prediction, explanation and the epistemology of future studies. Futures, 35(10):1027–1040. $$\phantom{0}$$ Hanson, N. R. (1959). On the symmetry between explanation and prediction. The Philosophical Review, 68(3):349–358.$$\phantom{0}$$ Rescher, N. (1958). On prediction and explanation. The British Journal for the Philosophy of Science, 8(32):281–290.$$\phantom{0}$$ Scheffler, I. (1957). Explanation, prediction, and abstraction. The British Journal for the Philosophy of Science, 7(28):293–309. | |
| Mar 15, 2017 at 18:46 | comment | added | hejseb | @RichardHardy, sure, despite being a stats grad student I don't really believe in inference anymore. It's a house of cards so fragile that it's unclear whether it's meaningful at all except in very strict and controlled circumstances. What is funny is that everyone knows this, but no one (well) cares. | |
| Mar 15, 2017 at 18:37 | history | edited | Richard Hardy | CC BY-SA 3.0 |
added 197 characters in body
|
| Mar 15, 2017 at 18:15 | comment | added | Richard Hardy | @hejseb, thanks! This issue is incredibly disturbing when I think about causal inference in economics (except, I guess, for experimental economics)... Looks like we could scrape most of the published results. Painful... | |
| Mar 15, 2017 at 18:07 | comment | added | hejseb | Leeb and Pötscher have studied this extensively. The distribution of the estimator is typically highly non-standard and you are perfectly right that inference is usually highly flawed because of this. This applies to any model selection procedure, be it AIC, OLS post lasso, pretesting etc. There was a paper in the Annals of Statistics in 2013 by Berk etc although which supposedly allows for valid inference. If you want to search further, just google post-selection inference. Hjort and Claeskens 2003 model averaging paper in JASA is a good read too. | |
| Mar 14, 2017 at 19:10 | comment | added | Richard Hardy | @AlecosPapadopoulos, These two sentences are sloppy as I still have not found a good way to express myself (evidently I do not understand the phenomenon perfectly). What I mean is that we are interested in inference as stated at the beginning, not some modification of it. | |
| Mar 14, 2017 at 18:52 | comment | added | Alecos Papadopoulos | You write "This makes the estimator distribution (and thus also inference) conditional on the change in the underlying model, which is due to the observed data. Clearly, the introduction of such conditioning is not satisfactory.". Why the last sentence? | |
| Mar 14, 2017 at 10:48 | comment | added | Richard Hardy | @DeltaIV, no, of course you did not say it was a duplicate. I said that as a preemptive strike :) But your links are really interesting and helpful! And I am well aware of Gelman, he really helped me understand statistical methodology better (although there is still a looong way to go!). | |
| Mar 14, 2017 at 10:47 | comment | added | DeltaIV | ps just one last comment on the unreliability of research done the way you described. You probably know it already, but Gelman's garden of forking paths does come to mind. | |
| Mar 14, 2017 at 10:37 | comment | added | DeltaIV | I never said that your question was a duplicate of mine, just that it might help, but evidently it doesn't. What about the other link, the pitch by Rob Tibshirani? On page 21, he explicitly mentions being able to control selective type I error, a sort of type I error when the hypothesis being tested is random (in the sense that it depends on the sample data). This seems to me more related to T-consistency than P-consistency (in your terminology), as no loss function is assumed in the definition of selective type I error. But, again, I am no expert in this kind of stuff and I may be wrong. | |
| Mar 14, 2017 at 10:33 | comment | added | Richard Hardy | @DeltaIV, that is, when doing inference, I am not interested in the least false parameters as under P-consistency, but rather I am interested in the true ones (the true partial derivative of $y$ w.r.t. $x$). | |
| Mar 14, 2017 at 10:25 | comment | added | Richard Hardy | @Delta, you understood me correctly. But basically it is not "my method", it is how 90% (?) of research in economics gets done (almost everything except for experimental economics)... which makes me strongly doubt the validity of any and all of their inference... Looking at your link, I see that there the question and the answers address something like P-consistency rather than T-consistency, which I am not interested in for this question here. So the link is interesting, but the questions are not duplicates. | |
| Mar 14, 2017 at 10:09 | comment | added | Momo | Good question. Have you heard of "indirect inference"? Sounds very similar to what you're describing bactra.org/notebooks/indirect-inference.html | |
| Mar 14, 2017 at 10:01 | comment | added | DeltaIV | If I understand correctly, you are collecting data, then selecting a model and then testing hypotheses. I may be wrong, but it seems to me that the selective inference paradigm investigated by Taylor and Tibshirani (among others) could be related to your problem. Otherwise, comments, answers and linked answers to this question might be of interest. | |
| Mar 14, 2017 at 9:23 | history | edited | Richard Hardy | CC BY-SA 3.0 |
added 135 characters in body
|
| Mar 11, 2017 at 13:21 | history | edited | Richard Hardy | CC BY-SA 3.0 |
edited tags
|
| Mar 8, 2017 at 19:32 | history | edited | Richard Hardy |
edited tags
|
|
| Mar 6, 2017 at 12:55 | history | edited | Richard Hardy | CC BY-SA 3.0 |
edited tags
|
| Mar 5, 2017 at 16:59 | history | edited | Richard Hardy |
edited tags
|
|
| Feb 27, 2017 at 18:02 | history | tweeted | twitter.com/StackStats/status/836275223313924097 | ||
| Feb 24, 2017 at 20:11 | comment | added | Richard Hardy | @DJohnson, In terms of inference, I think this is not the worst but perhaps the lightest sort of data mining. I could, for example, test whether $\gamma_2=0$ and make it support my theory in some creative way :) That would probably be the worst. But that is beyong the point. | |
| Feb 24, 2017 at 19:55 | comment | added | user78229 | In my view, the direct answer would have to be there is no way out. Otherwise, you would be guilty of the worst sort of data mining -- recasting the hypotheses to fit the data -- a capital offence in a strict, paradigmatic world. | |
| Feb 24, 2017 at 19:50 | comment | added | Richard Hardy | @DJohnson, Do I understand you correctly that the direct answer would be, there is no way out? If model $(1)$ does not happen to be the true data generating process but model $(2)$ is, we will not be able to test the subject-matter hypothesis using the data set at hand. Is that right? | |
| Feb 24, 2017 at 19:23 | comment | added | user78229 | Your discomfort is endemic to classic approaches to awarding PhDs: careful hypothesis specification, followed by an empirical test and ending with descriptive causal inference. In this world, the short answer is, "no," there is no way out. However, the world is evolving away from that strict paradigm. For instance, in a paper in the AER last year titled Prediction Policy Problems by Kleinberg, et al, they make the case for data mining and prediction as useful tools in economic policy making, citing instances where "causal inference is not central, or even necessary." It's worth a look. | |
| Feb 24, 2017 at 19:01 | history | asked | Richard Hardy | CC BY-SA 3.0 |