9
$\begingroup$

I have a general methodological question. It might have been answered before, but I am not able to locate the relevant thread. I will appreciate pointers to possible duplicates.

(Here is an excellent one, but with no answer. This is also similar in spirit, even with an answer, but the latter is too specific from my perspective. This is also close, discovered after posting the question.)


The theme is, how to do valid statistical inference when the model formulated before seeing the data fails to adequately describe the data generating process. The question is very general, but I will offer a particular example to illustrate the point. However, I expect the answers to focus on the general methodological question rather than nitpicking on the details of the particular example.


Consider a concrete example: in a time series setting, I assume the data generating process to be $$ y_t=\beta_0 + \beta_1 x_t+u_t \tag{1} $$ with $u_t \sim i.i.N(0,\sigma_u^2)$. I aim to test the subject-matter hypothesis that $\frac{dy}{dx}=1$. I cast this in terms of model $(1)$ to obtain a workable statistical counterpart of my subject-matter hypothesis, and this is $$ H_0\colon \ \beta_1=1. $$ So far, so good. But when I observe the data, I discover that the model does not adequately describe the data. Let us say, there is a linear trend, so that the true data generating process is $$ y_t=\gamma_0 + \gamma_1 x_t+\gamma_2 t + v_t \tag{2} $$ with $v_t \sim i.i.N(0,\sigma_v^2)$.

How can I do valid statistical inference on my subject-matter hypothesis $\frac{dy}{dx}=1$?

  • If I use the original model, its assumptions are violated and the estimator of $\beta_1$ does not have the nice distribution it otherwise would. Therefore, I cannot test the hypothesis using the $t$-test.

  • If, having seen the data, I switch from model $(1)$ to $(2)$ and change my statistical hypothesis from $H_0\colon \ \beta_1=1$ to $H'_0\colon \ \gamma_1=1$, model assumptions are satisfied and I get a well-behaved estimator of $\gamma_1$ and can test $H'_0$ with no difficulty using the $t$-test.
    However, the switch from $(1)$ to $(2)$ is informed by the data set on which I wish to test the hypothesis. This makes the estimator distribution (and thus also inference) conditional on the change in the underlying model, which is due to the observed data. Clearly, the introduction of such conditioning is not satisfactory.

Is there a good way out? (If not frequentist, then maybe some Bayesian alternative?)

$\endgroup$
  • 3
    $\begingroup$ Your discomfort is endemic to classic approaches to awarding PhDs: careful hypothesis specification, followed by an empirical test and ending with descriptive causal inference. In this world, the short answer is, "no," there is no way out. However, the world is evolving away from that strict paradigm. For instance, in a paper in the AER last year titled Prediction Policy Problems by Kleinberg, et al, they make the case for data mining and prediction as useful tools in economic policy making, citing instances where "causal inference is not central, or even necessary." It's worth a look. $\endgroup$ – Mike Hunter Feb 24 '17 at 19:23
  • 2
    $\begingroup$ In my view, the direct answer would have to be there is no way out. Otherwise, you would be guilty of the worst sort of data mining -- recasting the hypotheses to fit the data -- a capital offence in a strict, paradigmatic world. $\endgroup$ – Mike Hunter Feb 24 '17 at 19:55
  • 3
    $\begingroup$ If I understand correctly, you are collecting data, then selecting a model and then testing hypotheses. I may be wrong, but it seems to me that the selective inference paradigm investigated by Taylor and Tibshirani (among others) could be related to your problem. Otherwise, comments, answers and linked answers to this question might be of interest. $\endgroup$ – DeltaIV Mar 14 '17 at 10:01
  • 3
    $\begingroup$ @DeltaIV, that is, when doing inference, I am not interested in the least false parameters as under P-consistency, but rather I am interested in the true ones (the true partial derivative of $y$ w.r.t. $x$). $\endgroup$ – Richard Hardy Mar 14 '17 at 10:33
  • 3
    $\begingroup$ @RichardHardy, sure, despite being a stats grad student I don't really believe in inference anymore. It's a house of cards so fragile that it's unclear whether it's meaningful at all except in very strict and controlled circumstances. What is funny is that everyone knows this, but no one (well) cares. $\endgroup$ – hejseb Mar 15 '17 at 18:46
3
$\begingroup$

The way out is literally out of sample test, a true one. Not the one where you split sample into training and hold out like in crossvalidation, but the true prediction. This works very well in natural sciences. In fact it's the only way it works. You build a theory on some data, then you're expected to come up with a prediction of something that was not observed yet. Obviously, this doesn't work in most social (so called) sciences such as economics.

In the industry this works as in sciences. For instance, if the trading algorithm doesn't work, you're going to lose money, eventually, and then you abandon it. Cross validation and training data sets are used extensively in development and making a decision to deploy the algorithm, but after it's in production it's all about making money or losing. Very simple out of sample test.

$\endgroup$
  • $\begingroup$ Does that help estimate $\frac{\partial y}{\partial x}$? $\endgroup$ – Richard Hardy Mar 15 '17 at 19:32
  • $\begingroup$ @RichardHardy, yes, you test the same hypothesis on the new data. If it holds then you're good. If your model is misspecifyied then it should eventually fail, I mean other diagnostics too. You should see that the model is not working with new data. $\endgroup$ – Aksakal Mar 15 '17 at 20:24
  • $\begingroup$ OK, then it sounds like the good old prescription of splitting the sample into a subsample for model building and another for hypothesis testing. I should have included that consideration already in the OP. In any case, that seems like a sound strategy. The problem with macroeconomics, for example, would be that the same model would almost never fit unseen data well (as the data generating process is changing over time), so the exact same problem that we begin with would persist. But that is an example where basically any method fails, so it is not a fair criticism. $\endgroup$ – Richard Hardy Mar 15 '17 at 20:29
  • $\begingroup$ Meanwhile, in microeconomics in cross-sectional data setting it could work. +1 for now. On the other hand, once a model has been fit to all available data, this solution will not work. I guess that is what I was thinking when I wrote the question, and I am looking for answers that address the title question: inference from misspecified model. $\endgroup$ – Richard Hardy Mar 15 '17 at 20:33
  • 2
    $\begingroup$ I sympathize with your view. But since sample splitting into "old" and "new" is equivalent to collecting new data, I do not understand where you see a big difference between the two. $\endgroup$ – Richard Hardy Mar 16 '17 at 13:11
1
$\begingroup$

You could define a "combined procedure" and investigate its characteristics. Let's say you start from a simple model and allow for one, two or three more complex (or nonparametric) models to be fitted in case that the simple model doesn't fit. You need to specify a formal rule according to which you decide not to fit the simple model but one of the others (and which one). You also need to have tests for your hypothesis of interest to be applied under all the involved models (parametric or nonparametric).

With such a setup you can simulate the characteristics, i.e., with what percentage your null hypothesis is finally rejected in case it is true, and in case of several deviations of interest. Also you can simulate from all involved models, and look at things such as conditional level and conditional power given that data came from model X, Y, or Z, or given that the model misspecification test procedure selected model X, Y, or Z.

You may find that model selection doesn't do much harm in the sense that the achieved level is still very close to the level you were after, and the power is OK if not excellent. Or you may find that data-dependent model selection really screws things up; it'll depend on the details (if your model selection procedure is very reliable, chances are level and power won't be affected very strongly).

Now this isn't quite the same as specifying one model and then looking at the data and deciding "oh, I need another", but it's probably as close as you can get to investigating what would be the characteristics of such an approach. It's not trivial because you need to make a number of choices to get this going.

General remark: I think it is misleading to classify applied statistical methodology binarily into "valid" and "invalid". Nothing is ever 100% valid because model assumptions never hold precisely in practice. On the other hand, although you may find valid (!) reasons for calling something "invalid", if one investigates the characteristics of the supposedly invalid approach in depth, one may find out that it still works fairly well.

$\endgroup$
  • $\begingroup$ I wonder if this is realistic in practice aside from the simplest of problems. Computational cost of simulations would quickly exceed our capabilities in most of the cases, don't you think so? Your comment on validity is of course logical. However, without this simple yet useful (in aiding our reasoning) notion we would be even more lost than we are with it - that is my perspective. $\endgroup$ – Richard Hardy Jun 7 at 15:31
  • $\begingroup$ I'm not saying that this should e done every time such a situation is met in practice. It's rather a research project; however one take away message is that in my opinion, for the reasons given, data dependent model selection doesn't exactly invalidate inference that would have been valid otherwise. Such combined procedures may work rather well in many situations, although this is currently not properly investigated. $\endgroup$ – Lewian Jun 7 at 16:50
  • $\begingroup$ I guess if this was feasible, it would already be in use. The main problem might be infeasibility due to the large amount of modelling choices that are data dependent (back to my first comment). Or do you not see a problem there? $\endgroup$ – Richard Hardy Jun 7 at 17:06
  • $\begingroup$ There's the odd simulation in the literature exploring misspecification test/model selection first and then parametric inference conditional on the outcome of that. Results are mixed as far as I know. A "classical" example is here: tandfonline.com/doi/abs/10.1080/… $\endgroup$ – Lewian Jun 8 at 22:14
  • $\begingroup$ But you're right; modelling the full process with all kinds of possible modelling options would require lots of choices. I still think it'd be a worthwhile project, although not something that one could demand whenever models are selected from the same data to which they're fitted. Aris Spanos by the way argues against the idea that misspecification testing or model check on the data makes inference invalid. onlinelibrary.wiley.com/doi/abs/10.1111/joes.12200 $\endgroup$ – Lewian Jun 8 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.