Formulating a mathematical model for a problem is the one of the most subjective aspects of statistics, but also one of the most important. What are the best references dealing with this crucial but often underlooked topic? And which famous statistician said something along the lines of, "Let the data guide the model?"

In my opinion, Frank Harrell's "Regression Modeling Strategies" is a good reference. In fact, it is probably my favourite statistics book.

I've only studied less than half of the book so far, but have got lots of good stuff out of it, for example, representing predictors as splines to avoid assuming linearity, multiple imputation for missing data, and bootstrap model validation. Perhaps my favourite thing about the book is the general theme that an important goal is to get results which will replicate on new data, not results that only hold on the current data.

Additional benefits are Frank Harrell's R package rms which makes it easy to do many of the things described in the book, and his willingness to answer questions here and on R-help.

• (+1) A good companion textbook is Clinical Prediction Models, by EW Steyerberg (especially for those interested in clinical outcomes).
– chl
Aug 17 '11 at 10:40
• @chl Thanks for the suggestion. I hadn't heard of that book, and will be interested to have a look at it. Aug 17 '11 at 20:34
• @user152509 As far as I know, distributing an electronic copy would be illegal. If you can't buy the book or get it from a library, you can see some of the book at Google Books, and there are some related resources on the Vanderbilt Department of Biostatistics web page. Aug 17 '11 at 20:36

The latter statement seems to be in spirit of Sims critique ((1980) Macroeconomics and Reality, Econometrica, January, pp. 1-48.) where he

...advocates the use of VAR models as a theory-free method to estimate economic relationships, thus being an alternative to the "incredible identification restrictions" in structural models [from wiki]

But probably S.Johansen (one of the pioneers of cointegration analysis) could follow the same spirit. From what I was taught the model building sequence is like:

1. Clarify the primary aim of the model: forecasting, structural relationships (simulations), causal relationships, latent factors, etc.
2. Abstract model is the real world that could be "too real" to cover completely in your application, but it gives feeling (or understanding) about what is going on
3. Verbal model brings some theory or translates your understanding into statements and hypothesis to be tested, empirical (sometimes called stylized) facts are collected at this step
4. Mathematical model only now you can formulate your theory in the form of equations (difference, differential), such models are often to be deterministic (though one can merge this step with the latter one and consider stochastic differential equations for instance) thus you need...
5. Econometric (statistical) model adding stochastic parts, the theory and methods of applied statistics and probability theory, micro- and macro-econometrics.

• Any references for "Sim" or "Johansen"? Thanks! Aug 19 '11 at 9:40

The reference to "letting the data guide the model" can be attributed to George E. P. Box and Gwilym M. Jenkins. In Chapter 2 of their classic textbook, Time Series Analysis: Forecasting and Control (1976), it is said that:

The obtaining of sample estimates of the autocorrelation function and of the spectrum are non-structural approaches, analogous to the representation of an empirical distribution function by a histogram. They are both ways of letting the data from stationary series speak for themselves'' and provide a first step in the analysis of time series, just as a histogram can provide a first step in the distributional analysis of data, pointing the way to some parametric model on which subsequent analysis will be based.

This modelling procedure of letting the data do the talking, as advocated by Box & Jenkins, is obviously referred to throughout the literature on ARIMA modelling. For example, in the context of identifying tentative ARIMA models, Pankratz (1983) says:

Note that we do not approach the available data with a rigid, preconceived idea about which model we will use. Instead, we let the available data talk to us'' in the form of an estimated autocorrelation function and partial autocorrelation function.

So, it can be said that the idea of ''letting the data guide the model'' is a prevalent feature in time-series analysis.

Similar notions can, however, be found in other (sub)fields of study. For example, @Dmitrij Celov has correctly made reference to Christopher Sims' path breaking article, Macroeconomics and Reality (1980), which was a reaction against the use of large-scale simultaneous equation models in macroeconomics.

The traditional approach in macroeconomics was to use economic theory as a guide to build macroeconomic models. Often, the models were made up of hundreds of equations, and restrictions, such as pre-deciding the signs of some coefficients, would be imposed on them. Sims (1980) was critical of using this a priori knowledge to build macroeconomic models:

The fact that large macroeconomic models are dynamic is a rich source of spurious a priori' restrictions.

As already mentioned by @Dmitrij Celov, the alternative approach advocated by Sims (1980) was to specify vector autoregressive equations - which are (essentially) based on a variables' own lagged values and of lagged values of other variables.

Although I am a fan of the notion of `letting the data speak for itself'', I'm not too sure if this methodology can be extended fully into all areas of study. For example, consider doing a study in labour economics to try to explain the difference between wage rates among males and females within a given country. Selecting the set of regressors in such a model will probably be guided by human capital theory. In other contexts, the set of regressors can be selected based upon what interests us and what common sense tells us. Verbeek (2008) says:

It is good practice to select the set of potentially relevant variables on the basis of economic arguments rather than statistical ones. Although it is sometimes suggested otherwise, statistical arguments are never certainty arguments.

Really, I can only scratch the surface here because it's such a large topic, but the best reference that I've come across on modelling is Granger (1991). If your background is not economics, don't let the title of the book put you off. Most of the discussion does take place in the context of modelling economic series, but I'm sure those from other fields would get a lot out of it and find it useful.

The book contains excellent discussions about different modelling methodologies such as:

• The general-to-specific approach (or LSE methodology) as advocated by David Hendry.
• The specific-to-general approach.
• Edward Leamer's methodology (usually associated with the terms "sensitivity (or extreme bounds) analysis" & "Bayesian") .
• Coincidentally, Christophers Sims' approach is covered too.

It's worth noting that Granger (1991) is actually a collection of papers, so rather than trying to get a copy of the book, you can, of course, look up the table of contents and try find the articles on their own. (See link below.)