Unwritten laws and dirty tricks to influence the outcome of a regression analysis Background:
Some years ago, I was working for a professor specialized in macroeconometrics. As a student research assistant, it was my task to replicate other papers and "play around" with the data. I learned quickly that many published results were not even robust to small changes in the modeling setup (e.g. if you omit one variable, change the data range, add more lags). 
In theory, the researcher hypothesizes, does one regression and reports the results. But in practice it seems to be an unwritten law to estimate hundreds of regressions and publish one. 
My question: What are unwritten laws and dirty tricks to influence the outcome of a regression analysis? Do you know some reference  (academic/ semi-academic) where the author describes such an informal procedure to obtain preferable results?
 A: Just because an analysis is sensitive to changes to the model doesn't necessarily imply that the researcher tried many models and then published the one that worked.
One need only hypothesize the existence of a number of researchers each considering a somewhat similar question, each trying only one analysis, and the first one that tries the 'successful' model is the one who published.
The lack of any sensitivity analysis would then be more an indication of incompetence or carelessness rather than actual significance hunting by any one individual.
That's not to say it (significance hunting) never happens, it's just that you can't tell by the mere existence of such a situation that it was what went on in any particular case. In practice I see ignorant incompetence or flat carelessness (in the face of the publish or perish mentality, it's no surprise at all - what's the payoff for taking the time to do things with proper care?) much more often than deliberate misconduct; all it takes above that is a large enough cohort of people thinking about similar questions for all those "special case" results that don't really generalize to eventually turn up in the literature.
The source of what is effective significance hunting, even when everyone tries only one model is the existence of journals that want to publish 'significant' results. They act to create results that can't be reproduced (that are noise, basically). If the journals want to publish results that are nearly all false positives, all they need do is continue as they are. 
Even many journals that have an official policy geared toward avoiding these problems nevertheless have a de facto policy of only publishing significant results, because they accept the practices and recommendations of reviewers who insist on them in spite of the stated policy.
To get better results, they need to actively encourage a focus on sizes of effects, and concentrate a significant fraction of effort on publishing well-conducted studies that find nothing. [If the professional groups and journal editorial boards - and editors - can't comprehend the scientific importance of not finding anything going on (and the very dire consequences of ignoring the null results), they should probably get out of that business and go into something relatively more scientifically honest, like running a psychic reading phone line. At the least, any pretence of rigor should be dropped.]

In the vein of suggesting that may people are not doing it deliberately, here are a few more-or-less unconscious 'dirty tricks' that can inflate rates of finding significant results:


*

*doing model/variable selection without accounting for the effect of it. This is probably the biggest one, it can happen at a number of points in the research, and the researcher may not even be conscious that they're doing it (sometimes it's subtle).
Indeed, the basic 'modelling cycle' paradigm (like the flowchart found in Box and Jenkins, say) leads to this issue. That's not a criticism of that flowchart, by the way - but one must properly deal with the effect of these procedures.
It's worth reading Frank Harrell's book (Regression Modelling Strategies), in particular chapter 4. Related information can be found in a number of other places.

*focusing on model-assessment (particularly looking at things like normality, homoskedasticity or considering the addition of interaction terms or terms to pick up curvature), or just focusing harder on it when nothing is found - and then trying a new analysis when some failure of assumptions is detected - than when a desired result is found.

*trying to find out if there's a better post-hoc procedure when the first one you tried doesn't give the result you needed.
A: 
not even robust to small changes in the modeling setup

I'm analytical chemist/chemometrician. In my field, the related key words 
related to demonstrating/stating how robust the model/the whole analytical method is against certain influences are robustness and ruggedness (There's a whole body of literature including regulations on these topics).  They are applied to the whole method, not only to the data analysis, but the same principles apply (how much do the analysis results deteriorate if e.g. pH varies, a different lab does the work, some features are excluded, etc.).
The key point is that you have to sit down and think hard and put together a list of conditions to test the ruggedness for.  
As for the robustness of data analyses, here's some literature that may be of interest to you:


*

*One robustness parameter that is very easy to determine is the robustness/stability of model and predictions against perturbing the training data. In my work, that is e.g. "How much do models/predicitons vary if few training patients are exchanged for other training patients?" Such measures you can basically get for free if you are anyways doing repeated/iterated $k$-fold cross validation. 


*

*Beleites, C. & Salzer, R. Assessing and improving the stability of chemometric models in small sample size situations Anal Bioanal Chem, 2008, 390, 1261-1271.
DOI: 10.1007/s00216-007-1818-6

*Dixon, S. J. & Brereton, R. G. Comparison of performance of five common classifiers represented as boundary methods: Euclidean Distance to Centroids, Linear Discriminant Analysis, Quadratic Discriminant Analysis, Learning Vector Quantization and Support Vector Machines, as dependent on data structure Chemometrics and Intelligent Laboratory Systems, 2009, 95, 1 - 17.
DOI: 10.1016/j.chemolab.2008.07.010


*Here's a paper simulating different perturbing influences on measurements and then looking at the deterioration of the predictions:


*

*Sattlecker, M.; Stone, N.; Smith, J. & Bessant, C. Assessment of robustness and transferability of classification models built for cancer diagnostics using Raman spectroscopy J Raman Spectrosc, 2010, 897-903.
DOI: 10.1002/jrs.2798


*If you read German, I could send you my Diplom where I tried to find a good set of pre-processing steps for infrared spectra. It turned out that other than having a general sensible choice (from physical/chemical/biological knowledge of the application and the measurements), the only tested pre-processing method where the choice actually had a consistent influence was applying a rather strict quality control filter.
E.g. there's no question that normalization is needed because of the experimental set-up, but the actual choice e.g. min-max vs. area normalization didn't show any consistent influence. In turn, I conclude that the analysis is reasonably robust against the particular normalization method.

now about the dirty tricks:


*

*All kinds of things that lead to optimistic bias in validation results


*

*Data leaks between training and test data


*

*not splitting the data properly = at the topmost level in the data hierarchy (e.g. treating samples prepared from the same stock solution as independent, treating multiple measurements from the same patient/same time series as independent, etc.)

*Data-driven feature reduction (PCA, PLS) done on the whole data set, "validating" only the last level of the model (e.g. the regression done in PCA score space).  

*Same is true for any kind of pre-processing that involves more than one case: all these have to be done on the training data only, and then the results are applied to the test data. 

*Model selection bias: Also data-driven model optimization/selection without outer validation of the final model is a kind of data leak. 



*Creating "self-fulfilling prophecies" in the modeling process


*

*assign reference labels for classification using a cluster analysis

*exclude cases because they don't fit into the model without reporting proper criteria for outlier exclusion
(Note that in some applications an automated decistion/filter to reject bad cases/measurements that are out of the specified domain of applicability is possible and sensible, though) 


*(test) data sets not representative for the application. E.g. the applications I work on frequently deal with medical diagnoses. There are always difficult/borderline cases where it is hard to obtain reference diagnoses. However, excluding such cases from the data analysis creates an artificially easy problem that excludes all those cases for which the model would be needed most.
For a discussion in the context of semi-supervised models see
 Berget, I. & Næs, T. Using unclassified observations for improving classifiers J Chemom, 2004, 18, 103-111. DOI: 10.1002/cem.857 


*

*claiming that (resampling) validation results for unknown cases generalize to unknown future cases. Also leading to an optimistic bias (bullet 1). 

*Jumping from observations e.g. on cell line data or xenografts to conclusions about humans. Or from few precisely specified and selected groups of humans to applicability as medical screening tool, etc.


*Not having enough test cases to warrant the conclusion. At least a rough sanity check on the validation results should be done (e.g. is the confidence interval for the validation results narrow enough to allow practically relevant conclusions, i.e. with respect to what would be considered very good, reasonable, bad and too-bad-to-dare-reporting models.)

*Humans are biased towards recognizing patterns (as opposed to overlooking patterns). This can also lead to setting up too complex models which are overfit and not robust at all. 
All these points involve a trade-off between what is good and sensible and what is too much, and/or which influencing factors are important and which are not. Personally, I can live happily with modeling where all kinds of decisions are done by the data analyst as long as these decisions are reported and justified.
OTOH, I think one needs to judge carfully which level of validation is sensible in a given application. All (or most of these) these points can make sense in certain situations, but they limit  the conclusions that can be drawn. Which in itself is IMHO not a problem - this is just a very normal way to "pay" for using a method can can be practically applied. Problems IMHO arise from not being  aware of the limitations.
