# How can one tell if linear regression results are real, or just the result of researcher bias?

I recently took a class that focused mostly on linear regression, and it seems like regressions are very, very easy to manipulate, even unintentionally. Just by choosing different combinations of variables, or omitting vs. including some variables, I can dramatically affect regression results. Even further, this article on FiveThirtyEight describes "p-hacking" and shows how scientists can (often unintentionally) bias their regression results toward results that confirm their biases.

As an example, the article describes a study where several different research groups studied the relationship between race and red cards in football games, and says that because almost every group found significant results in the same direction, that's likely the truth. But isn't it possible that, even though the result makes sense, each of the scientists is just independently confirming their own bias and arriving at the same conclusion?

Can regression results ever be trusted?

(Note: this question was originally asked on Academia StackExchange, which is the wrong place.)

• @StrugglingStudent42 I think these are much too strict and some are incorrect. I can have a model that has $x$ and $x^2$, so linearity of the conditional mean in parameters, rather than in $x$, is what is assumed. Neither normality, homoscedasticity, or independence are absolutely necessary. – Dimitriy V. Masterov Apr 13 at 21:53

You write

"just by choosing different combinations of variables, or omitting vs. including some variables, I can dramatically affect regression results,"

but none of these criticisms are unique to linear regressions, and are valid for non-linear regression or any other analyses.

If I'm reading your question correctly, you're not really asking if linear regressions can be trusted, but rather whether research, or indeed any statistical analyses, can be trusted. That's a fairly subjective and philosophical question, and the article you linked provides some fine answers.

Although the article discusses the importance of examining the same data under different analyses and assumptions (this is sometimes called sensitivity or robustness analysis) to confirm a finding, what is equally important is examining whether the same finding holds using different data. Of course, there can still be problems here (i.e. confirmation or publication biases), but these two approaches, combined proper peer review, in theory (a big caveat!) should lead to trustworthy findings.

• I guess in my head I am comparing linear regressions to more traditional experimental methods. If I do a physical experiment to find out which AA battery (from several different companies) runs out of power the quickest on average, keeping all else equal, then if I run enough batteries and keep strict controls on the conditions I can be pretty sure that the results will be real. But if I gather a bunch of observational data on battery life, battery type, different conditions, etc. and run a regression on it, under what circumstances could I trust the results of that regression? – Merkava120 Apr 14 at 22:04

Indeed these questions are relevant to all ML predictions and not only to linear regression. Can regression results ever be trusted? - It depends on the specific problems, the specific results and in many time it is still subjective and depend on how you interpret the model result. Some guiding questions that can help deciding if the model results are helpful or not:

• Can it generalize? - i.e. Do you get good predictions on unseen data.
• Do you have good evaluation metric? - This way you can compare different models and usually a good practice is to compare your model to a simple baseline and see that you are consistently better than the simple baseline.
• Are you happy with results of the evaluation metric? - This is a tricky and subjective question. In some cases even slightly above baseline is very good. let's take as an example predicting sports game results. The baseline will be the odds of gambling agencies. If you are consistently slightly better than this baseline you can be happy and make a lot of money. On the other hand there are cases in which you are much better than baseline but it is hard to conclude if you can trust the results or be happy with the results. Let's take as an example predicting the probability of returning a loan. You can achieve good results in your evaluation metric but then you can ask yourself questions like, suppose that the model is doing very good on one type of people (specific race for example) and bad on some minority? is it good? are we happy?

To conclude: In many cases linear regression and other ML models can be trusted and do what they were trained for. But trust is philosophical questions and many time subjective and different people can have different interpretation.

I agree that this has little to do with linear regression. It has to do with all statistics. Where you gather and don't gather the data can also bias your results. There is a legitimate question to me if you can generalize from most results (something that much of the statistical literature pays little attention to), serious issues with data errors that I don't think gets adequate attention. Response rates on surveys for example are very low now (25 percent) -what about the 75 percent that do not respond. But what concerns me the most are the many articles/books that argue that there are flaws in various methods, most of which analyst are probably not aware of - does this invalidate past or present findings (how would I even know if I made one of these mistakes). At times the issues raised, such as if outliers invalidate key methods or whether analyst even understand the methods they use, worry me so much I have considered stop running statistics. One article that looked at the interpretation of logistic regression in elite medical journals found so many errors that I was stunned. I read a book once that argued essentially that much of the regression/ANOVA analysis done was entirely invalid given the impact of outliers (from a reputable source to). I read a multi-level analysis once that argued that for all grouped data all the linear models run had wrong SE.... :) I have read so many accounts of possible problems with statistical methods that (as essentially a non-academic analyst) it is pretty demoralizing.