0
$\begingroup$

A bit of a background to my questions: I'm coming from the field of strategic management, and the prevalent technique is large sample analysis is fixed-effects or random-effects regression whenever the structure of data allows. Given that note, I see many papers in "A Journals" that treat dependent variables such as R&D intensity or R&D expenditure (strictly positive values) with either fixed-effects or random-effects regression. My specific questions are as follows:

  1. How "wrong" it is, technically, to use such models for positive dependent variables? Specifically, in case of R&D intensity (R&D expenditure divided by sales), the value rarely goes above 2, so I think panel regression (xtreg in Stata) should give somewhat of an incorrect estimation?

  2. How "correct" it is, technically, to use panel Poisson regresstino(xtpoisson or glm in Stata) with a log link for such dependent variables? I've compared the results of xtpoisson and glm poisson family with xttobit (with ul and ll defined), and they're highly consistent, signaling to me that these (xtpoisson and glm) are far better estimators than xtreg.

My significant concern, leading me to asking the above questions, is that some relationships are significant using xtreg, but not xtpoisson, and vice versa. I see even the same authors picking methods for the same DV in different papers arbitrarily, making me very confused as a junior and inexperienced researcher. Your answers are much appreciated in advance, and any reference on the "correctness" or "incorrectness" on such scenarios is highly appreciated.

P.S.: I ran a model with R&D intensity as the DV and judging by sum of squared of residuals, the poisson panel model is doing far better than xtreg (128 vs. 432).

Thanks.

$\endgroup$

1 Answer 1

2
$\begingroup$

I've encountered a similar issue my own field. Researchers frequently use standard ordinary least squares regression to predict raw scores on cognitive tests. These tests are strictly positive, have firm upward bounds on their range, and have scores that are integers (and thus not truly continuous). The answer as to whether this is problematic lies in the aims of the model. Note that I'll talk mostly about non-multilevel models just because it's easier; however, all the same things apply when using multilevel (general) linear regressions for panel data.

In a technical sense, you are correct: standard regressions that assume unbounded, continuous data are inappropriate if your data are not those two things. A variety of alternative distributions, like the Poisson, are available and useful for that very reason. In the book Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences, Cohen et al. point out that ordinary least squares regression on bounded, discrete data can lead to biased standard error estimates in addition to potentially problematic predictions (e.g., what do we do if the model predicts a negative R&D intensity?). Note, however, that the assumption for ordinary least squares regression is normally distributed residuals, not necessarily a normally distributed outcome variable (though still the issue of bounded data can be a barrier to that assumption). So, there is theoretical ground to argue that these models are incorrect and that the inferences derived from them suffer due to their biased estimates. In the case that these models are being used to test theories, meaning that these inferences on significant predictors are the primary reason for running the models, the results may be spurious due to violations of certain assumptions.

Now, the other side of this is the relative robustness of most regression methods. When there is sufficiently large data, the assumptions of ordinary least squares regression become a bit more fuzzy, at least on the practical scale. Obviously, a large sample size does not eliminate the need to check assumptions; however, the relative amount of bias obtained from a model run on data that violate its statistical assumptions can be fairly minimal when there is large data. Essentially, if there is a lot of data with a strong central tendency, then estimates tend to get pulled away from the problematic extreme ranges (e.g., potentially negative estimates) and more toward the average. So, when the goal is to create accurate predictions of the dependent variable, then large enough sample sizes can still perform pretty well even if the statistical model is theoretically inappropriate.

As in all things statistics, the question of what is best with your model comes down to what you want to do with it. If you're testing theories and it's important that the predictors' significance testing is unbiased, then there's a lot of weight to ensuring that the selected modelling methods are appropriate for the data you have (assuming as well that your data are themselves typical for what you might observe in whatever population you're making inference on). If all you care about is getting decent predictive accuracy, then the model can be used and abused more liberally as long as you are explaining the whole way through that the aim is accuracy in predictions over accuracy in inferences (in other words: you need to be telling the audience not to look at the predictors too closely or make anything too much of what is or is not significant). If this is for a study that you intend to publish in some way, then I think it's helpful to think first about what you want to tell with the final product. Is it a critique of previous work and a demonstration on the importance of choosing the correct model? Is it a recommendation to utilize a better (albeit perhaps less familiar) regression method and demonstrate its strengths to an audience who may be more inclined to use "simpler" methods? Do you want to extend some theory in your field and need those nice, robust inferences that come from a well-done model? Whatever the goal is, just make sure that it's more than "you did it wrong, and I've done it right" or "I did this more sophisticated model and now look what I found." We all love a shiny new model, but if it doesn't have any utility, then it's just noise in the background.

Anecdotally, I compared theoretically-appropriate regressions (usually binomial regressions but some Poisson and negative binomials here and there for good measure) to the standard ordinary least squares regressions for predicting raw scores on a variety of cognitive tests using a decently sized data set (N = 3250). I found no practical difference in the predictive accuracy, R2, or predictor effect sizes (estimates differed on the order of 0.01 on average between the out-of-the-box, point-and-click regressions and my theoretically-informed regressions). That's also my unique case: I doubt the same would be true if your dependent variable often has range as small as 0-2, and my variables are very easy to assume having nice, well-behaved normal distributions in the population.

$\endgroup$
4
  • $\begingroup$ Thanks for the answer. In my case, I actually started my analysis using poisson, and just wanted to check OLS as a robustness measure, but out of three relationships, only one remained significant (sample size ~ 15700). I then calculated the residual sum of squared and noticed a huge difference (428 for OLS and 125 for poisson), leading me to investigate this more. I think the severe restriction (most R&D intensity values are really below 1.3) is the main cause. $\endgroup$ Commented Dec 27, 2020 at 13:23
  • $\begingroup$ Definitely seems like OLS is a misspecification then. I agree that the range restriction in the outcome variable is probably the reason for that. As a sanity check, you may want to consider checking the assumptions of both models. Should be that the data fit nicely with the Poisson model but have non-normal and heteroscedastic residuals for the OLS model. Just thinking about what might happen if a reviewer who's used to OLS asks why something else is reported $\endgroup$
    – Billy
    Commented Dec 28, 2020 at 0:29
  • $\begingroup$ Sure, that was exactly my main point of concer: reviewrs being so used to OLS. It's a great idea to run regression diagnostic on OLS as a proof, which I hadn't thought of! Thanks a lot! $\endgroup$ Commented Dec 28, 2020 at 1:52
  • $\begingroup$ Of course! Best of luck on the submission! $\endgroup$
    – Billy
    Commented Dec 28, 2020 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.