Is the methodology for my undergrad dissertation sufficient - should I use a hierarchical negative binomial model instead, despite beginner ability?

Question

As said in the title, I know almost nothing about statistics.

My hypothesis for my dissertation is that UK Members of Parliament with a larger margin of victory will do less work than those with a lower margin of victory -- because they will inevitably reelected to have less incentive to work hard. To determine this, I have collected data for all MPs on the frequency they carried out the following activities in parliament using various APIs and web scraping: sponsoring an amendment, sponsoring a private members bill, sponsoring an early day motions, voting in a division, membership of a select committee, writing a question to a minister, speaking a word in parliament.

I have then converted these variables into the amount of hours it would've taken to carry out that activity -- which was determined in discussion with parliamentary staff. For example, signing an early day motion should take around 4 minutes, so I divided the frequency of this activity by 15. Once all the variables have been converted into hours, I aggregate them to create a "Hours spent on legislative activity" variable.

I've read "Statistics for Dummies" to try and make some more sense of what I should do. I then decided to take a similar approach to Bowler (2010), who's done similar research but only looking at whether there is a correlation between the number of private members bills introduced and margin of victory -- using a Poisson model -- see this album for a table and graph. To do so he took the average margin of victory, then defined marginal constituencies as one standard deviation below the average, and safe seats one above the average. Here are my preliminary results (need to work out how to plot it and break it down by party):

So it seems to prove my hypothesis -- the average hours spent on legislative activity in marginals is more than normal and safe seats. I am pretty happy with this methodology, however I really want the methodology to be as close to "flawless" as possible so I can get higher marks. There are two other UK-based pieces of research, both from the same author but only measure one parliamentary activity:

• Kellermann (2013): https://imgur.com/a/VAjuQvV (methodology) "Posterior predictive intervals for the mean number of EDMs introduced as a function of party and vote margin"
  • this seems years above my level and I can't find any ELI5 explanations, but is a "Bayesian hierarchical negative binomial hurdle model" something I should look into doing -- is this even applicable to my dataset? Even if not it would be great if you guys could point me in the right direction so I can gain an understanding so I can explain why I shouldn't choose this kind of model. If it's too hard to explain, I have a few days I can spend learning so any resources you can point me to that I'd understand would be much appreciated!

• Kellermann (2015) -- "Relative Change in [Written] Question Frequency as a Function of Electoral Margin", uses "a negative binomial regression model with MP-level random effects. The random effects allow for systematic individual-specific differences in the frequency with which MPs ask questions, while the use of the negative binomial distribution allows for within-MP overdispersion in the number of questions asked by members from session to session."

  • This seems slightly less complicated than the one above, -- not sure why he chose a simpler approach in later research. Another comment suggested I should look into adding "random effects" so maybe this one is more appropriate?

If neither of these models are appropriate, are there any other changes I should make or models that I should consider? Thanks all!

Answer 1

For an undergraduate thesis, my opinion is that your methodology is sufficient as is, although your table is confusing; more discussion later.

Without meaning offense, if you want to use a statistical concept, you should be sure you can explain it in plain English, and that you can interpret any associated coefficients.

Linear vs count models

When you fit a linear regression model (aka ordinary least squares, which is the one you almost certainly learned in intro statistics), you're fitting a model that explains the mean of $y$ as a function of covariates. For example, say $y$ is time spent on legislative activity.

$y = \beta_0 + \beta_1x + \epsilon$

$x$ could be a binary dummy variable for being in a safe seat, or it could be the raw margin of victory in percent. Or we can make dummies for being in a safe seat and in a marginal seat (i.e. the average seat is the implied base category).

This sort of model is acceptable for most continuous outcomes. Some types of outcomes are better modeled with other models. You don't need for $y$ to be normally distributed. You do want $\epsilon$, the residual error, to be normally distributed if possible.

Some types of outcomes are inherently better modeled using a different type of regression. Counts of things, e.g. number of fish caught or number of days in a hospital, are typically modeled using Poisson or negative binomial models (although a linear model may still not be too far wrong!).

Now, in a Poisson model, you're in a very similar set up to OLS:

$E(y|x) = exp(\beta_0 + \beta_1x)$

Counts are typically discrete items, e.g. 0, 1, 2, 3, ... As far as I recall, your $y$ doesn't absolutely have to be discrete, but you'd ideally want some explanation about why you chose a Poisson model if they aren't (e.g. perhaps you can show that the mean squared error is lower with a Poisson model, or perhaps because that model is widely used in similar political science applications).

The Poisson model assumes that the variance of $y$ is equal to the mean. (Strictly speaking, I believe generalized Poisson models assumes variance is proportional to the mean.) Negative binomial models specifically model the mean, so if your variance isn't proportional to the mean, you'd be justified in using one. Generally, you can show the negative binomial model's dispersion parameter and test if it's equal to zero.

If none of the above is intelligible, I'd discourage you from fitting a count model.

Hierarchical models

If you had repeated observations of the same MPs over two or more time periods, this is one situation when you'd use a hierarchical model. You're essentially imagining that each person has their own mean amount of hours (controlling for whatever else is in your model), i.e.

$y_it = \beta_0 + \beta_1x_it + \u_i + \epsilon_it$

Above, $i$ indexes people, and $t$ indexes time (or it could be a different sort of clustering, e.g. maybe a political subdivision, on the assumption that MPs from the same subdivision might spend similar amounts of time legislating). If you think you are in this scenario, you should do some background reading on hierarchical or random effects models. You should learn to recognize when you're in this sort of situation, and you should learn what the variance of the random intercept ($\u_i$) and the residual variance ($\epsilon_it$) mean. I didn't see the context of the comment asking you about random effects.

Zero inflated and hurdle models

Maybe you have a lot of people with zero hours spent legislating. Maybe there are two separate processes at play: one causes MPs to decide if they should propose legislation or not, and a separate process determines the number of hours they spend legislating. That's the set up for the hurdle model you alluded to, as well as zero-inflated models.

At this point, I confess I'm less familiar with hurdle models, but I believe they have a probability model for if you clear the hurdle or not (e.g. zero hours in this example), and a separate model for the mean of the outcome conditional on clearing the hurdle. Even if you don't use Stata, their examples are very clearly written, so you might benefit from reading their description of hurdle models.

In contrast, a zero-inflated model assumes that one class (modeled by a probability model) always has $y = 0$, and that another class has $E(y)$ distributed Poisson or negative binomial (NB: that class can have a zero outcome as well; the distinction is between a structural zero class and a 'regular' class).

If you want to use one of these models, you'd want to do background reading to figure out scenarios where they conceptually work, you'd want to be able to explain the (now two sets of) coefficients in plain language, and you'd want to know the standard test (it's a likelihood ratio test) to compare a zero-inflated count model to a regular count model (and I'd assume there's a similar test for hurdle models).

Again, if you couldn't explain the above in plain language if asked, I'd probably recommend not using this class of model.

Bayesian vs Frequentist

Most of us are trained in frequentist statistics. Bayesian statistics does offer some conceptual advantages, but you will need to do some work to comprehend the basics. Google this one on your own, but for undergraduates, I'd generally suggest doing frequentist statistics.

With respect, your table is confusing

For each cell, you present the mean and SD of (estimated) hours spent on legislative activity. That's fine. However, what is your model? Did you fit one Poisson model to each cell? If so, why not fit a Poisson model with a dummy variable for type of seat to the whole data? What is mean difference? Is the t-statistic for each cell relative to a count of zero? What is "significance" - if it is 1 - p-value, why didn't you just present the p-value?

With respect, you mentioned some relatively advanced statistical models that I would expect someone with an MS in statistics or a PhD in an applied stats field or economics to be able to understand. At the same time, it's not clear that you really have the basics down. I would recommend you get the basics down before proceeding to something more advanced. As mentioned in comments, I think you'd be entirely justified fitting an OLS model, and discussing some of the models you had referenced in your post (provided you can at least provide a general sense of why they're improvements on OLS).