Let's say I'm studying a population of generic emergency calls to over the course of several months, and keeping track of the following independent variables:

  • month (when the call happened)
  • country (where the call originated)
  • typology (what the call was about: robbery, murder, fire, health issues, natural disasters, etc)

My target variable is whether or not the call was a false alarm, i.e., something that wasn't really an emergency. I know as a fact that the vast majority of calls (~90%) are false alarms.

I'm working with a statistical study where the samples are stratified by month and country, essentially sampling a number of calls separately in each country and each month, but across all typologies at random. In this way, I think the most frequent typologies are better represented.

In theory, if the sample is large enough, I can draw meaningful conclusions on what happened in a certain country on a certain month, because that's the population subset where I have a random sample -- by definition of stratified sampling.

Unfortunately, I know that the monthly sample for each country is not large enough, say on average 80 samples over 800 calls. I can't really draw any conclusion on how many calls are false alarms in a given country/month with a satisfactory margin of error and interval of confidence. Let alone doing so by typology, because I didn't even stratify the sampling for that variable. Hence, I'm looking for ways to aggregate my stratified monthly/country samples to derive some statistically valid conclusion on false alarms in countries, and possibly by call typology.

Can I assess how many calls are false alarms in each country if I consider the samples for a large enough number of months all together, and assume there are no month-dependent factors in whether or not a call is a false alarm? In this way, say for a year, I'd have in a country a sample of $80*12 = 960$ calls over a population of $800*12 = 9600$, which at least size-wise looks better representative than a sample of 80 over 800. I'm afraid that since the 960 samples aren't random anymore (they are equally split by month) this could lead to a fundamentally study-invalidating bias even if I assume time-independence.

Let's assume I can do that -- Can I assess how many calls are false alarms for each typology if I consider the samples for a large enough number of months and countries all together, and assume there are no month- and country-dependent factors in whether or not a call is a false alarm? This case is similar to the previous one, but the original sampling didn't stratify by typology -- only by country and month. Can that be an issue, or since the sample has been taken at random across all typologies it's representative enough at least for the most common call typologies?

  • 1
    $\begingroup$ Mixed midels (hierarchical models) are perfect for the scenario you are talking about, are you familiar with these methods? ie lme4 package in R? $\endgroup$
    – NULL
    Nov 7, 2019 at 14:42
  • $\begingroup$ No, not really. As you can read from the text, I'm attacking the problem from the "traditional" theory for sampling, intervals of confidence and margin of errors. I.e., I have a population of N individuals, I sample n of those (with/without replacement), hence I have a MOE and CI which I can calculate with critical values based on the normal distribution, and possibly finite populations corrections if sampling without replacement. I'd love an answer on how to tackle this problem with mixed models though, but I'll start reading right away. $\endgroup$
    – st1led
    Nov 7, 2019 at 14:49

1 Answer 1


Okay, so here is what I recommend:

What not do to:

  • Collapsing by ignoring certain level (for example month) could either be fine but more probably not, as it will have some effect and confounds your inference since you did not control for it.
  • Assuming things and not really testing that assumptions hold. I'm not saying you should actually perform statistical test for the assumptions (even though you often can), but simply evaluating the effect of changing the assumptions on the result of your inference.

Enough of things not do to, let's talk about more interesting stuff, what you can do:

  • Assume you have no idea how to tackle this and start learning how people often attack this sort of problems which is through using mixed/random/hierarchical models.
  • Read Wikipedia pages for terms like hierarchical models, partial pooling and random effects.
  • Download the magnificent book of Statistical Rethinking (version 2, freely available) from Richard McElreath. The pdf is available here. It is password protected (hint: info about where to find the password is provided in the latest course's webpage).
  • Read the book and also go through the lectures (latest version) for example here
  • Now you will have a very good understanding of how to tackle this.
  • You could either go through the lme4 docs and try to get things done in a frequentist way, or you might want to use a fully Bayesian way, it all depends on you.
  • There are literally tons of documents, article, books on how to do these things, but the most important thing is to learn what these models are and what they can do for you and this is exactly what McElreath has done, in an outstanding way, IMHO.

Here are some other books I recommend reading (after that book) if you are into this:

1 - Data Analysis Using Regression and Multilevel/Hierarchical Models

2 - Mixed Models: Theory and Applications with R

Good luck!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.