Skip to main content

Questions tagged [rare-events]

Situations where a level of a categorical variable occurs very rarely (eg, a rare disease). This can be a problem especially when the variable is the response variable in a model.

Filter by
Sorted by
Tagged with
5 votes
2 answers
488 views

Detect rare high-value measurements in a series of measurements

We do a measurement on 1000 samples to detect if a chemical element A is present, and for each measurement, two cases can happen : the element A is not present, and the values we get are a "...
Basj's user avatar
  • 588
0 votes
0 answers
27 views

Interpreting hazard ratio in table

This poster has been making headlines lately due to its subject and conclusions. Specifically, I'm interested in understanding how the hazard ratios are being calculated in this table: For example, ...
Bryan Shalloway's user avatar
0 votes
1 answer
25 views

Restricted mean survival time for rare events

I like the restricted mean survival time (and the difference in RMST between cohorts) in survival analyses. However, it does not really convey the effect of an exposure on rare outcomes. For instance, ...
Max's user avatar
  • 1
5 votes
0 answers
90 views

Is there an English translation of Ladislaus Bortkiewicz’s book “The Law of Small Numbers" (1898)?

I finished reading Deaths by horsekick in the Prussian army – and other ‘Never Events’ in large organisations. It led me down a rabbit hole of thinking about Poisson point processes modelling demand ...
Galen's user avatar
  • 8,754
3 votes
2 answers
53 views

Is the Kaplan-Meier estimator appropriate when I have observed only one event?

Let's say I have a database with 21 patients and one of these patients has died. In such a scenario, can we apply overall survival analysis using the Kaplan-Meier model or no? Meaning that does it ...
An116's user avatar
  • 367
1 vote
0 answers
61 views

Test set creation for a rare category classifier

I want to make a classifier for a very rare category. The base rate in a random sample is about 0.01%, estimated from finding about 10 positive examples using a zero-shot classifier on 100,000 ...
Quarticle's user avatar
1 vote
3 answers
210 views

Poisson regression for rare events?

Poisson regression is commonly used to analyse count data. However, when we deal with rare events it does not seem to be appropriate any more. At least, graphical criteria to assess the model fit like ...
Student's user avatar
  • 21
5 votes
2 answers
83 views

Classification error when estimating population size of rare phenomena

I need to understand how a particular statistical challenge has been formally recognised or is commonly described in literature, and what the best academic resources are that discuss it. Here's the ...
geotheory's user avatar
  • 647
2 votes
1 answer
156 views

Difference-in-difference regression design

I'm working with a large panel dataset tracking various units (i-dimension) over an extended period (t-dimension). These units are classified as either 'blue' or 'brown', and some (not all) of these ...
John's user avatar
  • 277
0 votes
0 answers
22 views

Modelling Incidents in Survival Analysis

I'm working with a mixture cure model in a panel dataset covering multiple products across several years. The time-to-event, defined as the duration between the auction start and product sale, is ...
John's user avatar
  • 277
2 votes
1 answer
28 views

Resulting beta distribution from two different samples

Let’s say each sample consists of 300 units inspected for defects. I have historic data from 100 samples in the past that give me an idea of what I expect the defect rate to be. I have a new batch to ...
cocoP's user avatar
  • 21
0 votes
0 answers
20 views

Model Selection on Coxme

I am building a "time to event" model using the coxme package in R. I have a lot of mixed effects (15) that I want to input into a global model and find the best fit for - is there a model ...
michellemoyah's user avatar
0 votes
0 answers
21 views

Hold-out Set -- Smallest Acceptable Size given somewhat rare event?

I am contemplating the appropriate percent train-test split (e.g. 90%-10%, 80%-20%) Note I am asking about model evaluation, not cross validation/model building. This is for creating a hold out set to ...
purple-blade's user avatar
1 vote
1 answer
150 views

Logistic Regression: classification table vs model results

I am running a binary logistic regression in SPSS. My sample size is 906, but the event of interest is only observed 66 times (so ~7% of the sample). I read several threads on this site and posts from ...
Emm's user avatar
  • 11
1 vote
0 answers
27 views

Definition of rare event in competing risk

I want to define rare event in competing risk. How many percent of the event must exist to be able to say that the event is rare? It can be said based on the EPV. But if the occurrence of the event is ...
user390812's user avatar
1 vote
1 answer
70 views

Poisson GLM with rare outcome (very negative coefficient)

Suppose I have the model \begin{align*} Y_i \sim \text{Poisson}(e^{\beta_0 + \beta_1t_i + \beta_2X_i + \beta_3 t'_i}) \end{align*} where $\boldsymbol{\beta} = (-1, 0, -10, 0)$ for $t_i = i$ for $i = 1,...
Tom Chen's user avatar
  • 611
15 votes
5 answers
2k views

What theories, papers, or books examine low-probability events, particularly as the number of trials approaches infinity?

Question: What are influential, canonical, or otherwise useful works considering low-probability events? My background: Applied or computational statistics, not theory or pure statistics. How I ...
Mark White's user avatar
  • 10.5k
1 vote
1 answer
57 views

Stability of results in non-inferiority test in small sample size with rare events

I am working with a dataset including 65 participants, 35 in the control group and 30 in the experimental group. The objective is to assess wether the experimental group is non-inferior to the control ...
Listeb's user avatar
  • 11
0 votes
0 answers
29 views

Leads to detect rare events in human related multi-variable time series

I am currently working on a subject and struggling to find a way to tackle it. Ok, so, let's say I want to predict whenever a human is going to faint in a rollercoaster. I have multi-variables time ...
Jules Civel's user avatar
1 vote
1 answer
280 views

Estimate recall for extremely rare events

I want to estimate the recall of a binary classifier. I have a dataset of ~1B examples but I don't know the ground truth, the only thing I know is that positives are extremely rare. I can randomly ...
Ricky's user avatar
  • 13
0 votes
0 answers
210 views

Mitigating cell count <5 or almost zero

Mitigating problem when cell count $<5$ or almost zero a. 2x2 table with cell count zero b. When event rate is small in logistic regression: such that when seen further by an independent ...
SR1's user avatar
  • 31
4 votes
2 answers
273 views

Monte-carlo simulation and extrapolation

I am reviewing some work and the proposed solution seems to me not to be reliable. But I fail to find any references or even consistently formulate why I think this approach does not work. Assume you ...
Raphael's user avatar
  • 314
1 vote
0 answers
33k views

Calculate probability of dying in school shooting using spatial statistics

Some public commentators say that the chance of dying in a school shooting is very low, so they conclude that Americans over-react to them. https://www.washingtonpost.com/outlook/school-shootings-are-...
Iterator516's user avatar
0 votes
0 answers
33 views

Small N analysis for a field experiment question, and rare events logit

so, I am running a "Field" experiment, meaning, people in a real life situation are subject to certain treatment, and I am trying to measure the effect of said treatment. In this case its an ...
M.T's user avatar
  • 3
1 vote
0 answers
75 views

How to fit a distribution if the samples have not been draw randomly

I read the posts here and here. The real-life problem is: In a rare event simulation catastrophic events occur extremely seldom. The performance of my underlying system has an unknown distribution and ...
Chris's user avatar
  • 36
2 votes
1 answer
99 views

Random effects model for meta-analysis of rare events

The Cochrane Handbook states that incorporation of heterogeneity into an estimate of a treatment effect should be a secondary consideration when attempting to produce estimates of effects from sparse ...
user47679's user avatar
2 votes
0 answers
134 views

Statistical test for rare events in meta-analysis

As a pedagogical work, I am planning to perform a meta-analysis with a rare event as the primary outcome. Here is the type of data I plan to collect (the numbers here are fictive): study no. events ...
user47679's user avatar
1 vote
1 answer
382 views

Coefficients interpretation in rare events case - Logistic regression

So, basicaly, me and my team are trying to target our marketing campaigns based on logistic regression's coefficients. The idea is to understand the dimensions that increase or decrease the ...
Hermit97's user avatar
1 vote
1 answer
271 views

Should train and test dataset label distribution match reality?

Assume I’m building a binary classification model to detect a disease that according to research is present in 3% of the population. I’m given a balanced dataset to train the model where half of the ...
Insu Q's user avatar
  • 355
0 votes
0 answers
189 views

CausalImpact - Choose the predictor time series as the shifted series

I asked this at the contributor's repo, but did not get any answers (seems somehow dead). It is about CausalImpact, to measure the impact of an event. I am interested in the CausalImpact to analyse ...
ctrid's user avatar
  • 1
2 votes
2 answers
708 views

Logistic regression with rare events, all events have large residuals and are influential points

I am running a logistic regression model where the outcome is relatively rare (250 out of 5000). My main interest is to see if there are differences between age groups, sex, educational levels, income ...
Barrett's user avatar
  • 151
1 vote
0 answers
122 views

Panel data with extremely rare events

I have panel data consisting of a dependent variable (count) that is greater than zero in only 0.01% of observations. The number of panel units is roughly 3800 and I have 5475 consecutive days of ...
PencilBox's user avatar
1 vote
2 answers
68 views

How can I prove something rarely happens using statistics and mathematics?

I want to prove that something rarely happens. I have a dataset generated by air-conditioners and I need to prove that it is not worth providing statistics including mean, max, median, min, and max ...
xabzakabecd's user avatar
  • 3,475
0 votes
0 answers
33 views

How to do a t-test on rare events distributions?

I have a dataset, which contains several features (feature_1, feature_2, ..., feature_n) and ...
quant's user avatar
  • 511
1 vote
0 answers
61 views

Probability of n people having at least k successes

I have a set of numbers that corresponds to the number of objects per person, $X = (1, 1, 2, 5, 10, 1, ...)$ The data's discrete, where it's highly inflated by 1 (no zeros) and you have a very long ...
Anonymous Scientist's user avatar
1 vote
1 answer
911 views

Sample size for binary outcome with rare events

I want to compute the sample size to compare two groups for a binary outcome where we expect rare events. I will do an example with R. Assume the following expected probabilities of observing an event ...
user3631369's user avatar
3 votes
0 answers
60 views

How to find the [marginal] effect of X on Y when Y is binary and very rare. Can I make groups of similar X and model counts of Y instead?

tl;dr How to model the causal impact of X on binary Y when Y is very rare. Can I make groups and model count instead? Background/What I tried I want to know what the effect of the number of "...
kennethparcell's user avatar
1 vote
1 answer
245 views

Estimating the binomial distribution parameter $p$ for rare events

Suppose we have an extremely large collection of red balls and green balls. If we let $R$ and $G$ be the events of drawing a red ball and drawing a green ball, respectively, and if we let $\Pr(R)=p$ ...
Laars Helenius's user avatar
0 votes
0 answers
41 views

Question about why dependent variables and power with regressions?

Say I am estimating $y_i = \beta x_i + \eta_i$, and Y is either a rare outcome (random example: say deaths of mothers during pregnancy in a developed economy), or if y is an aggregate outcome made ...
Steve's user avatar
  • 661
1 vote
0 answers
409 views

Comparing logistic and Poisson regression for small event rates

I am working through the book An Introduction to Generalized Linear Models (4th Edition) by Dobson and am trying to understand question 9.7, which involves comparing Poisson and logistic regression ...
Kellan Baker's user avatar
1 vote
0 answers
37 views

Modeling rare event

I'm working with a dataset with roughly 14K records, and am trying to predict a binary variable (death). While the total dataset size is sufficient, the patients with deaths I'm trying to predict make ...
Tyler 's user avatar
  • 11
2 votes
0 answers
55 views

Extreme birthday problem

I have an extreme version of the birthday problem. I want to know: The probability that $m$ individuals will share a birthday The expected $m$ given the number of individuals The slight complication ...
Anonymous Scientist's user avatar
0 votes
0 answers
47 views

Unexpectedly poor fit for logistic regression predicting connectivity between vertices in a graph

I have built a graph where each vertex has an associated time point and the edges between vertices are weighted by a pairwise distance measurement ("distance" is a small percentage, ...
Connor Chato's user avatar
1 vote
1 answer
149 views

Hypothesis testing for large N small k

I've got a set of differentially expressed biomarkers that I want to check for the significance of this observation. For a similar problem, I've seen the hypergeometric test being used, where $k$ = ...
Anonymous Scientist's user avatar
1 vote
0 answers
42 views

What are the problems with using a predictor with a low base rate in a GLM?

A reviewer has questioned my use of a dummy coded predictor that has a low base rate of occurrence. Specifically, they question how well my conclusions would generalize given that the event occurs ...
heuristicwondering's user avatar
1 vote
1 answer
228 views

Logistic regression with both rare outcome and exposure

I'm conducting a population study analyzing a rare outcome (prevalence 2-3%) with a rare exposure (~225 exposures, 800 000+ non-exposed). This gives me 5-8 cases among the exposed. Data on 7-8 ...
Peter Z's user avatar
  • 23
0 votes
0 answers
50 views

How can I obtain reliable variance estimates for an exposure with low frequency?

I have a sample of 3000 observations. I wish to model the effect of a rare binary exposure (frequency of less than 50) on a binary outcome (frequency of 300). I also wish to adjust for approximately ...
Bicep's user avatar
  • 137
2 votes
2 answers
747 views

Classification Models giving probabilities at extreme end only

I am building a binary classification model with proportion of 1 is at only 3% and total 70000 data points.I have 5 variables out of which 3 are coming out to be important. I have built model using ...
pankaj negi's user avatar
0 votes
0 answers
26 views

Multivariate rare events

My data is something like this: I have U urns, and I have taken a bag of $n$ objects from each urn. Each urn has $N$ objects, and I have sampled $n$ with replacement. $n$ is comprised of coloured ...
Anonymous Scientist's user avatar
0 votes
1 answer
20 views

Probability of Event - 3 persons in a club win a lottery and also happen to have all black cars

In a club every year they celebrate a special anniversary with a lottery. The Club has 3,000 members. 3 Club members win in the lottery every year (3 different prizes, but that is not material here)....
user8270077's user avatar