1
$\begingroup$

I'm doing a survival analysis (where i'm a newby by the way) for churn, to try to understand how long an insurance policy stays "alive". My universe are all policies that were created between 1Jan2021 and 30Jun2023

I plotted the survival function but i cannot really understand why on the 30months the curve just drops... I understand that the total months a policy can be alive are 33 months because of my time frame but cannot really understand how to deal with the case where policies don't have the same time to survive... here's my curve: enter image description here

And here is my code :

    current_date = pd.to_datetime("2023-06-30")  # Your current date

# Define a function to calculate the time to churn in months
def calculate_time_to_churn(row):
    if not pd.isnull(row['DTANUL']):
        return (row['DTANUL'] - row['DTEMISS']).days // 30
    return (current_date - row['DTEMISS']).days // 30

# Apply the function to create the 'Time_to_Churn' column
df['Time_to_Churn'] = df.apply(calculate_time_to_churn, axis=1)

df['Censored'] = df['DTANUL'].isnull().astype(int)

# Fit a Kaplan-Meier estimator to your data using the time in months
kmf = KaplanMeierFitter()
kmf.fit(durations=df['Time_to_Churn'], event_observed=(1 - df['Censored']))


# Create the survival plot
plt.figure(figsize=(8, 6))
kmf.plot()

# Highlight specific time points
highlight_times = [12, 24, 30]  # Time points to highlight in months

for t in highlight_times:
    survival_prob_at_t = kmf.predict(t)
    plt.scatter([t], [survival_prob_at_t], marker='o', color='red')
    plt.annotate(f"P({t}mo) = {survival_prob_at_t:.2%}", (t, survival_prob_at_t), textcoords="offset points", xytext=(0, 10), ha='center')

plt.title('Survival Function')
plt.xlabel('Time (months)')
plt.ylabel('Survival Probability')
plt.show()

I would appreciate any help with this :)

$\endgroup$
4
  • $\begingroup$ I find it odd that your data starts from a timeframe roughly 30 months ago and that there's a steep drop off at exactly that duration. Might this be an error in how events/censored data is coded? $\endgroup$ Nov 7, 2023 at 14:34
  • $\begingroup$ well maybe, that's what i'm trying to figure out.. i just created the "censored" flag in every cases that a customer didn't churn does it makes sense? because they might have churn after the last period i have (31-06-2023)... but the thing is when i look at the distribution of policies by time until churn, i only have 2 customers that have 33 months until churn... $\endgroup$ Nov 7, 2023 at 16:37
  • 1
    $\begingroup$ How many data points overall, or are followed-up past 30 months? It may be a low-N issue at the later timepoints. $\endgroup$ Nov 7, 2023 at 17:13
  • $\begingroup$ I have 126.558 policies of which the majority ( 10.775 ) have a time to churn of 1 month then until 22 months of time to churn the distribution is around 2k policies but at 30 months i have 739, at 31 months 49 , at 32 months 15 and then at 33 months only 2 policies $\endgroup$ Nov 8, 2023 at 10:18

2 Answers 2

5
$\begingroup$

Survival curves can exhibit sharp drops at the end in scenarios where the longest followed-up cases happen to end in events rather than censoring. Imagine a scenario where all but one person has had an event or been censored. There is only a single example of cases that have survived that long, so if that person has an event, the curve drops to 0% survival no matter what it was previously. The survival curve becomes a poor estimator with few remaining cases, and can show dramatic shifts based on just a few cases having events when the followed-up population is small.

I think what we may be seeing here is not that a lot of people are ending their policies just after 30 months, we're seeing that we have very little data out beyond 30 months, and those few data points happened to be cancellations. That said, it puzzles me why the confidence intervals are so narrow throughout the plot, and especially at the end - the blocky dropoff is commonly observed in scenarios with low N, but the CI would be much wider if that were the case.

$\endgroup$
2
  • 1
    $\begingroup$ so basically i should consider time period not of 30 months but for instance 24 since after that i have very low N and so the curve will not reproduce reliable results, right? $\endgroup$ Nov 8, 2023 at 10:23
  • 1
    $\begingroup$ @user8419142 Pretty much, at some point the remaining follow-up population is too small to make reliable conclusions - had you seen one person with a 10-year old policy, you'd still have basically no ability to make inferences about what happens in years 3-10. The plot may be visually misleading if it's just showing noise, so you may want to truncate. That said, the CI should blow up as your remaining sample size shrinks, making it obvious that the "% surviving" estimate has lots of variability. I'm very surprised that the shaded CI regions aren't much bigger at the right side of the plot. $\endgroup$ Nov 8, 2023 at 18:49
3
$\begingroup$

Assuming your code is correct (and I can't tell, I don't know the language you are coding in) then the obvious answer is "because a lot of people drop their policies at 30 months".

Why do people drop at 30 months? That's not really a statistical question, but I do note that 30 months is 2.5 years. Are there a lot of policies that last 2.5 years?

More generally, I would look for substantive reasons why this might occur.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.