3
$\begingroup$

I have encountered this problem on how to predict the probability of a periodically happening event occurring at a given time.

For example, we have an event called being_an_undergrad. There are many data points: bob is an undergrad from (1999 - 2003), Bill is an undergrad from (1900 - 1903), Alice is an undergrad from (1900 - 1905), and there are many other data points such as (2010 - 2015), (2011 - 2013) ....

There are many events(data points) of being_an_undergrad. The duration interval varies, it might be 1 year, 2 years, 3 years, .... or even 10 years. But the majority is around 4 years.

However, I am wondering given all the data points above. If I now know that Jason starts college in 2021, and how can I calculate/predict the probability that he will still be an undergrad in 2022? and 2023? and 2024 .... 2028, etc.

Improve this question
$\endgroup$
4
  • $\begingroup$ Are you able to find the distribution of durations of being an undergrad? $\endgroup$
    – Dave
    Commented Mar 27, 2021 at 12:22
  • $\begingroup$ Hi @Dave ! Thanks a lot for your comment! Actually I would say the distribution is not obvious or typical in this case. At first I thought it might be normal distribution, but after visualization of the data, I found that it is not very typical. May I also ask if there are any ways to determine the distribution of the data besides data visualization? $\endgroup$
    – Leonard
    Commented Mar 27, 2021 at 14:33
  • $\begingroup$ How do you wish to deal with people who start as undergrads, then drop out of school, then re-enroll as undergrads again? $\endgroup$
    – EdM
    Commented Mar 27, 2021 at 14:34
  • $\begingroup$ Hi @EdM ! For this situation, I treat them as two seperate data points. For example, <Leo, isUndergrad, 2010 - 2011>(1 year) and <Leo, isUndergrad, 2013 - 2018> (5 years). $\endgroup$
    – Leonard
    Commented Mar 27, 2021 at 14:38

1 Answer 1

Reset to default
1
$\begingroup$

You want to know the probability that someone will be in the undergraduate state under specified conditions. You also seem to have data only for year-long time periods. So a simple approach would be to treat this as a logistic regression to predict the probability of that status at the end of each year-long time period.

For each individual you set an appropriate time = 0, maybe the year that the individual finishes pre-undergraduate training. Label your outcome variable at the end of each time period as 0 for non-undergraduate and 1 for undergraduate. With logistic regression, model the probability of state 1 at the end of the time period as a function of relevant predictors at the beginning of the time period: e.g.: age at study entry, the year that represents time = 0 for the individual, demographic and socio-economic data potentially changing in each time period. The cumulative prior years as an undergraduate at the start of the time period would be critical to include; you might also want to include the state at the beginning of the time period as a predictor.

The standard data structure would be a "long" format in which each row of data has the individual ID, the time expressed relative to the individual's time = 0 reference, the predictor values at the start of the time period, and the state of the individual at the end of the time period. Then logistic regression can model the probability of undergraduate status pretty flexibly as functions of your predictors.* For any set of predictor variables, you then run predictions of probability in undergraduate status as a function of time after time = 0.

*This is essentially a discrete-time survival model that allows for transitions between two states. A related approach, particularly if you have data on actual transition times and not just status at the ends of annual time periods, would be a multi-state survival model. The R survival package has a vignette on multi-state models. The diagram at the bottom right of Figure 1 is closest to your situation, with a reversible transition between Health and Illness corresponding to the transitions between your undergraduate and non-undergraduate states. If you have information on graduation, I suppose you could consider that to correspond to the Death state in the figure.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi @EdM ! Thanks a lot for your very detailed answer! I upvoted your answer! But sadly as I still do not have enough credit, it seems that my upvote could not be seen explicitly. But anyway, I am really grateful for your detailed answer! $\endgroup$
    – Leonard
    Commented Mar 28, 2021 at 15:03
  • $\begingroup$ @Leonard accepting the answer, as you did, is fine. I got my first Gold Badge on this site for multiple answers like this--accepted but without other upvotes--as an "Unsung Hero." Keep visiting and contributing to the site; you can learn much here. $\endgroup$
    – EdM
    Commented Mar 28, 2021 at 15:14
  • $\begingroup$ Thanks for your devotion and encouragement @EdM ! StackExchange sites are indeed very good communities $\endgroup$
    – Leonard
    Commented Mar 28, 2021 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.