Predicting when a proportion will reach a certain value

Question

Suppose I am trying to model women entering the field of computer science and would like to make a prediction about when the proportion of women in studying computer science in college will reach $.5$.

I have data spanning back several years about the number of women and men entering college for each particular year, and also have a list on the proportion of women in college for some particular year. In other words, 1st list is number of people entering school, and the 2nd list is the total proportion of women across all grade levels. From this yearly data, I would then like to make a prediction about how many years it will take for parity to occur.

What statistical model would work well for this? So far I've only been able to rule out tests that are clearly flawed

1. Linear regression obviously wouldn't be very helpful as we are dealing with proportions and won't want unbounded predictions as time progresses
2. Logistic regression seems valid, but having it reach $1$ as time progresses is not desired.
3. Beta Regression seemed feasible, but I want to make a forecast rather than a guess on the mean true proportion.

This answer seemed pretty feasible, but a bit hacky and I wanted to know if there was a well established standard way of doing this. It seems like it would be a pretty standard problem (similar with estimating vaccination rates) so I was hoping to gain some insight

Answer 1

The typical approach is to fit your model, but your problem is that you don't even have a model. So the question is more like a modelling problem than a statistics problem. This requires domain knowledge.

2. Logistic regression seems valid, but having it reach 1 as time progresses is not desired.

Fitting a logistic growth curve doesn't need to end up at a fixed final level of 1. A problem is that the motivation for logistic growth may not be not very strong. The model relates to a differential equation $$f^\prime \, = \, \frac{k}{a} f(a-f) \, = \, kf - \frac{k}{a} f^2 \, \underbrace{\approx \, k f}_{\text{if $f \ll a$}}$$

It can be seen as an exponential growth model where the growth rate is a second order polynomial instead of first order. For small values or large values the model will be approximately following an exponential curve.

If there is no direct motivation then the models might still fit and predict well for short time scales because growth is more or less linear on a small time scale anyway.

But does it matter in that case (a small order polynomial model fit) that we use a model for exponential growth?

• On the one hand no: Either the extrapolation to 0.5 will only be a short time and a linear model will be fine as well, or the extrapolation will be a long time away and the logistic model may not be accurate. The latter situation will be the case when the growth rate stagnates close to a 0.5 value.

• On the other hand yes: The potential use of the logistic curve can be to fit a model where the value stagnates around 0.5 or something close, and the question revolves around the question when the random fluctuations will first create a situation where there is a year that more women than men enter.

Answer 2

On the one hand, I would still answer as in the linked thread, and indeed believe that a transformation, followed by a standard forecasting method, would indeed by "a well established standard way."

So on the other hand, I would be interested in what exactly about it looks "hacky" to you.

On the third hand, as long as your target variable is well within the unit interval, whether you transform or not may not make all that much of a difference. Yes, if you take a trended method (whether it's linear regression, or ARIMA, or ETS), your predictions (and especially the prediction intervals) will eventually go outside the unit interval, but I would honestly just try it and see whether this is a problem that really comes up within a reasonable forecast horizon. After all, your target proportion is just 0.5. This free online textbook is an excellent introduction to time series forecasting.

On the fourth hand, you can certainly try some more fancy ways of forecasting. For instance, rather than forecasting the proportion directly, you could try forecasting the numerator and the denominator of your proportion separately and divide the forecasts. This will give a slight bias, but I think this is the least of your worries for this kind of forecast. If you then have multiple forecasts (e.g., a direct forecast of the proportion and a ratio of separate forecasts), you can take the average - this very often improves forecast accuracy.

On the fifth hand, a simple time series forecast is likely only a zero-th order approximation. This is absolutely not bad, because in forecasting, one should always include very simple methods as benchmarks - these can be very hard to beat by more "sophisticated" approaches. As Glen_b writes, there is no simple clear trend in female participation in particular fields of study, especially MINT, and that the female participation tends to be lower in countries with greater gender equality has been called the "gender equality paradox", and people write papers about it (e.g., Stoet & Geary, 2018, and that's just the first one I came across in my literature database). So you should ideally use some theoretical underpinning of the process you are forecasting. Unfortunately, although there are lots of people doing demographical forecasting, there is nothing I could point you to immediately (nothing, e.g., in the Forecasting Encyclopedia, a pity).

Answer 3

As said by Sextus Empiricus it's much more of a "physics" modeling problem than a statistical problem.

I would want a model that takes somwhat a control volume approach:

1. The number of women in CS grows by the number of women graduating CS degrees + some factor of women graduating from other degrees.
2. The number of men in CS grows the same way as in 1.
3. The number of men/ women decrease by the amount of men/women retiring at the retirement age and some amount of people switching carrers.
4. This becomes a differential equation that you need to fit the coefficients to based on the past data you have.