Suppose we are building an "early warning system", that will output a risk score every month over 1 year, say Jan. to Dec..

The tricky part is that number of available variables/columns will be different (increase over time). For example, at beginning (Jan.), we only have $5$ features/columns for each instance, every month we will have more features. So, by December, we will have say hundreds of features.

What is a good practice of dealing with such data? Should we build $12$ different models that only work for specific month? Building $12$ models and switching among them would be a messy implementation. Is there any better way of doing this?

Here is an example of data I was talking about:

  • Month 1: Student age, gender, income, ...

  • Month 2: all month 1 features/columns + Student math unit 1 test score, science unit 1 test score, ...

  • Month 3: all month 2 features/columns + Math GRE score, Writing GRE score ...

  • Month 4: all month 3 features/columns + Survey response data ...

The idea is that different types of the data is only available at certain time of the year. In above example, student can only take unit test 1 at Feb. and GRE at March, and give survey response at Apr. etc.

  • $\begingroup$ Could you explain how, if at all, this differs from a time series? $\endgroup$ – whuber May 23 '17 at 17:28
  • $\begingroup$ To clarify, is the data you are adding sequential? Or are you actually adding different features? In other words, if say your feature is "length", does each month get that feature, or are you adding completely new features like "height"? $\endgroup$ – Alex R. May 23 '17 at 17:29
  • $\begingroup$ @whuber the data in my mind is really different from time series. Please see my example. $\endgroup$ – hxd1011 May 23 '17 at 17:33
  • $\begingroup$ @hxd1011: Sorry I took a while to get around to checking this. Please see my answer below; as also hinted by whuber while on face value you do not have a time-series, conceptually you do have time-evolving data that need to be properly "translated" into features. $\endgroup$ – usεr11852 May 28 '17 at 14:54

I think a cross between a neural network and a vanilla RNN would work for this.

predict1                        predict2

     |                                      |

[Module1] ---h1 ---> [Module2] --->h2--->....

    |                                        |

    |                                        |

   x1                                     x2

So each xi is relative to each month, meaning that if you add features, xi, will grow in size. Each module's $m$ weights are relative to the set of features available in that month $m$, and the memory input of the last module. So unlike an RNN, the modules change with each extra month of data.

Note that you can grow the size of your memory hi, to accomidate for larger xi. To train the model, for each data point you would first identify how many months $m$ of data you have, and then apply the modules $m$ times, to get a sequence of $m$ predictions, which you can then evaluate using whatever error metric you want.

  • $\begingroup$ could you tell more more what is $h_i$? the output of module $i$? $\endgroup$ – hxd1011 May 23 '17 at 18:29
  • $\begingroup$ They are the hidden memory units that are passed along by vanilla RNNs. So it's just a vector. See any guide on RNNs for more detail. For example (replace $s_t$ with $h_t$): wildml.com/2015/09/… $\endgroup$ – Alex R. May 23 '17 at 18:37
  • $\begingroup$ thanks for answering. I am not familiar with RNN and will read more about it. $\endgroup$ – hxd1011 May 23 '17 at 18:39
  • 1
    $\begingroup$ +1 you definitely helped me a lot by providing a "more elegant way of managing models". I was thinking to build 12 models, then in production, it is very hard (in terms of coding) to train them separately and in production switch models at different time. $\endgroup$ – hxd1011 May 23 '17 at 18:51

As it has been already noted building $P$ distinct models for each of the $P$ periods is probably an overkill both in terms of training as well as deployment. I think there two major points that must be considered:

  1. treating the initial question as producing a probability for a student to "churn" (to use a standard marketing term) during her life-cycle.
  2. treating the "new features" as additional information to existing features rather than stand-alone new pieces of information.

These points combined culminate to the reformulation of the original problem as one where the number of features does not change over time; we have a fixed number of features that potentially evolve across time.

I briefly elaborate each point separately:

Point 1. To start with, the fact we have some predefined periods is somewhat immaterial. In every time-point from enrolment to graduation a student has a "probability to churn". We should treat the period as one more feature. It is extremely unlikely that the baseline probability to drop-out is constant across periods, as such the "academic age" is an important feature. (Probably using absolute days is even better) Just with that we move the goal-posts from $P$ models to $1$. If our classifier can handle missing values we are sorted. On to the next step!

Point 2. We must consider what the features at hand show. You write: "Student math unit 1 test score (...), Math GRE score, (...)"; what I read is: "Maths aptitude". We can use the standard metrics for short-time series prediction; last value, median, linear trend, etc. we can dress it up as we like, but bottom line is we measure Maths aptitude. We have multiple aptitude indicators for subject $X$ that inform our general $X$-specific features as time evolves. The same extend to survey-response data. Students always have positive or negative attitudes towards some aspect of their courses and these evolve over time.

Point 2+1. (Some extra sub-points deriving from above) The missingness mechanism is probably very important (eg. students who miss exams are almost certainly more probably to drop-out than the ones who never missed an exam and this is probably an additive effect); this will be a pain to include in $P$ different models but extends naturally from points 1 & 2. Similarly, "market conditions" and seasonality matter (eg. students who got A's when everyone else got A's or B's are probably not as strong as the one's that got B's when most the class scrapped C's and D's).

In general I have the slight impression you jumped from EDA straight to learning and did not labour enough the feature engineering part of the question.

  • $\begingroup$ Great points!, and I agree I am moving too fast to a "machine learning problem.". However, the reason is we want to let computer to do more "feature engineering", and human to do more with "diversify" the data sources. The end goal is building a black box model faster. $\endgroup$ – hxd1011 May 28 '17 at 15:24
  • $\begingroup$ I would suggest that the end goal is to get a right model faster. $\endgroup$ – usεr11852 May 28 '17 at 15:39
  • $\begingroup$ Cool. I am glad you find them helpful. Deviation because you mentioned "bosses/real world": in 10-fold! $\endgroup$ – usεr11852 May 28 '17 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.