3
$\begingroup$

I was reading this SE about repeated measures and I was a little bit confused. I have a data set consisting of claim information for million locations for up to 5 years and 99.9% of the observations in the data have zero claim values. I have 100 features related to each location which are pretty much the same in all years (for example the area of the location and etc). so I could have 5 observations for the same location in the data (when the value of all the features are the same), with different target value. (see below)

Location ID | year |feature 1|feature 2|Target

| location 1 | 2005 | 1.5 | 3 | 0

| location 1 | 2006 | 1.5 | 3 | 0

| location 1 | 2007 | 1.5 | 3 | 100

| location 1 | 2008 | 1.5 | 3 | 0

| location 1 | 2009 | 1.5 | 3 | 0

Here are my questions:

  1. Is this an example of repeated measures?
  2. Would it impact my predictions in case I use non parametric, tree based models such as GBM or Xgboost?
  3. If yes, how should I treat this in my modeling approach? Do I have to change the data/modeling approach?
$\endgroup$

1 Answer 1

4
$\begingroup$

Yes, this is a case of repeated/correlated outcomes on the same location.

For predictions, you can exploit that you have previous information from the same location. You can try to do this in multiple ways: Firstly, by lagged features (e.g. number of X in the previous year) and secondly, by using the location as a categorical feature (different algorithms differ in their treatment of such features, one-hot-encoding can be pretty inefficient, while other alternatives can be better - e.g. embedding layers in neural networks, which are a bit like multi-dimensional random effects in GLMs). Cross-validating properly how much some of these should matter actually requires a good number of locations (rather than just lots of observations - i.e. new locations may be more valuable than more observations about locations you already have data for). Thirdly, there's methods that explicitly try to model the time series nature of the data such as time-series methods (AR, MA, ARMA, ARIMA etc. models, or neural networks like LSTMs, transformers and so on).

On the other hand, this also means that you have less information than it seems (i.e. observations on totally different observations would be better for predicting something about a new location, while for predicting about a location in the training data this is not necessarily a downside). You should certainly make sure that depending on what you are trying to predict that your cross-validation scheme (or bootstrapping theme, or whatever else you are using) is appropriate (e.g. if you want to predict for new locations, then whole locations should be either in your training fold or in the validation fold, never split across the two).

If you are only interested in pure prediction and do not care about the uncertainty around your prediction, then doings some of the things that exploit knowledge about the location (e.g. lagged features, categorical feature for location etc.) should be done. However, where at least some of the approaches may break-down a bit is when trying to quantify the uncertainty around predictions (e.g. standard errors, prediction intervals).

$\endgroup$
1
  • $\begingroup$ Thanks Björn for your response. If I treat each row independently, meaning that claim occurrence is independent of the year, I can't use lagged features option. If I want to treat location ID as a categorical variable, I don't think I have enough data in each location to use it as a random effect or use hierarchical Bayesian. I have millions of locations, some have multiple exposure years but a huge chunk have only one exposure year. I will try the cross validation option based on location but again, if each incident is independent, I don't see how that is necessary. $\endgroup$
    – Rio
    Commented Apr 23, 2020 at 20:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.