0
$\begingroup$

I have a data set of companies having different feature variables like number of employees, sector, revenue or location. And I also have a target variable (energy consumption) I want to predict by means of supervised learning.

As I have data from several years, some of the companies appear more often in my dataset; e.g. I could have data from Microsoft for the years 2014 - 2017, thus Microsoft appears four times in my dataset. Obviously, examples coming from the same company but from different years are closer correlated than data from different companies.

My question is: How should I split the data into training and test set under these conditions? Is this correlation between some samples a problem, even if I provide the company name as a feature to the model (thus enabling it to learn the individual characteristics of the companies)? Should I restrict my data to examples coming from the same year? Or do I somehow need a different model design which is more like a longitudinal study?

Thanks a lot!

$\endgroup$
  • $\begingroup$ Do you want to predict for each of these companies separetely (that is use company as a DV) or do you want a global prediction? $\endgroup$ – user2974951 Oct 24 '18 at 10:00
  • 1
    $\begingroup$ Looks like you need a time series model for each company. $\endgroup$ – user2974951 Oct 24 '18 at 10:03
  • 1
    $\begingroup$ Yes, if you want separate predictions for each company, which makes sense. $\endgroup$ – user2974951 Oct 24 '18 at 10:07
  • 1
    $\begingroup$ Given that these measurements represent different things you cannot merge them, the result would be nonsense. $\endgroup$ – user2974951 Oct 24 '18 at 10:42
  • 1
    $\begingroup$ It depends on the assumptions you're making. For example if you assume a model can adequately predict energy consumption without using the company name as a feature, then it's fine to treat the rows as independent. You can build one single model. The only caveat I would add is to ask how will the model be used? If it is only ever being used on data in past years, then you can split at random. But if you want to use it to predict unseen, future years, then you will need to consider year in your train test split (i.e. the oldest record in test must be newer than the newest in train) $\endgroup$ – Dan Oct 24 '18 at 10:54
1
$\begingroup$

When you have dependencies in your observation, dividing the data into a testing and a training set is not trivial, as you have pointed out. The main issue is that if you remove some of the observations that are of the same company, then you will have a test set and a training set that are depenend of each other. The alternative is to separate the observations into two sets in such a way that each company only appears in one of the two sets, but then you risk missing important information if there's a large difference between companies.

Roberts et al. (2017) discusses the topic and can be worth looking up:

References

Roberts, D. R., Bahn, V. , Ciuti, S. , Boyce, M. S., Elith, J. , Guillera‐Arroita, G. , Hauenstein, S. , Lahoz‐Monfort, J. J., Schröder, B. , Thuiller, W. , Warton, D. I., Wintle, B. A., Hartig, F. and Dormann, C. F. (2017), Cross‐validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40: 913-929. doi:10.1111/ecog.02881

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.