2
$\begingroup$

I am using a random forest regressor and I split the independent variables with shuffle = True, I get a good r squared but when I don't shuffle the data the accuracy gets reduced significantly. I am splitting the data as below-

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25,random_state=rand, shuffle=True)

I fit the random forest regressor as below-

reg = RandomForestRegressor()
reg.fit(X_train, y_train)

When I do the cross-validation with unshuffled data, the accuracy is not good, but it improves when the data is shuffled.

This is how the response variable I am trying to predict looks like-

enter image description here

What could be the reason behind that?

$\endgroup$
6
  • $\begingroup$ What is the command you’re using to fit the random forest? Please edit to clarify. $\endgroup$
    – Sycorax
    Mar 24 at 1:02
  • $\begingroup$ @Sycorax just edited the question $\endgroup$
    – lsr729
    Mar 24 at 2:33
  • $\begingroup$ X_al_train, y_al_train isn't the output of any code you've provided. $\endgroup$
    – Sycorax
    Mar 24 at 2:37
  • 1
    $\begingroup$ Does the data have a time index (eg. date when the data point is recorded)? $\endgroup$
    – dipetkov
    Mar 24 at 13:36
  • 1
    $\begingroup$ @dipetkov yes the data have a time index, its yearly data $\endgroup$
    – lsr729
    Mar 24 at 16:52

2 Answers 2

4
$\begingroup$

In our conversation, you revealed two important facts not mentioned in your original question:

  • Your data is (implicitly) indexed by time as it consists of yearly measurements.
  • You are not interested in prediction at all but in estimation.

Large differences in performance with and without shuffling are a hint that the order of the data contains information. Often it's information about time, eg, the order in which the data points occurred or were collected.

Understanding why/how can help you to a) avoid overestimating the predictive performance of the model and, sometimes, b) reconsider how you approach the problem altogether. Your problem seems to be in the latter category.

Don't use data from the future to predict the past.

A common situation where shuffling (before splitting into train/validation/test) is not appropriate is when the data is ordered in time. In that case you want to use the first portion of your data for training and the second portion for validation (the two parts don't need to be the same size). This correctly reflects the fact that the model won't have data from the future when it's deployed in real life.

Since your data has a time index (years), it's not appropriate to shuffle before splitting even if the results are worse. It's a more realistic, if disappointing, evaluation of model performance.

Estimation is not the same as prediction.

If you are interested in estimation, a random forest is not the right approach. You want to specify a flexible functional form for the relationship between features and response and then estimate its parameters. The model also has to take into consideration the serial nature of the observations. There is no need to split the data for training and validation; train on all the data and evaluate how well the model fits instead. Regression with correlated errors might be a good place to start.


Here are some additional thoughts that I leave here because your other question got closed down:

You plot total crop area against time and the most striking pattern is that less land is dedicated to crops since the 70s. Do any of the predictors dropped/rose sharply during that period of time? Was there a large shift from agriculture to manufacturing? Important changes in policy? Do you have any data on economic activity other than agriculture? Should you even lump together data from before and after ~1970s? Maybe not esp. if you don't have data on broader economic conditions apart from agriculture.

$\endgroup$
10
  • $\begingroup$ Thanks, I have posted the plot of my response variable. my confusion is that I am not doing time series analysis. I am providing some independent data and not a time component, and trying to predict a variable. Now, if the response variable is predictable using the independent data, the order of data that is feed should not matter. $\endgroup$
    – lsr729
    Mar 24 at 20:59
  • $\begingroup$ So here are the two cases you are considering: a) Train the model on data from 1955 to 2010 and check how well it does on 2011 to 2020. b) Give the model a random sample of years and test it on the rest. b) might very well mean that the model "saw" 2010 and 2012 and you ask it to predict 2011. Does b) make any sense to you? $\endgroup$
    – dipetkov
    Mar 24 at 21:14
  • $\begingroup$ That's true, b) gives high accuracy and a) gives very poor accuracy. But my understanding is that, if I am providing independent parameters then why the time is important here? $\endgroup$
    – lsr729
    Mar 24 at 21:21
  • $\begingroup$ I guess that makes sense as long as you are not planning to use your model for predicting 2022 because then the model is not going to know what happens in 2021 and 2023. $\endgroup$
    – dipetkov
    Mar 24 at 21:28
  • 1
    $\begingroup$ I wasn't clear enough. If you'd like to ask that question, that's fine. Just do in a post of its own and provide all relevant context. $\endgroup$
    – dipetkov
    Mar 24 at 21:43
3
$\begingroup$

When the data aren't shuffled, the data are split into two parts along the first axis. Observing a large difference in performance between the shuffled and non-shuffled data indicates that something about how the data X, y are sorted is meaningful in some way.

As an example, imagine that your data are sorted by one of the features, and this feature is important for prediction. Without information about that feature in the 25% of the data you've reserved for testing (test_size=0.25), the model can't predict well for those values of the feature.


In a comment, OP has clarified that the data are a time series (yearly data). So what's probably happened is that shuffling the data is letting the model use information from the future to predict the past, essentially giving the model precognition. To avoid optimistic bias, you should split the data so that you're always training the model on data from the past and evaluating it on data from the future.

$\endgroup$
7
  • $\begingroup$ Thanks for the answer and links. I have a question- what's probably happened is that shuffling the data is letting the model use information from the future to predict the past why does the model care about the order of the data? I am not including the time as independent variable, I am just feeding the model some shuffled data. $\endgroup$
    – lsr729
    Mar 24 at 20:44
  • $\begingroup$ Sure, but even without timestamps, the features vary with time, and knowing the future means you can predict the "gaps" in the time-series more precisely. $\endgroup$
    – Sycorax
    Mar 24 at 21:13
  • $\begingroup$ That's true, but how does the model know that the features depend on time? I believe the model should look for only the independent data provided to it in the training. Is it probably because the response variable decreases with time (I have uploaded the plot in the question)? $\endgroup$
    – lsr729
    Mar 24 at 21:34
  • 1
    $\begingroup$ It doesn't know about time, it just knows about things that are correlated with time, and that they're also correlated with the outcome. Random forest particularly is bad at extrapolating outside the range of the training data, so a strong trend in your target means it will be hard for the model to predict the future, but much easier to predict the past if you tell it the future. $\endgroup$
    – Sycorax
    Mar 24 at 21:50
  • $\begingroup$ I see. What other model can you like to suggest in this problem? $\endgroup$
    – lsr729
    Mar 24 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.