Why data shuffling has such a dramatic effect in K-Neighbours regression?

Question

I am trying to use the K-Neighbouts for regression and I find to my surprise that not shuffling the training data has a huge effect on the quality of the prediction.

With shuffling. 98% training data:

Without shuffling the data (sequential data). 98% training data:

I am using the Sklearn library in python, and the shuffling function:

train, test = train_test_split(data, test_size=0.02)

Why am I getting this results?

Is the clustering "initial guess" the issue here?

The test suggests that depending on the order the data is fed to the algorithm the neighbours change. I would expect this not to happen.

PS: In the problem I want to predict both the sign of the values and their magnitude, hence the right plot and its sign prediction histogram.

Code without shuffling

import numpy as np
import pandas as pd
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import RobustScaler

df = SOME_PANDAS_DATAFRAME

# split in training and test
train_proportion = 0.98
train_size = int(len(df) * train_proportion)
test_size = len(df) - train_size
train, test = df.iloc[0:train_size, :], df.iloc[train_size:len(df), :]

trainX = train[in_series_train].values
trainY = train[out_series_train].values.reshape(-1)
testX = test[in_series_test].values
testY = test[out_series_test].values.reshape(-1)

# fit model no training data
n_neighbors = 5
leaf_size = 6
p = 2

model = KNeighborsRegressor(n_neighbors=n_neighbors, weights='distance', algorithm='auto',
                            leaf_size=leaf_size, p=p, metric='minkowski', n_jobs=1)

scaler = RobustScaler()

print('Fitting...')
trainX_ = scaler.fit_transform(trainX)
model.fit(trainX_, trainY)

print('Predicting...')
testX_ = scaler.fit_transform(testX)
predY = model.predict(testX_)

Variation to shuffle the data

train, test = train_test_split(df, test_size=0.02)

This is the only code variation

Answer 1

After giving it a lot of thought, I found the problem.

What I am trying to forecast is a value of a magnitude at a certain hour of the day given a number of inputs. I turns out that when the data is shuffled, the neighbours are the values of the magnitude for at other hours of the same day. And those values are the ones I need to forecast.

I guess the lesson here is that if you want to forecast time series you cannot shuffle the data.