How do I avoid time leakage in my KNN model?

Question

I am building a KNN model to predict housing prices. I'll go through my data and my model and then my problem.

Data -

# A tibble: 81,334 x 4
   latitude longitude close_date          close_price
      <dbl>     <dbl> <dttm>                    <dbl>
 1     36.4     -98.7 2014-08-05 06:34:00     147504.
 2     36.6     -97.9 2014-08-12 23:48:00     137401.
 3     36.6     -97.9 2014-08-09 04:00:40     239105.

Model -

library(caret)
training.samples <- data$close_price %>%
  createDataPartition(p = 0.8, list = FALSE)
train.data  <- data[training.samples, ]
test.data <- data[-training.samples, ]

model <- train(
  close_price~ ., data = train.data, method = "knn",
  trControl = trainControl("cv", number = 10),
  preProcess = c("center", "scale"),
  tuneLength = 10
)

My problem is time leakage. I am making predictions on a house using other houses that closed afterwards and in the real world I shouldn't have access to that information.

I want to apply a rule to the model that says, for each value y, only use houses that closed before the house for that y. I know I could split my test data and my train data on a certain date, but that doesn't quite do it.

Is it possible to prevent this time leakage, either in caret or other libraries for knn (like class and kknn)?

Answer 1

You have to be careful you don't confuse two different outcomes. If you're trying to predict the value of a home today, then you have access to all the data above and you're not actually leaking any data. However, the closing price is not the value of the home today, so you're not currently training a model against home values, but against previous closing price.

One way around this is that you might say that a home that sold in the last 3 months is an accurate reflection of the home's present day value. You could then reduce your set of homes to only those that have sold in the last 3 month as homes with an actual outcome. The estimated value of those homes is the closing price of those homes, at anytime in the last 3 months.

The problem there is you've lost data on all homes older than 3 months. Let's ignore that and say you built a model anyway. Now you have a model, be it kNN or anything else, that will give you the present day value of a home given it's latitude and longitude. Remember close price is your target outcome, and close date is deemed irrelevant right now. How good is your model? Well you can wait a months until you collected data on new houses, enter in the longitude and latitude and validate your results.

If you don't want to wait 3 months, then you can turn back time. Pretend today is 3 months ago, then repeat the same exercise by first removing all data in your dataset that occurs in the past 3 months. For your training set, all homes older than 3 months and newer than 6 months can be part of your model building. Once you've built a model, you can validate it against all homes newer than 3 months.