# Standardising these datasets so that they are comparable

Context: Let's say I have two datasets from 2000 and 2010. Each dataset consists of an example for each area of a city along with variables A, B and C, which are all numerical variables which describe how the area performs in some measure. Each area also has a score, variable D. Imagine we are trying to build a model which predicts which areas will have the highest variable D in dataset 2010 based on variables A-C in dataset 2000.

Eventually we want this model to be used on variables A-C in dataset 2010 to predict variable D in the year 2020. The problem is that variables A-C will most likely change over time e.g. one variable could be crime rate which falls over time in general. This makes it hard to model because boundaries, rules etc which are found by the model will be using absolute values of these variables. For example, in 2000 a good area may be one with crime rate below 40%, but in 2010 this may be average and instead a good area is one with crime rate less than 30%.

Fortunately, theory says that it is not the actual values of variables A-C which matter, but rather how areas perform relative to each other.

For example, let's say that in 2000 for variable A:

Area 1:  50%
Area 2:  40%
Area 3:  60%


In 2010 for variable A, we have:

Area 1:  40%
Area 2:  35%
Area 3:  45%


We want to turn those scores in values where only the other areas are concerned, so 2000 could become:

Area 1:  0.5
Area 2:  0
Area 3:  1


Then 2010:

Area 1:  0.5
Area 2:  0
Area 3:  1


In reality though, the distributions aren't that neat.

Problem: How do we model the data to make the resulting model most valid for future decades?

Attempts so far: I have considered the following:

1. Zero-Mean and Unit Variance Scaling: Scaling each dataset separately means that values above 0 will be above the mean in both situations but a reading e.g. 0.4 in 2000 could signify different things in different decades because the variance gets scaled to be 1 overall. Also the mean is affected by outliers.
2. Zero-Median and Unit Variance Scaling: Similar to above, but more robust against outliers.
3. Min-max scaling: Set the minimum at 0 and the maximum at 1 for each variable and scale the data for each dataset to that range. The problem here is that this is affected by outliers and it also does not consider the distribution - althought I may have to assume that it just stays the same.
• If you are trying to predict the future based on factors A,B and C, you must have some type of model that takes in these three arguments and spits out a prediction. Why not make YEAR one of the factors in your model? Commented Aug 12, 2016 at 13:36
• There is only data from 2000 and 2010 unfortunately, so what I am trying to do is create a model which is maximally valid for predicting from 2010 to 2020. The aim of the model is to take a snapshot of the city and predict which areas will improve in the scoring. Commented Aug 12, 2016 at 13:47