I have a new data set which uses a 0 to 10 rating scale. I also have a very old data set which uses a 0 to 4 rating scale. There is certainly overlap between the old and the new if I were to simply update the old data to the new scale. In other words, on old 3 could be anywhere from 4 to 6 on the new scale. The old data set has a very large number of samples, and being able to use it would greatly enhance the quality of the results.

Any thoughts on how I can make use of the old data set?


$\DeclareMathOperator\E{\mathbb E}\newcommand\P{\mathbb P}$You could create a probabilistic model of what you think is going on: perhaps an old 3 has equal probability of being a new 4, 5, or 6. Maybe it has probability $\tfrac16$ to be a 4 or a 6 and $\tfrac23$ to be a 5. Whatever you think is most reasonable.

Then, for any given training example, call this random variable over labels $Y$. When the model predicts a label $\hat y$, since our training loss is now a random variable $\ell(Y, \hat y)$ (where $\ell$ is our loss function), it's probably most reasonable to minimize the expected loss $\mathbb E[\ell(Y, \hat y)]$.

When $\ell$ is the squared loss, then this is \begin{align} \E\left[ (\hat y - Y)^2 \right] &= \E\left[ \hat y^2 - 2 Y \hat y + Y^2 \right] \\&= \hat y^2 - 2 \E[ Y ] \, \hat y + \E[ Y^2 ] \\&= \left( \hat y - \E[Y] \right)^2 + \left( \E[ Y^2 ] - \E[Y]^2 \right) \\&= \left( \hat y - \E[Y] \right)^2 + \mathrm{Var}[Y] .\end{align} So, just using the mean of your label mapping actually gives you the same loss function, up to a constant shift of $\mathrm{Var}[Y]$. But since that shift is constant, minimizing it gives you the same model, and there's no reason to worry about the actual expected loss thing in practice – just map each data point to the mean of what you think the mapping should be.

If you're using a different loss function, this property might not hold. For example, for the absolute loss function, the best single number to summarize with is the median of $Y$, but in that case the loss function is no longer only offset by a constant, and if you really want to minimize the average loss you'll have to use that more complex loss function directly.

If you'd really rather not manually come up with a mapping, it's also possible to learn it, but that's more complicated. The machine learning community refers to this as a particular kind of "transfer learning"; I can point you in some directions if you're interested in that.

If your extra dataset is old, it's quite likely that you should be thinking about the possibility of covariate shift between the two datasets. This is where the marginal distribution of inputs $P(x)$ is different between the two datasets, i.e. maybe your old data had more phone-based reports and fewer internet ones (or whatever problem appropriate to your domain). This also falls under the domain of transfer learning, and can possibly be addressed in concert with the remapping of scales.

| cite | improve this answer | |
  • $\begingroup$ Dougal,Thanks for the response so quickly. It is appreciated. I will give some more detail which I think will clear things up. We collected details on the deterioration of steel in acidic environments. Because the data was field collected and because of technology limitations, the loss of steel (dependent variable) is qualitatively measured using a rating scale. There are quite a few predictors that are being regressed. $\endgroup$ – Kevin White Jan 26 '16 at 12:30
  • $\begingroup$ In this study several thousand data points were collected. Based on the results of the original work, it was determined that a larger scale was necessary to properly rate the metal loss (and provide a more linear temporal distribution of the ratings). Recently, we have collected about 300 additional data points using this new 0-10 scale. The question is what is the best way to incorporate the old data into the regression analysis? Our thought, I believe, is what you say above. Simply rerank the old date using the new scale. Where there is overlap, use what we believe to be the mean. $\endgroup$ – Kevin White Jan 26 '16 at 12:40
  • $\begingroup$ @KevinWhite Yep, great – that seems like exactly the setting I'm talking about then. If you're using squared loss, just using the mean is therefore justified; if you're doing some kind of logic regression or such that actually takes into account the discrete levels, it probably won't correspond exactly to minimizing the expected loss, but it may still be a reasonable thing to do. $\endgroup$ – djs Jan 26 '16 at 19:34
  • $\begingroup$ +1 Ive done this sort of thing when we change pricing algorithms at work. Never knew it was called "transfer learning". $\endgroup$ – Matthew Drury Jan 27 '16 at 6:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.