How to use old data with a different rating scale from the new data? I have a new data set which uses a 0 to 10 rating scale.  I also have a very old data set which uses a 0 to 4 rating scale. There is certainly overlap between the old and the new if I were to simply update the old data to the new scale.  In other words, on old 3 could be anywhere from 4 to 6 on the new scale.  The old data set has a very large number of samples, and being able to use it would greatly enhance the quality of the results.
Any thoughts on how I can make use of the old data set?
 A: $\DeclareMathOperator\E{\mathbb E}\newcommand\P{\mathbb P}$You could create a probabilistic model of what you think is going on: perhaps an old 3 has equal probability of being a new 4, 5, or 6. Maybe it has probability $\tfrac16$ to be a 4 or a 6 and $\tfrac23$ to be a 5. Whatever you think is most reasonable.
Then, for any given training example, call this random variable over labels $Y$. When the model predicts a label $\hat y$, since our training loss is now a random variable $\ell(Y, \hat y)$ (where $\ell$ is our loss function), it's probably most reasonable to minimize the expected loss $\mathbb E[\ell(Y, \hat y)]$.
When $\ell$ is the squared loss, then this is
\begin{align}
\E\left[ (\hat y - Y)^2 \right]
&= \E\left[ \hat y^2 - 2 Y \hat y + Y^2 \right]
\\&= \hat y^2 - 2 \E[ Y ] \, \hat y + \E[ Y^2 ]
\\&= \left( \hat y - \E[Y] \right)^2 + \left( \E[ Y^2 ] - \E[Y]^2 \right)
\\&= \left( \hat y - \E[Y] \right)^2 + \mathrm{Var}[Y]
.\end{align}
So, just using the mean of your label mapping actually gives you the same loss function, up to a constant shift of $\mathrm{Var}[Y]$. But since that shift is constant, minimizing it gives you the same model, and there's no reason to worry about the actual expected loss thing in practice – just map each data point to the mean of what you think the mapping should be.
If you're using a different loss function, this property might not hold. For example, for the absolute loss function, the best single number to summarize with is the median of $Y$, but in that case the loss function is no longer only offset by a constant, and if you really want to minimize the average loss you'll have to use that more complex loss function directly.

If you'd really rather not manually come up with a mapping, it's also possible to learn it, but that's more complicated. The machine learning community refers to this as a particular kind of "transfer learning"; I can point you in some directions if you're interested in that.
If your extra dataset is old, it's quite likely that you should be thinking about the possibility of covariate shift between the two datasets. This is where the marginal distribution of inputs $P(x)$ is different between the two datasets, i.e. maybe your old data had more phone-based reports and fewer internet ones (or whatever problem appropriate to your domain). This also falls under the domain of transfer learning, and can possibly be addressed in concert with the remapping of scales.
