Mean squared error for data with skewed distribution

Question

I am doing regression task and the response variables in my dataset have a skewed distribution. Say, for the sake of simplicity, that I have a model Y~X and Y(response variable) is in [1,5] but there are many more values in the [4,5] range than the [1,2] range. As a result a majority predictor that predicts 4.5 for Y regardless of the value of X can complete with linear regression if I just look at MSE error.

I am wondering if there is any methodical way to correct MSE to consider this case and maybe penalize the majority predictor more when it gets a Y=1 wrong than when it gets Y=5 wrong. Basically I am looking for a fair error measure for skewed data.

Update: for the sake of simplicity let's say the majority predictor predicts 4.5 for everything regardless of the value of X. My predictor predicts 1 acuratly but always predicts 4.4 for 5's.

the test set for Y is one 1 and the rest are 200 numbers and each equals 5. Based on MSE the majority predictor is better than my predictor but it does not make any sense.

I want to modify MSE to favor accurate predictions for 1's compared to accurate predictions for 5's. Perhaps I can multiply each residual by the inverse of the frequency of the actual Y? also how can I use such an MSE for cases that Y is continuous?

update 2: So some folks suggested that maybe Y should be sampled more properly and I need to find a subsample of my data that provides a uniform distribution of Y. This is not possible in my case. Say I am crawling amazon and most of the ratings that I see are 5 (since amazon removes under performing products) but there are some items with rating 1 as well. Now if I use a majority predictor that predicts 5 everywhere it is going to beat my SVM in terms of MSE value but the majority predictor brings no value to my system. Moreover I do not want to throw out my data just to make the distribution of ratings uniform. I believe it should be one by properly selecting the metric (error measure)

Attached is an example distribution.

Answer 1

One possible pragmatic response is inspired by the motivation behind the logit transformation used in logistic regression. In that situation, the original response is constrained between 0 and 1 and hence treating it as though it has a normal distribution causes all sorts of problems. Part of the response is to transform by log(y/(1-y)), where y is the modelled response.

Your problem is comparable, just that the limits are 1 and 5 rather than 0 and 1. If you want to analyse your data using methods motivated by Normal-distribution assumptions (into which I would categorise use of squared errors), you could consider transforming it on to a scale with no upper or lower bounds as below (in R, but the code should make sense if you're not familiar with it; the key line is the one that creates the "z" variable from the original "y"):

# generate some skewed data in the [1,5] space:
y <- rnorm(1000,4,1)
y[y<1] <- runif(sum(y<1),1,3)
y[y>5] <- runif(sum(y>5),3.5,5)

# transform it similar to logit transform
z <- log((y-1)/4 / (1-(y-1)/4))

# plot the results
par(mfrow=c(1,2))
hist(y); hist(z)

The new z variable is much more suited for OLS or whatever other similar techniques you might want to try.

Answer 2

If I understand correctly, the problem you are facing is that by using MSE you are developing a poor predictor. This is a common issue on skewed-population problems. For example, if you were trying to predict if a person has cancer or not (binary) in a popluation where 99% of people dont have cancer, by analyzing a routine blood exam, a model trained by minimizing MSE would say that nobody has cancer, and would by 99% precise.

One way to address this issue is to use Fscore instead of Precision, or MSE. Fscore is an error metric that uses both Precision and Recall. http://en.wikipedia.org/wiki/F1_score

I learnt this in Andrew Ng's Machine Learning Course on Coursera. Here is a video of the specific class on error metrics for skewed classes. http://www.youtube.com/watch?v=uj605bVFH8Y

Hope this helps!