I am doing regression task and the response variables in my dataset have a skewed distribution. Say, for the sake of simplicity, that I have a model Y~X and Y(response variable) is in [1,5] but there are many more values in the [4,5] range than the [1,2] range. As a result a majority predictor that predicts 4.5 for Y regardless of the value of X can complete with linear regression if I just look at MSE error.
I am wondering if there is any methodical way to correct MSE to consider this case and maybe penalize the majority predictor more when it gets a Y=1 wrong than when it gets Y=5 wrong. Basically I am looking for a fair error measure for skewed data.
Update: for the sake of simplicity let's say the majority predictor predicts 4.5 for everything regardless of the value of X. My predictor predicts 1 acuratly but always predicts 4.4 for 5's.
the test set for Y is one 1 and the rest are 200 numbers and each equals 5. Based on MSE the majority predictor is better than my predictor but it does not make any sense.
I want to modify MSE to favor accurate predictions for 1's compared to accurate predictions for 5's. Perhaps I can multiply each residual by the inverse of the frequency of the actual Y? also how can I use such an MSE for cases that Y is continuous?
update 2: So some folks suggested that maybe Y should be sampled more properly and I need to find a subsample of my data that provides a uniform distribution of Y. This is not possible in my case. Say I am crawling amazon and most of the ratings that I see are 5 (since amazon removes under performing products) but there are some items with rating 1 as well. Now if I use a majority predictor that predicts 5 everywhere it is going to beat my SVM in terms of MSE value but the majority predictor brings no value to my system. Moreover I do not want to throw out my data just to make the distribution of ratings uniform. I believe it should be one by properly selecting the metric (error measure)
Attached is an example distribution.