I have a data set with about 100 000 respondents and have tracked them watching 500 different programs. Each cell contains how long a respondent has watched the program. As you might have guessed, this results in a rather sparse data set, since there is nobody who watched all the 500 programs in the past 3 weeks.
If I run this data set using machine learning algorithms, I get rather poor results. So I would actually like to standardize my data a bit (in R). My problem is as follows: I would like to incorporate the fact that not all programs are as popular. So for example (this program is not in my data set, but only for illustration purposes), I think a lot of people watch the Big Bang Theory, so a high duration for that program might not be as informative as a high duration for Pokemon. So in order to fix this, we can simply subtract the average of the program from every unit in the column of that program.
The problem is as follows, for example for Pokemon we see a lot of zeroes. All these zeroes will be equal to minus the average if I do it this way. This is rather problematic, since my sparse matrix that I first had will not be sparse anymore.
Second, it is a bit weird that we have negative duration (i.e. negative how long someone watches something).
My questions are:
Are there alternative ways how I can normalize a program (i.e. a column) so I can get reduce the "effects of a popular program"?
Is this similar to transforming it into a unit vector?
Does it make sense to only calculate the average of the column without the zeroes and then subtract this "average" from all the nonzero values of the column?
Note: that I am asking for advice, so any suggestion with respect to normalization is welcome.
Update Response to Cam:
For the first approach I agree, that it might help.
In my question I was referring to subtracting the column mean from the durations of each person. That way I can standardize across column right? However if I do this for the people that only watch the show, then it will seem that for the zero elements in the column that they have watched the show as long as the average, which doesn't seem to be appropriate.
The second approach in your answer, however has a different problem in my opinion, because if I have a column:
A | 0 | 0 | 1 | 8 | 6 |
For program A, the average is 3 and standard deviation is 3.74, so for respondent one and two, we have 0.8021, so it will turn all my zero values into nonzero values, which can be a problem for the storage.