# How to reduce the impact of variables that have high values for most respondents?

I have a data set with about 100 000 respondents and have tracked them watching 500 different programs. Each cell contains how long a respondent has watched the program. As you might have guessed, this results in a rather sparse data set, since there is nobody who watched all the 500 programs in the past 3 weeks.

If I run this data set using machine learning algorithms, I get rather poor results. So I would actually like to standardize my data a bit (in R). My problem is as follows: I would like to incorporate the fact that not all programs are as popular. So for example (this program is not in my data set, but only for illustration purposes), I think a lot of people watch the Big Bang Theory, so a high duration for that program might not be as informative as a high duration for Pokemon. So in order to fix this, we can simply subtract the average of the program from every unit in the column of that program.

The problem is as follows, for example for Pokemon we see a lot of zeroes. All these zeroes will be equal to minus the average if I do it this way. This is rather problematic, since my sparse matrix that I first had will not be sparse anymore.

Second, it is a bit weird that we have negative duration (i.e. negative how long someone watches something).

My questions are:

1. Are there alternative ways how I can normalize a program (i.e. a column) so I can get reduce the "effects of a popular program"?

2. Is this similar to transforming it into a unit vector?

3. Does it make sense to only calculate the average of the column without the zeroes and then subtract this "average" from all the nonzero values of the column?

Note: that I am asking for advice, so any suggestion with respect to normalization is welcome.

Update Response to Cam:

For the first approach I agree, that it might help.

In my question I was referring to subtracting the column mean from the durations of each person. That way I can standardize across column right? However if I do this for the people that only watch the show, then it will seem that for the zero elements in the column that they have watched the show as long as the average, which doesn't seem to be appropriate.

The second approach in your answer, however has a different problem in my opinion, because if I have a column:

 A |
0 |
0 |
1 |
8 |
6 |


For program A, the average is 3 and standard deviation is 3.74, so for respondent one and two, we have 0.8021, so it will turn all my zero values into nonzero values, which can be a problem for the storage.

• Why not dividing by the maximum value of each column? In that way you will have values from 0 to 1 in each column. Then why not trying to divide by the mean, or median and see which is better for your objective. You can also divide by the program's duration (episode duration), as maybe someone watches 20' a 20' episode and another one watches 40' a 60' episode. Just some thoughts..... – AntoniosK Oct 9 '15 at 22:56

A simple way to normalise the data such that each program has the same impact regardless of popularity is simply to divide each value by the sum of values in that column. For example, if your data only had 3 people (person A, B and C) and they each watched Pokemenon for:

Person A = 0 minutes

Person B = 5 minutes

Person C = 10 minutes

Then your normalised data would look as follows:

Person A = 0/(10+5+0)=0

Person B = 5/(10+5+0)=5/15=0.33

Person C = 10/(10+5+0)=10/15=0.67

The nice thing about this is that now the sum of values in each column will be equal. For example, even if people watch the big bang theory much more, the values for big bang theory will still sum to one:

Person A = 0 minutes -> 0

Person B = 500 minutes -> 0.33

Person C = 1000 minutes -> 0.67

The important thing to note here is that your interpretation changes. Values are no longer measured in minutes but are instead equal to the percentage of time that person spent watching show X relative to the total amount of time spent watching show X.

Alternatively, if you use your method of subtracting the mean from each value but want to standardise across columns you could look at the standard deviation of each column and find how many standard deviations each score is from the mean. For the examples above, in both cases Person C is 1 standard deviation above the mean, Person A is 1 standard deviation below and Person B is 0 standard deviations above/below. This is a nice way of finding which people tend to watch the show much more than other people and makes it easier to compare extreme scores across shows of different popularity

In terms of taking the mean of only those people that watch the show, this seems like a perfectly reasonable thing to do, except you should be sure to not that now your interpretations will be looking at how that person compares to other people that watch the show rather than looking at how that person compares to your entire sample/population.