I'm working on a project that generates a list of decimal numbers every hour. The amount of values in the list can range between 1 - 6000 depending on the some variables (time of day etc). Each of the values will range between -1.0 and 1.0. Anything below -0.1 is considering a negative result and anything greater than 0.1 a positive result as there is a -0.1 - 0.1 rate of error in the calculations (anything in that range being regarded as neutral). The more the negative or positive the number is indicating an overall more negative or positive for that specific calculation.
I need to produce a single number that is representative of the list for each hour. While technically the values can range -1.0 - 1.0 most will be around the -0.4 - 0.4 range.
Would it be best to use mean, median, weighted-mean, something else I haven't thought of, or a combination (e.g. outputting two things that when looked at together give a good representation), to get the best representation of the lists?
The project is preforming sentiment analysis on messages. At at time interval messages as collected, processed, and then assigned a score as above. What I need is to generate a single number from each interval that is representative of the general sentiment. It doesn't matter too much if the number is wrong in some outside cases as it's only used as a guide, looked at in perspective with other data (number of total messages, ratio of positive/negative/neutral).
I initially did try simply calculating the average but that result in all the lists (there are 8 per interval) coming out around the same +/- 0.01 which when looking at the raw data was indicative of the actual collected results. I've also tried running median calculations on it and that seems to produce more representative figures but I don't know enough about statistics to know if that is true and a good formula to use, or if I should be using something else.
Basically the important part is that you can compare the current outputted value to other lists outputted values and previously outputted values for the same list to get a feel to the up or down trend.